微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

Spark-Solr错误,可能是日志记录或hadoop主版本

如何解决Spark-Solr错误,可能是日志记录或hadoop主版本

spark-solr https://github.com/lucidworks/spark-solr

错误

错误是当我启动$ spark-shell

ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console. Set system property 'org.apache.logging.log4j.simplelog.StatusLogger.level' to TRACE to show Log4j2 internal initialization logging.
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/jars/slf4j-log4j12-1.7.25.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/opt/cloudera/parcels/CDH-6.3.1-1.cdh6.3.1.p0.1470567/lib/spark/jars/spark-solr-3.6.0-shaded.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
java.lang.IllegalArgumentException: Unrecognized Hadoop major version number: 3.0.0-cdh6.3.1
  at org.apache.hadoop.hive.shims.Shimloader.getMajorVersion(Shimloader.java:174)
  at org.apache.hadoop.hive.shims.Shimloader.loadShims(Shimloader.java:139)
  at org.apache.hadoop.hive.shims.Shimloader.getHadoopShims(Shimloader.java:100)
  at org.apache.hadoop.hive.conf.HiveConf$ConfVars.<clinit>(HiveConf.java:368)
  at org.apache.hadoop.hive.conf.HiveConf.<clinit>(HiveConf.java:105)
  at java.lang.class.forName0(Native Method)
  at java.lang.class.forName(Class.java:348)
  at org.apache.spark.util.Utils$.classForName(Utils.scala:242)
  at org.apache.spark.sql.SparkSession$.hiveClassesArePresent(SparkSession.scala:1117)
  at org.apache.spark.repl.Main$.createSparkSession(Main.scala:102)
  ... 62 elided
<console>:14: error: not found: value spark
       import spark.implicits._
              ^
<console>:14: error: not found: value spark
       import spark.sql
              ^

spark-solr中的.jar是spark-solr-3.6.0-shaded.jar和spark-solr-3.6.0.jar

我的设置是Ubuntu 18,Solr 7.4,Spark 2.4.0,Scala 2.11.12

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。