java.lang.ClassNotFoundException：azkanban hive 作业中的 org.apache.hadoop.conf.Configuration

如何解决java.lang.ClassNotFoundException：azkanban hive 作业中的 org.apache.hadoop.conf.Configuration

CDH 6.3.2

阿兹卡班 3.90.0

我想在 azkaban 中运行 hive 工作,但我得到一个错误

23-06-2021 11:21:05 CST hive-jdbc ERROR - Job run Failed!
java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
    at azkaban.jobtype.AbstractHadoopJavaProcessJob.setupHadoopJobProperties(AbstractHadoopJavaProcessJob.java:68)
    at azkaban.jobtype.HadoopHiveJob.run(HadoopHiveJob.java:49)
    at azkaban.execapp.JobRunner.runJob(JobRunner.java:830)
    at azkaban.execapp.JobRunner.doRun(JobRunner.java:607)
    at azkaban.execapp.JobRunner.run(JobRunner.java:568)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.classNotFoundException: org.apache.hadoop.conf.Configuration
    at java.net.urlclassloader.findClass(urlclassloader.java:382)
    at java.lang.classLoader.loadClass(ClassLoader.java:418)
    at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
    at java.lang.classLoader.loadClass(ClassLoader.java:351)
    ... 10 more
23-06-2021 11:21:05 CST hive-jdbc ERROR - org/apache/hadoop/conf/Configuration cause: java.lang.classNotFoundException: org.apache.hadoop.conf.Configuration

in commonprivate.properties

hadoop.classpath=hadoop classpath
hadoop.home=/opt/cloudera/parcels/CDH/lib/hadoop
hive.home=/opt/cloudera/parcels/CDH/lib/hive
spark.home=/opt/cloudera/parcels/CDH/lib/spark
hadoop.conf=/etc/hadoop/conf
jobtype.global.classpath=${hadoop.conf},/opt/cloudera/parcels/CDH/jars/*,${hive.home}/conf,${spark.home}/conf

在 hive/private.properties 中

jobtype.classpath=${hadoop.home}/conf,${hive.aux.jar.path}
jobtype.class=azkaban.jobtype.HadoopHiveJob

hive.aux.jar.path=${hive.home}/auxlib

解决方法

与CDH

修改azkaban-exec-server-3.90.0/bin/internal/internal-start-executor.sh。我能够通过以下调整使其工作。

if [ "$HADOOP_HOME" != "" ]; then
echo "Using Hadoop from $HADOOP_HOME"
#CLASSPATH=$CLASSPATH:$HADOOP_HOME/conf:$HADOOP_HOME/*
CLASSPATH=$CLASSPATH:/etc/hadoop/conf:/opt/cloudera/parcels/CDH/jars/*
JAVA_LIB_PATH="-Djava.library.path=$HADOOP_HOME/lib/native/Linux-amd64-64"
else
echo "Error: HADOOP_HOME is not set. Hadoop job types will not run properly."
fi