目录
1. 基于docker部署cdh
基于Docker搭建CDH单机单节点集群_litlit023的专栏-CSDN博客
2. 遇到的异常
2.1 flink下缺少hadoop相关依赖
基于docker 部署cdn后，启动flink on yarn模式，报了以下错误
./bin/yarn-session.sh -n 2 -jm 1024 -tm 1024
报了以下错误
Error: A JNI error has occurred, please check your installation and try again
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException
at java.lang.class.getDeclaredMethods0(Native Method)
at java.lang.class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.class.privategetmethodRecursive(Class.java:3048)
at java.lang.class.getmethod0(Class.java:3018)
at java.lang.class.getmethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:650)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:632)
Caused by: java.lang.classNotFoundException: org.apache.hadoop.yarn.exceptions.YarnException
at java.net.urlclassloader.findClass(urlclassloader.java:382)
at java.lang.classLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.classLoader.loadClass(ClassLoader.java:351)
... 7 more解决方法：
提示flink启动缺少Hadoop yarn的相关jar包，从官网下载Flink依赖Hadoop的相关jar包；
进入官网，下载对应的相关依赖jar包，放到flink lib目录下即可
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/yarn/exceptions/YarnException - RICH-ATONE - 博客园Apache Flink: Downloads
根据官网部署Flink on YARN过程中遇到的问题及解决方法 - 简书
2.2 jdk7造成的错误
Container exited with a non-zero exit code 1
Failing this attempt. Failing the application.
If log aggregation is enabled on your cluster, use this command to further investigate the issue:
yarn logs -applicationId application_1643103905134_0001
at org.apache.flink.yarn.YarnClusterDescriptor.startAppMaster(YarnClusterDescriptor.java:1219)
at org.apache.flink.yarn.YarnClusterDescriptor.deployInternal(YarnClusterDescriptor.java:607)
at org.apache.flink.yarn.YarnClusterDescriptor.deploySessionCluster(YarnClusterDescriptor.java:419)
... 7 more
2022-01-25 17:49:17,827 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cancelling deployment from Deployment Failure Hook
2022-01-25 17:49:17,840 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
2022-01-25 17:49:17,845 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Killing YARN application
2022-01-25 17:49:17,931 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Killed application application_1643103905134_0001
2022-01-25 17:49:18,033 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Deleting files in hdfs://quickstart.cloudera:8020/user/root/.flink/application_1643103905134_0001.
[root@vm01 flink-1.14.3]# yarn logs -applicationId application_1643103905134_0001
22/01/25 17:49:32 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
Container: container_1643103905134_0001_01_000001 on quickstart.cloudera_8041
===============================================================================
LogType:jobmanager.err
Log Upload Time:星期二 一月 25 17:49:19 +0800 2022
LogLength:160
Log Contents:
Unrecognized VM option 'MaxMetaspaceSize=268435456'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
LogType:jobmanager.out
Log Upload Time:星期二 一月 25 17:49:19 +0800 2022
LogLength:0
Log Contents:Unrecognized VM option 'MaxMetaspaceSize=268435456'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
需要将cdh组件所用的jdk改为jdk8以上，我的环境时基于docker来部署的cdh 5.x，因此进入到容器内，修改容器内的jdk为jdk8即可
修改jdk为自己的jdk8
进入到容器docker exec -it mycdh3 bash
docker cp /work/jdk/jdk1.8.0_271 02155f0f3679:/root/
cd /usr/java/jdk1.7.0_67-cloudera
rm -rf ./*
cp -r /root/jdk1.8.0_271/* /usr/java/jdk1.7.0_67-cloudera/修改jdk版本：
java - Unrecognized option: -MaxMetaspaceSize=256m - Stack Overflow
How to configure java home on flink job on yarn - JiKe DevOps Community
java - Unrecognized option: -MaxMetaspaceSize=256m - Stack Overflow
Exposing a port on a live Docker container - Stack Overflow
flink on yarn 集群模式启动报错及解决方案汇总_大数据技术博客-CSDN博客_flink on yarn 报错
flink on yarn遇到的yarn资源不足问题_无名子七叶莲的博客-程序员宝宝 - 程序员宝宝
3. 启动flink on yarn
模式
[root@vm01 flink-1.14.3]# ./bin/yarn-session.sh -jm 1024 -tm 1024
2022-01-25 20:38:55,885 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, localhost
2022-01-25 20:38:55,894 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2022-01-25 20:38:55,896 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2022-01-25 20:38:55,897 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1728m
2022-01-25 20:38:55,897 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2022-01-25 20:38:55,898 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2022-01-25 20:38:55,898 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2022-01-25 20:38:57,062 INFO org.apache.flink.runtime.security.modules.HadoopModule [] - Hadoop user set to root (auth:SIMPLE)
2022-01-25 20:38:57,095 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-3655855445473519286.conf.
2022-01-25 20:38:57,390 WARN org.apache.flink.yarn.configuration.YarnLogConfigUtil [] - The configuration directory ('/work/flink/streamx_dir/flink-1.14.3/conf') already contains a LOG4J config file.If you want to use logback, then please delete or rename the log configuration file.
2022-01-25 20:38:57,951 INFO org.apache.hadoop.yarn.client.RMProxy [] - Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032
2022-01-25 20:38:58,732 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:38:58,736 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:38:58,737 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction network memory (57.600mb (60397978 bytes)) is less than its min value 64.000mb (67108864 bytes), min value will be used instead
2022-01-25 20:38:59,287 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Cluster specification: ClusterSpecification{masterMemoryMB=1024, taskManagerMemoryMB=1024, slotsPerTaskManager=1}
2022-01-25 20:39:00,148 WARN org.apache.hadoop.util.NativeCodeLoader [] - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2022-01-25 20:39:05,590 INFO org.apache.flink.runtime.util.config.memory.ProcessMemoryUtils [] - The derived from fraction jvm overhead memory (102.400mb (107374184 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
2022-01-25 20:39:05,671 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Submitting application master application_1643114118293_0002
2022-01-25 20:39:05,740 INFO org.apache.hadoop.yarn.client.api.impl.YarnClientImpl [] - Submitted application application_1643114118293_0002
2022-01-25 20:39:05,740 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Waiting for the cluster to be allocated
2022-01-25 20:39:05,745 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - deploying cluster, current state ACCEPTED
2022-01-25 20:39:24,071 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - YARN application has been deployed successfully.
2022-01-25 20:39:24,072 INFO org.apache.flink.yarn.YarnClusterDescriptor [] - Found Web Interface quickstart.cloudera:36283 of application 'application_1643114118293_0002'.
JobManager Web Interface: http://quickstart.cloudera:36283
版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。
