如何解决部署在AKS上时,无法将Spark History Server的日志目录配置为Storage Blob
我正在尝试在AKS上部署Spark History Server,并想将其日志目录指向Storage Blob。为了实现这一目标,我将配置放在 values.yaml 文件中,如下所示:-
wasbs:
enableWASBS: true
secret: azure-secrets
sasKeyMode: false
storageAccountKeyName: azure-storage-account-key
storageAccountNameKeyName: azure-storage-account-name
containerKeyName: azure-blob-container-name
logDirectory: wasbs:///test/piyush/spark-history
pvc:
enablePVC: false
nfs:
enableExampleNFS: false
首先,我使用以下命令创建天蓝色秘密:-
kubectl create secret generic azure-secrets --from-file=azure-storage-account-name --from-file=azure-blob-container-name --from-file=azure-storage-account-key
然后,我运行以下命令集:-
helm repo add stable https://kubernetes-charts.storage.googleapis.com
helm install stable/spark-history-server --values values.yaml --generate-name
但是在执行操作时,出现以下错误:-
2020-10-05 19:17:56 INFO HistoryServer:2566 - Started daemon with process name: 12@spark-history-server-1601925447-57c5476fb-5wh6q
2020-10-05 19:17:56 INFO SignalUtils:54 - Registered signal handler for TERM
2020-10-05 19:17:56 INFO SignalUtils:54 - Registered signal handler for HUP
2020-10-05 19:17:56 INFO SignalUtils:54 - Registered signal handler for INT
2020-10-05 19:17:56 WARN NativeCodeLoader:62 - Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
2020-10-05 19:17:56 INFO SecurityManager:54 - Changing view acls to: root
2020-10-05 19:17:56 INFO SecurityManager:54 - Changing modify acls to: root
2020-10-05 19:17:56 INFO SecurityManager:54 - Changing view acls groups to:
2020-10-05 19:17:56 INFO SecurityManager:54 - Changing modify acls groups to:
2020-10-05 19:17:56 INFO SecurityManager:54 - SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(root); groups with view permissions: Set(); users with modify permissions: Set(root); groups with modify permissions: Set()
2020-10-05 19:17:56 INFO FsHistoryProvider:54 - History server ui acls disabled; users with admin permissions: ; groups with admin permissions
Exception in thread "main" java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:280)
at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala)
Caused by: java.lang.RuntimeException: java.lang.classNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2195)
at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2654)
at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667)
at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94)
at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703)
at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:364)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:117)
at org.apache.spark.deploy.history.FsHistoryProvider.<init>(FsHistoryProvider.scala:86)
... 6 more
Caused by: java.lang.classNotFoundException: Class org.apache.hadoop.fs.azure.NativeAzureFileSystem not found
at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2101)
at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2193)
... 16 more
任何类型的帮助或建议将不胜感激。
谢谢!
版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。