微信公众号搜"智元新知"关注
微信扫一扫可直接关注哦!

无法启动管理器:GridManagerAdapter,由类 IgniteSpiException 引起无法使用给定的类加载器 sun.misc.Launcher 反序列化对象

如何解决无法启动管理器:GridManagerAdapter,由类 IgniteSpiException 引起无法使用给定的类加载器 sun.misc.Launcher 反序列化对象

I am using "gridgain-community-8.8.1" package to bring up ignite in one AWS EC2 instance and access that from AWS EMR Spark-3.0.1 job. While trying to get the ignite instance from spark job using "Ignition.getorStart(igniteConf)",its throwing exceptions. I am using same java openJDK8 and gridgain library 8.8.1 for both EC2 instance and EMR spark job.

Below ignite config I have used to bring up ignite in EC2 instance,ignite was successfully brought up: (using ./gridgain-community-8.8.1/bin/ignite.sh aws-s3-ip-finder.xml)

aws-s3-ip-finder.xml 内容为:

<beans xmlns="http://www.springframework.org/schema/beans"
       xmlns:util="http://www.springframework.org/schema/util"
       xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="         http://www.springframework.org/schema/beans
    http://www.springframework.org/schema/beans/spring-beans.xsd
    http://www.springframework.org/schema/util
    http://www.springframework.org/schema/util/spring-util.xsd">
    <bean class="org.apache.ignite.configuration.IgniteConfiguration">

        <!-- Enable peer class loading. -->
        <!-- <property name="peerClassLoadingEnabled" value="true"/> -->
        <!-- Set deployment mode. -->
        <!-- <property name="deploymentMode" value="CONTINUOUS"/> -->

        <property name="discoverySpi">
            <bean class="org.apache.ignite.spi.discovery.tcp.TcpdiscoverySpi">
                <property name="ipFinder">
                    <bean class="org.apache.ignite.spi.discovery.tcp.ipfinder.s3.TcpdiscoveryS3IpFinder">
                        <property name="awsCredentialsProvider" ref="aws.creds"/>
                        <property name="bucketName" value="a206768-la-ignite-s3-ip-finder"/>
                    </bean>
                </property>
            </bean>
        </property>
    </bean>

    <!-- Instance Profile based credentials -->
    <bean id="aws.creds" class="com.amazonaws.auth.InstanceProfileCredentialsProvider">
        <constructor-arg value="false" />
    </bean>
</beans>

下面是我连接到这个 EC2 ignite 实例的 Scala 代码

def connectToIgniteClient(): Ignite = {
    logInfo(">>> Connecting to ignite-client")
    
    val instanceProfileCreds = new InstanceProfileCredentialsProvider(false)
    logInfo(">>> IP finding")

    val ipFinder = new TcpdiscoveryS3IpFinder()
    ipFinder.setAwsCredentialsProvider(instanceProfileCreds)
    ipFinder.setBucketName("a206768-la-ignite-s3-ip-finder")
    logInfo(">>> S3 connected")

    val discoverySpi = new TcpdiscoverySpi()
    discoverySpi.setIpFinder(ipFinder)
    logInfo(">>> Spi connected")

    val igniteConf = new IgniteConfiguration()
    igniteConf.setClientMode(true)
    igniteConf.setdiscoverySpi(discoverySpi)
    logInfo(">>> Config set")

    Ignition.getorStart(igniteConf) //Exception is thrown from this line
  }

以下是来自 EMR Spark 集群的异常日志:

21/03/25 17:32:24 INFO Client: Application report for application_1616693342082_0001 (state: RUNNING)
21/03/25 17:32:24 INFO Client: 
     client token: N/A
     diagnostics: N/A
     ApplicationMaster host: 10.150.80.195
     ApplicationMaster RPC port: -1
     queue: default
     start time: 1616693536055
     final status: UNDEFINED
     tracking URL: http://ip-10-150-81-252.xxxx.aws-int.thomsonreuters.com:20888/proxy/application_1616693342082_0001/
     user: hadoop
21/03/25 17:32:24 INFO YarnClientSchedulerBackend: Application application_1616693342082_0001 has started running.
21/03/25 17:32:24 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 45257.
21/03/25 17:32:24 INFO NettyBlockTransferService: Server created on ip-10-150-81-252.xxxx.aws-int.thomsonreuters.com:45257
21/03/25 17:32:24 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/03/25 17:32:24 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver,ip-10-150-81-252.xxxx.aws-int.thomsonreuters.com,45257,None)
21/03/25 17:32:24 INFO BlockManagerMasterEndpoint: Registering block manager ip-10-150-81-252.xxxx.aws-int.thomsonreuters.com:45257 with 1028.8 MiB RAM,BlockManagerId(driver,None)
21/03/25 17:32:24 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver,None)
21/03/25 17:32:24 INFO BlockManager: external shuffle service port = 7337
21/03/25 17:32:24 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver,None)
21/03/25 17:32:24 INFO ServerInfo: Adding filter to /metrics/json: org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
21/03/25 17:32:24 INFO ContextHandler: Started o.s.j.s.ServletContextHandler@194224ca{/metrics/json,null,AVAILABLE,@Spark}
21/03/25 17:32:24 INFO SingleEventLogFileWriter: Logging events to hdfs:/var/log/spark/apps/application_1616693342082_0001.inprogress
21/03/25 17:32:25 INFO Utils: Using initial executors = 50,max of spark.dynamicAllocation.initialExecutors,spark.dynamicAllocation.minexecutors and spark.executor.instances
21/03/25 17:32:25 WARN YarnSchedulerBackend$YarnSchedulerEndpoint: Attempted to request executors before the AM has registered!
21/03/25 17:32:25 INFO YarnClientSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.0
21/03/25 17:32:25 INFO YarnSchedulerBackend$YarnSchedulerEndpoint: ApplicationMaster registered as NettyRpcEndpointRef(spark-client://YarnAM)
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> Spark initialized
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> Loading Person data to ignite caches
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> Connecting to ignite
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> Connecting to ignite-client
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> IP finding
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> S3 connected
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> Spi connected
21/03/25 17:32:25 INFO IgniteCommunicatorPerson: >>> Config set
21/03/25 17:32:25 WARN : Failed to resolve default logging config file: config/java.util.logging.properties
Console logging handler is not configured.
21/03/25 17:32:25 WARN G: Ignite work directory is not provided,automatically resolved to: /mnt/var/lib/hadoop/steps/s-FV02OVOM7RT4/ignite/work
21/03/25 17:32:26 INFO IgniteKernal: 

21/03/25 17:32:26 INFO IgniteKernal: Config URL: n/a
21/03/25 17:32:26 INFO IgniteKernal: IgniteConfiguration [igniteInstanceName=null,pubPoolSize=8,svcPoolSize=8,callbackPoolSize=8,stripedPoolSize=8,sysPoolSize=8,mgmtPoolSize=4,dataStreamerPoolSize=8,utilityCachePoolSize=8,utilityCacheKeepAliveTime=60000,p2pPoolSize=2,qryPoolSize=8,buildIdxPoolSize=1,igniteHome=null,igniteworkdir=/mnt/var/lib/hadoop/steps/s-FV02OVOM7RT4/ignite/work,mbeanSrv=com.sun.jmx.mbeanserver.JmxMBeanServer@4d1f1ff5,nodeId=c2f73ab1-066c-4fc1-b1d2-2ebd6ca79311,marsh=BinaryMarshaller [],marshLocJobs=false,daemon=false,p2pEnabled=false,netTimeout=5000,netCompressionLevel=1,sndRetryDelay=1000,sndRetryCnt=3,metricsHistSize=10000,metricsUpdateFreq=2000,metricsExpTime=9223372036854775807,discoSpi=TcpdiscoverySpi [addrRslvr=null,sockTimeout=0,ackTimeout=0,marsh=null,reconCnt=10,reconDelay=2000,maxAckTimeout=600000,soLinger=5,forceSrvMode=false,clientReconnectdisabled=false,internalLsnr=null,skipAddrsRandomization=false],segPlc=STOP,segResolveAttempts=2,waitForSegOnStart=true,allResolversPassReq=true,segChkFreq=10000,commSpi=TcpCommunicationSpi [connectGate=org.apache.ignite.spi.communication.tcp.internal.ConnectGateway@23f8036d,ctxInitLatch=java.util.concurrent.CountDownLatch@68f69ca3[Count = 1],stopping=false,clientPool=null,nioSrvWrapper=null,stateProvider=null,connectionRequestor=null],evtSpi=org.apache.ignite.spi.eventstorage.NoopEventStorageSpi@1e3566e,colSpi=NoopCollisionSpi [],deploySpi=LocalDeploymentSpi [],indexingSpi=org.apache.ignite.spi.indexing.noop.NoopIndexingSpi@2842c098,addrRslvr=null,encryptionSpi=org.apache.ignite.spi.encryption.noop.NoopEncryptionSpi@2820b369,tracingSpi=org.apache.ignite.spi.tracing.NoopTracingSpi@46b21632,clientMode=true,rebalanceThreadPoolSize=1,rebalanceTimeout=10000,rebalanceBatchesPrefetchCnt=2,rebalanceThrottle=0,rebalanceBatchSize=524288,txCfg=TransactionConfiguration [txSerEnabled=false,dfltIsolation=REPEATABLE_READ,dfltConcurrency=pessimistic,dfltTxTimeout=0,txTimeoutOnPartitionMapExchange=0,deadlockTimeout=10000,pessimisticTxLogSize=0,pessimisticTxLogLinger=10000,tmLookupClsName=null,txManagerFactory=null,useJtaSync=false],cacheSanityCheckEnabled=true,discoStartupDelay=60000,deployMode=SHARED,p2pMissedCacheSize=100,locHost=null,timeSrvPortBase=31100,timeSrvPortRange=100,failureDetectionTimeout=10000,sysWorkerBlockedTimeout=120000,clientFailureDetectionTimeout=30000,metricslogFreq=60000,connectorCfg=ConnectorConfiguration [jettyPath=null,host=null,port=11211,noDelay=true,directBuf=false,sndBufSize=32768,rcvBufSize=32768,idleQryCurTimeout=600000,idleQryCurCheckFreq=60000,sndQueueLimit=0,selectorCnt=4,idleTimeout=7000,sslEnabled=false,sslClientAuth=false,sslCtxFactory=null,sslFactory=null,portRange=100,threadPoolSize=8,msginterceptor=null],odbcCfg=null,warmupClos=null,atomicCfg=AtomicConfiguration [seqReserveSize=1000,cacheMode=PARTITIONED,backups=1,aff=null,grpname=null],classLdr=null,platformCfg=null,binaryCfg=null,memCfg=null,pstCfg=null,dsCfg=null,activeOnStart=null,autoActivation=null,clusterStateOnStart=null,sqlConnCfg=null,cliConnCfg=ClientConnectorConfiguration [host=null,port=10800,sockSndBufSize=0,sockRcvBufSize=0,tcpNoDelay=true,maxOpenCursorsPerConn=128,idleTimeout=0,handshakeTimeout=10000,jdbcEnabled=true,odbcEnabled=true,thinCliEnabled=true,useIgniteSslCtxFactory=true,thinCliCfg=ThinClientConfiguration [maxActiveTxPerConn=100,maxActiveComputeTasksPerConn=0]],mvccVacuumThreadCnt=2,mvccVacuumFreq=5000,authEnabled=false,failureHnd=null,commFailureRslvr=null,sqlCfg=sqlConfiguration [longQryWarnTimeout=1000,sqlQryHistSize=1000,sqlGlobalMemoryQuota=60%,sqlQueryMemoryQuota=0,sqlOffloadingEnabled=false,dfltQryTimeout=0]]
21/03/25 17:32:26 INFO IgniteKernal: Daemon mode: off
21/03/25 17:32:26 INFO IgniteKernal: OS: Linux 4.14.200-155.322.amzn2.x86_64 amd64
21/03/25 17:32:26 INFO IgniteKernal: OS user: hadoop
21/03/25 17:32:26 INFO IgniteKernal: PID: 14773
21/03/25 17:32:26 INFO IgniteKernal: Language runtime: Java Platform API Specification ver. 1.8
21/03/25 17:32:26 INFO IgniteKernal: VM information: OpenJDK Runtime Environment 1.8.0_272-b10 Amazon.com Inc. OpenJDK 64-Bit Server VM 25.272-b10
21/03/25 17:32:26 INFO IgniteKernal: VM total memory: 2.0GB
21/03/25 17:32:26 INFO IgniteKernal: Remote Management [restart: off,REST: off,JMX (remote: off)]
21/03/25 17:32:26 INFO IgniteKernal: Logger: JavaLogger [quiet=true,config=null]
21/03/25 17:32:26 INFO IgniteKernal: IGNITE_HOME=null
21/03/25 17:32:26 INFO IgniteKernal: VM arguments: [-DEMR_CLUSTER_ID=j-3MH2DITFW5UJE,-DEMR_RELEASE_LABEL=emr-6.2.0,-DEMR_STEP_ID=s-FV02OVOM7RT4,-Xmx2048M,-XX:OnOutOfMemoryError=kill -9 %p,-XX:+UseConcmarkSweepGC,-XX:CMSInitiatingOccupancyFraction=70,-XX:MaxHeapFreeRatio=70,-XX:+CMSClassUnloadingEnabled]
21/03/25 17:32:26 INFO IgniteKernal: Configured caches [in 'sysMemPlc' dataRegion: ['ignite-sys-cache']]
21/03/25 17:32:26 WARN IgniteKernal: Please set system property '-Djava.net.preferIPv4Stack=true' to avoid possible problems in mixed environments.
21/03/25 17:32:26 WARN GridDiagnostic: Initial heap size is 236MB (should be no less than 512MB,use -xms512m -Xmx512m).
21/03/25 17:32:26 INFO IgnitePluginProcessor: Configured plugins:
21/03/25 17:32:26 INFO IgnitePluginProcessor:   ^-- None
21/03/25 17:32:26 INFO IgnitePluginProcessor: 
21/03/25 17:32:26 INFO FailureProcessor: Configured failure handler: [hnd=StopNodeOrHaltFailureHandler [tryStop=false,timeout=0,super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYstem_WORKER_BLOCKED,SYstem_CRITICAL_OPERATION_TIMEOUT]]]]
21/03/25 17:32:26 INFO TcpCommunicationSpi: Successfully bound communication NIO server to TCP port [port=47100,locHost=0.0.0.0/0.0.0.0,selectorsCnt=4,selectorSpins=0,pairedConn=false]
21/03/25 17:32:26 WARN TcpCommunicationSpi: Message queue limit is set to 0 which may lead to potential OOMEs when running cache operations in FULL_ASYNC or PRIMARY_SYNC modes due to message queues growth on sender and receiver sides.
21/03/25 17:32:26 WARN NoopCheckpointSpi: Checkpoints are disabled (to enable configure any GridCheckpointSpi implementation)
21/03/25 17:32:26 WARN GridCollisionManager: Collision resolution is disabled (all jobs will be activated upon arrival).
21/03/25 17:32:26 INFO IgniteKernal: Security status [authentication=off,tls/ssl=off]
21/03/25 17:32:27 INFO PdsFoldersResolver: Consistent ID used for local node is [c2f73ab1-066c-4fc1-b1d2-2ebd6ca79311] according to persistence data storage folders
21/03/25 17:32:27 WARN IgniteH2Indexing: Serialization of Java objects in H2 was enabled.
21/03/25 17:32:27 INFO QueryMemoryManager: sql query global quota was set to 1267571097. Current memory tracking parameters: [qryQuota=0,globalQuota=1267571097,offloadingEnabled=false]
21/03/25 17:32:27 INFO QueryMemoryManager: sql query memory quota was set to 0. Current memory tracking parameters: [qryQuota=0,offloadingEnabled=false]
21/03/25 17:32:27 INFO ClientListenerProcessor: Client connector processor has started on TCP port 10800
21/03/25 17:32:27 INFO GridRestProcessor: REST protocols do not start on client node. To start the protocols on client node set '-DIGNITE_REST_START_ON_CLIENT=true' system property.
21/03/25 17:32:28 INFO IgniteKernal: Non-loopback local IPs: 10.150.81.252,fe80:0:0:0:10bf:75ff:fe95:1e9%eth0
21/03/25 17:32:28 INFO IgniteKernal: Enabled local MACs: 12BF759501E9
21/03/25 17:32:28 INFO ClusterProcessor: Cluster ID and tag has been read from Metastorage: null
21/03/25 17:32:28 INFO IgniteClusterImpl: Shutdown policy was updated [oldVal=null,newVal=null]
21/03/25 17:32:28 WARN TcpdiscoveryS3IpFinder: Amazon client configuration is not set (will use default).
21/03/25 17:32:28 INFO ResourceProfile: Default ResourceProfile created,executor resources: Map(cores -> name: cores,amount: 4,script:,vendor:,memory -> name: memory,amount: 4743,vendor: ),task resources: Map(cpus -> name: cpus,amount: 1.0)
21/03/25 17:32:28 ERROR IgniteKernal: Failed to start manager: GridManagerAdapter [enabled=true,name=o.a.i.i.managers.discovery.GriddiscoveryManager]
class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpdiscoverySpi [addrRslvr=null,sockTimeout=5000,ackTimeout=5000,marsh=JdkMarshaller [clsFilter=org.apache.ignite.marshaller.MarshallerUtils$1@f996445],skipAddrsRandomization=false]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:280)
    at org.apache.ignite.internal.managers.discovery.GriddiscoveryManager.start(GriddiscoveryManager.java:986)
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1931)
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1284)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2108)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1771)
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1150)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:670)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:611)
    at org.apache.ignite.Ignition.getorStart(Ignition.java:410)
    at un.api.aws.client.IgniteCommunicatorPerson$.connectToIgniteClient(IgniteCommunicatorPerson.scala:158)
    at un.api.aws.client.IgniteCommunicatorPerson$.loadPersonDataToIgniteCache(IgniteCommunicatorPerson.scala:63)
    at un.api.aws.client.IgniteCommunicatorPerson$.connectToIgniteClusterandLoadData(IgniteCommunicatorPerson.scala:37)
    at un.api.aws.client.IgniteCommunicatorPerson$.main(IgniteCommunicatorPerson.scala:21)
    at un.api.aws.client.IgniteCommunicatorPerson.main(IgniteCommunicatorPerson.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:936)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1015)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1024)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@70dea4e
    at org.apache.ignite.spi.discovery.tcp.TcpdiscoverySpi.checkFailedError(TcpdiscoverySpi.java:2019)
    at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1996)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
    at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:315)
    at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:57)
21/03/25 17:32:28 ERROR IgniteKernal: Got exception while starting (will rollback startup routine).
class org.apache.ignite.IgniteCheckedException: Failed to start manager: GridManagerAdapter [enabled=true,name=org.apache.ignite.internal.managers.discovery.GriddiscoveryManager]
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1936)
    at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:1284)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2108)
    at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1771)
    at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1150)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:670)
    at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:611)
    at org.apache.ignite.Ignition.getorStart(Ignition.java:410)
    at un.api.aws.client.IgniteCommunicatorPerson$.connectToIgniteClient(IgniteCommunicatorPerson.scala:158)
    at un.api.aws.client.IgniteCommunicatorPerson$.loadPersonDataToIgniteCache(IgniteCommunicatorPerson.scala:63)
    at un.api.aws.client.IgniteCommunicatorPerson$.connectToIgniteClusterandLoadData(IgniteCommunicatorPerson.scala:37)
    at un.api.aws.client.IgniteCommunicatorPerson$.main(IgniteCommunicatorPerson.scala:21)
    at un.api.aws.client.IgniteCommunicatorPerson.main(IgniteCommunicatorPerson.scala)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:936)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1015)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1024)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: class org.apache.ignite.IgniteCheckedException: Failed to start SPI: TcpdiscoverySpi [addrRslvr=null,skipAddrsRandomization=false]
    at org.apache.ignite.internal.managers.GridManagerAdapter.startSpi(GridManagerAdapter.java:280)
    at org.apache.ignite.internal.managers.discovery.GriddiscoveryManager.start(GriddiscoveryManager.java:986)
    at org.apache.ignite.internal.IgniteKernal.startManager(IgniteKernal.java:1931)
    ... 24 more
Caused by: class org.apache.ignite.spi.IgniteSpiException: Failed to deserialize object with given class loader: sun.misc.Launcher$AppClassLoader@70dea4e
    at org.apache.ignite.spi.discovery.tcp.TcpdiscoverySpi.checkFailedError(TcpdiscoverySpi.java:2019)
    at org.apache.ignite.spi.discovery.tcp.ClientImpl$MessageWorker.body(ClientImpl.java:1996)
    at org.apache.ignite.internal.util.worker.GridWorker.run(GridWorker.java:119)
    at org.apache.ignite.spi.discovery.tcp.ClientImpl$1.body(ClientImpl.java:315)
    at org.apache.ignite.spi.IgniteSpiThread.run(IgniteSpiThread.java:57)
21/03/25 17:32:28 INFO IgniteKernal:

任何解决问题的建议将不胜感激。我在 EMR spark 作业中使用了 scala-version#2.12.12、spark-version#3.0.1 和 OpenJDK-8(也适用于 EC2 实例)。

谢谢, 灿丹

版权声明:本文内容由互联网用户自发贡献,该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务,不拥有所有权,不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容, 请发送邮件至 dio@foxmail.com 举报,一经查实,本站将立刻删除。

相关推荐


Selenium Web驱动程序和Java。元素在(x,y)点处不可单击。其他元素将获得点击?
Python-如何使用点“。” 访问字典成员?
Java 字符串是不可变的。到底是什么意思?
Java中的“ final”关键字如何工作?(我仍然可以修改对象。)
“loop:”在Java代码中。这是什么,为什么要编译?
java.lang.ClassNotFoundException:sun.jdbc.odbc.JdbcOdbcDriver发生异常。为什么?
这是用Java进行XML解析的最佳库。
Java的PriorityQueue的内置迭代器不会以任何特定顺序遍历数据结构。为什么?
如何在Java中聆听按键时移动图像。
Java“Program to an interface”。这是什么意思?