Kafka Streams EOS 模式 - 通知关闭

分类：编程问答

如何解决Kafka Streams EOS 模式 - 通知关闭

我有一个 Kafka Streams 应用程序，即使在调试级别，它也没有任何适当的日志记录就关闭了 -

2020-12-18 14:25:36:875 +0000 [Thread-7] INFO  o.apache.kafka.streams.KafkaStreams:? - stream-client [trinity-client-pandprat-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c] State transition from REBALANCING to PENDING_SHUTDOWN
    2020-12-18 14:25:36:973 +0000 [kafka-streams-close-thread] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [trinity-client-pandprat-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1] Informed to shut down
    2020-12-18 14:25:36:974 +0000 [kafka-streams-close-thread] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [trinity-client-pandprat-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1] State transition from STARTING to PENDING_SHUTDOWN
    2020-12-18 14:25:36:974 +0000 [XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1] DEBUG org.apache.kafka.clients.Metadata:? - [Consumer clientId=XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1-consumer,groupId=XXXXXX-estestes5-null] Updating last seen epoch from 0 to 0 for partition input-event-stream-client-pandprat-estestes5-0
    2020-12-18 14:25:37:075 +0000 [XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1] DEBUG org.apache.kafka.clients.Metadata:? - [Consumer clientId=XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1-consumer,groupId=XXXXXX-estestes5-null] Updated cluster metadata updateVersion 3 to MetadataCache{cluster=Cluster(id = ibD7yxLZQQSg24kQTlFnZA,nodes = [b-9.XXXXX.ap-southeast-1.amazonaws.com:9092 (id: 9 rack: apse1-az3),b-7.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 7 rack: apse1-az1),b-8.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 8 rack: apse1-az2),b-5.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 5 rack: apse1-az3),b-4.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 4 rack: apse1-az2),b-6.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 6 rack: apse1-az1),b-1.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 1 rack: apse1-az3),b-3.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 3 rack: apse1-az2),b-2.XXXX.ap-southeast-1.amazonaws.com:9092 (id: 2 rack: apse1-az1)],partitions = [Partition(topic = input-event-stream-client-pandprat-estestes5,partition = 0,leader = 9,replicas = [9,2,3],isr = [9,offlineReplicas = [])],controller = b-3.XXXXXX-msk-temp.k1lph1.c2.kafka.ap-southeast-1.amazonaws.com:9092 (id: 3 rack: apse1-az2))}
    2020-12-18 14:25:37:172 +0000 [XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1] INFO  o.a.k.c.c.i.ConsumerCoordinator:? - [Consumer clientId=XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1-consumer,groupId=XXXXXX-estestes5-null] Revoking previously assigned partitions []
    2020-12-18 14:25:37:172 +0000 [kafka-coordinator-heartbeat-thread | XXXXXX-estestes5-null] DEBUG o.a.k.c.c.i.AbstractCoordinator:? - [Consumer clientId=XXXXXX-estestes5-null-b9346744-6bb4-464d-aeaa-9311ab16ce2c-StreamThread-1-consumer,groupId=XXXXXX-estestes5-null] Heartbeat thread started

Kafka 版本 - 2.3.1 代理版本 - 2.2.1 没有抛出异常。 还可以看到类似的场景，其中应用程序也从 RUNNING 移动到 PENDING_SHUTDOWN。请参阅下面的日志 -

Jan 7,2021 @ 15:03:12.253  2021-01-07 09:33:12:252 +0000 [kafka-streams-close-thread] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Informed to shut down

    Jan 7,2021 @ 15:03:12.253  2021-01-07 09:33:12:252 +0000 [kafka-streams-close-thread] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] State transition from RUNNING to PENDING_SHUTDOWN

    Jan 7,2021 @ 15:03:06.252  2021-01-07 09:33:06:252 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] State transition from PARTITIONS_ASSIGNED to RUNNING

    Jan 7,2021 @ 15:03:06.157  2021-01-07 09:33:06:157 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  c.a.t.e.EventSequenceProcessor:? - EventSequencer Processor getting initialized with bufferFlushInterval : 100,maxBufferSize : 10000,useExternalKnowledgeTime : true,forwardingLimit: 6000,forwardingIntervalInMillis: 6000

    Jan 7,2021 @ 15:03:06.155  2021-01-07 09:33:06:154 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.AssignedStreamsTasks:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Transitioning stream task 0_0 to running

    Jan 7,2021 @ 15:03:05.183  2021-01-07 09:33:05:183 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.AssignedStreamsTasks:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Transitioning stream task 0_0 to restoring

    Jan 7,2021 @ 15:03:05.180  2021-01-07 09:33:05:179 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.ProcessorStateManager:? - task [0_0] Registering state store event-seq-state-store to its state manager

    Jan 7,2021 @ 15:03:05.168  2021-01-07 09:33:05:167 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.AssignedStreamsTasks:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Initializing stream tasks [0_0]

    Jan 7,2021 @ 15:03:05.163  2021-01-07 09:33:05:162 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] partition assignment took 604 ms.

    Jan 7,2021 @ 15:03:05.163      current active tasks: [0_0]

    Jan 7,2021 @ 15:03:05.163      previous active tasks: []

    Jan 7,2021 @ 15:03:05.163      current standby tasks: []

    Jan 7,2021 @ 15:03:05.163  
    Jan 7,2021 @ 15:03:04.953  2021-01-07 09:33:04:952 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.ProcessorStateManager:? - task [0_0] Register global stores []

    Jan 7,2021 @ 15:03:04.653  2021-01-07 09:33:04:652 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.ProcessorStateManager:? - task [0_0] Created state store manager for task 0_0 with the acquired state dir lock

    Jan 7,2021 @ 15:03:04.653  2021-01-07 09:33:04:653 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Creating producer client for task 0_0

    Jan 7,2021 @ 15:03:04.559  2021-01-07 09:33:04:558 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] at state PARTITIONS_REVOKED: partitions [input-XXXXXXXX-tf-1test0107-0] assigned at the end of consumer rebalance.

    Jan 7,2021 @ 15:03:04.559      current suspended active tasks: []

    Jan 7,2021 @ 15:03:04.559      current suspended standby tasks: []

    Jan 7,2021 @ 15:03:04.559  2021-01-07 09:33:04:558 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] State transition from PARTITIONS_REVOKED to PARTITIONS_ASSIGNED

    Jan 7,2021 @ 15:03:04.559  2021-01-07 09:33:04:558 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.internals.TaskManager:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Adding assigned tasks as active: {0_0=[input-XXXXXXXX-tf-1test0107-0]}

    Jan 7,2021 @ 15:03:04.559  
    Jan 7,2021 @ 15:03:04.559  2021-01-07 09:33:04:558 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Creating tasks based on assignment.

    Jan 7,2021 @ 15:03:04.386  2021-01-07 09:33:04:386 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Assigned tasks to clients as {60f8a4be-b576-4ed0-9615-b91021cd76e0=[activeTasks: ([0_0]) standbyTasks: ([]) assignedTasks: ([0_0]) prevActiveTasks: ([]) prevStandbyTasks: ([]) prevAssignedTasks: ([]) capacity: 1]}.

    Jan 7,2021 @ 15:03:04.385  2021-01-07 09:33:04:385 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Assigning tasks [0_0] to clients {60f8a4be-b576-4ed0-9615-b91021cd76e0=[activeTasks: ([]) standbyTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) prevAssignedTasks: ([]) capacity: 1]} with number of replicas 0

    Jan 7,2021 @ 15:03:04.385  2021-01-07 09:33:04:384 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Completed validating internal topics {XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog=InternalTopicMetadata(config=UnwindowedChangelogTopicConfig(name=XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog,topicConfigs={}),numPartitions=1)} in partition assignor.

    Jan 7,2021 @ 15:03:04.385  2021-01-07 09:33:04:384 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Created state changelog topics [InternalTopicMetadata(config=UnwindowedChangelogTopicConfig(name=XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog,numPartitions=1)] from the parsed topology.

    Jan 7,2021 @ 15:03:04.058  2021-01-07 09:33:04:058 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.InternalTopicManager:? - stream-thread [main] Topic XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog is unknown or not found,hence not existed yet.

    Jan 7,2021 @ 15:03:04.058  2021-01-07 09:33:04:058 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.InternalTopicManager:? - stream-thread [main] Going to create topic XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog with 1 partitions and config {cleanup.policy=compact}.

    Jan 7,2021 @ 15:03:04.049  2021-01-07 09:33:03:955 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Starting to validate internal topics {XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog=InternalTopicMetadata(config=UnwindowedChangelogTopicConfig(name=XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog,2021 @ 15:03:04.049  2021-01-07 09:33:03:955 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.InternalTopicManager:? - stream-thread [main] Trying to check if topics [XXXXXXX-tf-1test0107-null-event-seq-state-store-changelog] have been created with expected number of partitions.

    Jan 7,2021 @ 15:03:03.955  2021-01-07 09:33:03:954 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Starting to validate internal topics {} in partition assignor.

    Jan 7,2021 @ 15:03:03.955  2021-01-07 09:33:03:954 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Completed validating internal topics {} in partition assignor.

    Jan 7,2021 @ 15:03:03.955  2021-01-07 09:33:03:955 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Created repartition topics [] from the parsed topology.

    Jan 7,2021 @ 15:03:03.953  2021-01-07 09:33:03:953 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Constructed client metadata {60f8a4be-b576-4ed0-9615-b91021cd76e0=ClientMetadata{hostInfo=null,consumers=[XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer-e7ce11d1-7b5a-478f-8aaa-1e46df67bbf3],state=[activeTasks: ([]) standbyTasks: ([]) assignedTasks: ([]) prevActiveTasks: ([]) prevStandbyTasks: ([]) prevAssignedTasks: ([]) capacity: 1]}} from the member subscriptions.

    Jan 7,2021 @ 15:03:03.952  2021-01-07 09:33:03:952 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.StreamsPartitionAssignor:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer] Current minimum supported version remains at 4,last seen supported version was 4

    Jan 7,2021 @ 15:03:00.355  2021-01-07 09:33:00:354 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.InternalTopologyBuilder:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] found [input-XXXXXXXX-tf-1test0107] topics possibly matching regex

    Jan 7,2021 @ 15:03:00.355  2021-01-07 09:33:00:354 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.i.InternalTopologyBuilder:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] updating builder with SubscriptionUpdates{updatedTopicSubscriptions=[input-XXXXXXXX-tf-1test0107]} topic(s) with possible matching regex subscription(s)

    Jan 7,2021 @ 15:03:00.353      current assigned active tasks: []

    Jan 7,2021 @ 15:03:00.353  2021-01-07 09:33:00:352 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] State transition from STARTING to PARTITIONS_REVOKED

    Jan 7,2021 @ 15:03:00.353  2021-01-07 09:33:00:352 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.internals.TaskManager:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Suspending all active tasks [] and standby tasks []

    Jan 7,2021 @ 15:03:00.353  2021-01-07 09:33:00:352 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] at state STARTING: partitions [] revoked at the beginning of consumer rebalance.

    Jan 7,2021 @ 15:03:00.353      suspended standby tasks: []

    Jan 7,2021 @ 15:03:00.353      current assigned standby tasks: []

    Jan 7,2021 @ 15:03:00.353  
    Jan 7,2021 @ 15:03:00.353  2021-01-07 09:33:00:353 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] partition revocation took 1 ms.

    Jan 7,2021 @ 15:03:00.353      suspended active tasks: []

    Jan 7,2021 @ 15:02:58.752  Event Sequencer Server started,listening on 2301

    Jan 7,2021 @ 15:02:57.151  2021-01-07 09:32:57:150 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] State transition from CREATED to STARTING

    Jan 7,2021 @ 15:02:57.151  2021-01-07 09:32:57:150 +0000 [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Starting

    Jan 7,2021 @ 15:02:57.151  2021-01-07 09:32:57:151 +0000 [main] INFO  c.a.t.c.IngestionTopicConsumer:? - StreamThread Metadata : ThreadMetadata{threadName=XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1,threadState=STARTING,activeTasks=[],standbyTasks=[],consumerClientId=XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-consumer,restoreConsumerClientId=XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1-restore-consumer,producerClientIds=[],adminClientId=XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-admin} 

    Jan 7,2021 @ 15:02:57.051  2021-01-07 09:32:57:050 +0000 [main] WARN  o.a.k.c.consumer.ConsumerConfig:? - The configuration 'admin.retry.backoff.ms' was supplied but isn't a known config.

    Jan 7,2021 @ 15:02:57.051  2021-01-07 09:32:57:050 +0000 [main] WARN  o.a.k.c.consumer.ConsumerConfig:? - The configuration 'admin.retries' was supplied but isn't a known config.

    Jan 7,2021 @ 15:02:56.760  2021-01-07 09:32:56:760 +0000 [main] DEBUG o.a.k.s.p.i.InternalTopicManager:? - stream-thread [main] Configs:

    Jan 7,2021 @ 15:02:56.760  
    Jan 7,2021 @ 15:02:56.753  2021-01-07 09:32:56:752 +0000 [main] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Creating consumer client

    Jan 7,2021 @ 15:02:55.857  2021-01-07 09:32:55:856 +0000 [main] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tf-1test0107-null-60f8a4be-b576-4ed0-9615-b91021cd76e0-StreamThread-1] Creating restore consumer client

    Jan 7,2021 @ 15:02:55.350  2021-01-07 09:32:55:349 +0000 [main] INFO  c.a.t.c.IngestionTopicConsumer:? - Initializing Ingestion Topic Consumer

    Jan 7,2021 @ 15:02:55.350  2021-01-07 09:32:55:349 +0000 [main] INFO  com.arcesium.trinity.EventSequencer:? - Initializing Ingestion Topic Consumer

    Jan 7,2021 @ 15:02:53.551      Source: input-topic (topics: [input-XXXXXXXX-tf-1test0107])

    Jan 7,2021 @ 15:02:53.551      Processor: event-sequencer (stores: [event-seq-state-store])

    Jan 7,2021 @ 15:02:53.551      Sink: output-event-topic (topic: output-XXXXXXXX-tf-1test0107)

    Jan 7,2021 @ 15:02:53.551        <-- event-sequencer

    Jan 7,2021 @ 15:02:53.551  
    Jan 7,2021 @ 15:02:53.551        <-- input-topic

    Jan 7,2021 @ 15:02:53.551  2021-01-07 09:32:53:460 +0000 [main] INFO  c.a.t.config.EventSequencerConfig:? - Topology initialized: Topologies:

    Jan 7,2021 @ 15:02:53.551     Sub-topology: 0

    Jan 7,2021 @ 15:02:53.551        --> event-sequencer

    Jan 7,2021 @ 15:02:53.551        --> output-event-topic

    Jan 7,2021 @ 15:02:30.854  Listening for transport dt_socket at address: 2311

我还看到每当触发重新平衡时都会发生关机。请参阅下面的日志 -

Jan 7,2021 @ 08:43:13.555  2021-01-07 03:13:13:554 +0000 [kafka-streams-close-thread] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] Informed to shut down

    Jan 7,2021 @ 08:43:13.555  2021-01-07 03:13:13:554 +0000 [kafka-streams-close-thread] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] State transition from PARTITIONS_REVOKED to PENDING_SHUTDOWN

    Jan 7,2021 @ 08:42:50.009  2021-01-07 03:12:50:009 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask:? - task [0_0] Committing

    Jan 7,2021 @ 08:42:50.009  2021-01-07 03:12:50:009 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] DEBUG o.a.k.s.p.i.RecordCollectorImpl:? - task [0_0] Flushing producer

    Jan 7,2021 @ 08:42:50.009  2021-01-07 03:12:50:009 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] DEBUG o.a.k.s.p.i.ProcessorStateManager:? - task [0_0] Flushing all stores registered in the state manager

    Jan 7,2021 @ 08:42:50.008      current assigned active tasks: [0_0]

    Jan 7,2021 @ 08:42:50.008  2021-01-07 03:12:50:008 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] DEBUG o.a.k.s.p.internals.TaskManager:? - stream-thread [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] Suspending all active tasks [0_0] and standby tasks []

    Jan 7,2021 @ 08:42:50.008  2021-01-07 03:12:50:008 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamTask:? - task [0_0] Suspending

    Jan 7,2021 @ 08:42:50.008      current assigned standby tasks: []

    Jan 7,2021 @ 08:42:50.008  2021-01-07 03:12:50:008 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] INFO  o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] State transition from RUNNING to PARTITIONS_REVOKED

    Jan 7,2021 @ 08:42:50.008  2021-01-07 03:12:50:008 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] DEBUG o.a.k.s.p.internals.StreamThread:? - stream-thread [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] at state RUNNING: partitions [input-XXXXXXX-tzobnwpj-0] revoked at the beginning of consumer rebalance.

    Jan 7,2021 @ 08:42:50.008  
    Jan 7,2021 @ 08:42:49.842  2021-01-07 03:12:49:842 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] WARN  o.a.k.c.consumer.internals.Fetcher:? - [Consumer clientId=XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1-consumer,groupId=XXXXXXX-tzobnwpj-null] Received unknown topic or partition error in fetch for partition input-XXXXXXX-tzobnwpj-0

    Jan 7,2021 @ 08:42:49.652  2021-01-07 03:12:49:652 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] WARN  o.a.k.c.consumer.internals.Fetcher:? - [Consumer clientId=XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1-consumer,2021 @ 08:42:49.651  2021-01-07 03:12:49:650 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] WARN  o.a.k.c.consumer.internals.Fetcher:? - [Consumer clientId=XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1-consumer,2021 @ 08:42:49.649  2021-01-07 03:12:49:649 +0000 [XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1] WARN  o.a.k.c.consumer.internals.Fetcher:? - [Consumer clientId=XXXXXXX-tzobnwpj-null-adce3dfa-92cb-4f33-a806-badadffaadc7-StreamThread-1-consumer,groupId=XXXXXXX-tzobnwpj-null] Received unknown topic or partition error in fetch for partition input-XXXXXXX-tzobnwpj-0

有人知道为什么会发生这种情况吗？

解决方法

日志行 "Informed to shut down" 表明调用了 shutdown 的 StreamThread 方法。 这只能从 2 个地方调用：-

One - KafkaStream close 方法 - 用于实际完全关闭 Kafka 流（最终关闭所有 StreamThreads） 但是您的调试日志并不表明完整的 Kafka 流正在关闭。如果是这种情况，您的日志中会出现以下情况

 log.debug("Stopping Streams client with timeoutMillis = {} ms.",timeoutMs);

两个 - RebalanceListener - onPartitionsAssigned 方法

if (streamThread.assignmentErrorCode.get() == StreamsPartitionAssignor.Error.INCOMPLETE_SOURCE_TOPIC_METADATA.code()) {
                log.error("Received error code {} - shutdown",streamThread.assignmentErrorCode.get());
                streamThread.shutdown();
                return;
            }

这可能意味着由于 INCOMPLETE_SOURCE_TOPIC_METADATA，您的 StreamThread 正在接收关闭请求。 这也可能是暂时性问题，也可能是由于元数据不完整（例如主题名称不存在或拼写错误等）而导致的永久性故障

版权声明：本文内容由互联网用户自发贡献，该文观点与技术仅代表作者本人。本站仅提供信息存储空间服务，不拥有所有权，不承担相关法律责任。如发现本站有涉嫌侵权/违法违规的内容， 请发送邮件至 dio@foxmail.com 举报，一经查实，本站将立刻删除。

相关推荐

matplotlib报错：AttributeError: module 'backend_interagg' has no attribute 'FigureCanvas'. Did you mean: 'FigureCanvasAgg'?
使用本地python环境可以成功执行 import pandas as pd import matplotlib.pyplot as plt # 设置字体 plt.rcParams[&#39;font.sans-serif&#39;] = [&#39;SimHei&#39;] # 能正确显示负号 p
gitlab登录失败，报错：This challenge page was accidentally cached by an intermediary and is no longer available.
设置时间 控制面板
后端开发常见错误
错误1：Request method ‘DELETE‘ not supported 错误还原：controller层有一个接口，访问该接口时报错：Request method ‘DELETE‘ not supported 错误原因：没有接收到前端传入的参数，修改为如下 参考 错误2：cannot r
docker常见错误
错误1：启动docker镜像时报错：Error response from daemon: driver failed programming external connectivity on endpoint quirky_allen 解决方法：重启docker -&gt; systemctl r
idea常见错误
错误1：private field ‘xxx‘ is never assigned 按Altʾnter快捷键，选择第2项 参考：https://blog.csdn.net/shi_hong_fei_hei/article/details/88814070 错误2：启动时报错，不能找到主启动类 #
pip安装依赖失败
报错如下，通过源不能下载，最后警告pip需升级版本 Requirement already satisfied: pip in c:\users\ychen\appdata\local\programs\python\python310\lib\site-packages (22.0.4) Coll
maven常见错误
错误1：maven打包报错 错误还原：使用maven打包项目时报错如下 [ERROR] Failed to execute goal org.apache.maven.plugins:maven-resources-plugin:3.2.0:resources (default-resources)
cloud项目中常见错误
错误1：服务调用时报错 服务消费者模块assess通过openFeign调用服务提供者模块hires 如下为服务提供者模块hires的控制层接口 @RestController @RequestMapping(&quot;/hires&quot;) public class FeignControl
springboot常见错误
错误1：运行项目后报如下错误 解决方案 报错2：Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project sb 解决方案：在pom.
springmvc拦截器中使用redisTemplate报空指针异常
参考 错误原因 过滤器或拦截器在生效时，redisTemplate还没有注入 解决方案：在注入容器时就生效 @Component //项目运行时就注入Spring容器 public class RedisBean { @Resource private RedisTemplate&lt;String
vite报错：npm ERR! command C:WINDOWSsystem32cmd.exe /d /s /c C:UsersychenAppDataLocalTempnpx-dde1c848.cmd
使用vite构建项目报错 C:\Users\ychen\work&gt;npm init @vitejs/app @vitejs/create-app is deprecated, use npm init vite instead C:\Users\ychen\AppData\Local\npm-
VM VirtualBox中安装centos时报错：设置基础软件仓库时出错
参考1 参考2 解决方案 # 点击安装源 协议选择 http:// 路径填写 mirrors.aliyun.com/centos/8.3.2011/BaseOS/x86_64/os URL类型 软件库URL 其他路径 # 版本 7 mirrors.aliyun.com/centos/7/os/x86
kafka常见问题
报错1 [root@slave1 data_mocker]# kafka-console-consumer.sh --bootstrap-server slave1:9092 --topic topic_db [2023-12-19 18:31:12,770] WARN [Consumer clie
hive常见问题
错误1 # 重写数据 hive (edu)&gt; insert overwrite table dwd_trade_cart_add_inc &gt; select data.id, &gt; data.user_id, &gt; data.course_id, &gt; date_format(
装载数据时报错：Failed to execute spark task, with exception 'org.apache.hadoop.hive.ql.metadata.HiveException(Failed to create spark client.)'
错误1 hive (edu)&gt; insert into huanhuan values(1,&#39;haoge&#39;); Query ID = root_20240110071417_fe1517ad-3607-41f4-bdcf-d00b98ac443e Total jobs = 1
flume常见问题
报错1：执行到如下就不执行了，没有显示Successfully registered new MBean. [root@slave1 bin]# /usr/local/software/flume-1.9.0/bin/flume-ng agent -n a1 -c /usr/local/softwa
hadoop集群常见错误
虚拟及没有启动任何服务器查看jps会显示jps，如果没有显示任何东西 [root@slave2 ~]# jps 9647 Jps 解决方案 # 进入/tmp查看 [root@slave1 dfs]# cd /tmp [root@slave1 tmp]# ll 总用量 48 drwxr-xr-x. 2
hive常见错误
报错1 hive&gt; show databases; OK Failed with exception java.io.IOException:java.lang.RuntimeException: Error in configuring object Time taken: 0.474 se
Linux常见错误
报错1 [root@localhost ~]# vim -bash: vim: 未找到命令 安装vim yum -y install vim* # 查看是否安装成功 [root@hadoop01 hadoop]# rpm -qa |grep vim vim-X11-7.4.629-8.el7_9.x
使用sqoop一直卡在：mapreduce.Job: Running job: job_1703173956074_0001
修改hadoop配置 vi /usr/local/software/hadoop-2.9.2/etc/hadoop/yarn-site.xml # 添加如下 &lt;configuration&gt; &lt;property&gt; &lt;name&gt;yarn.nodemanager.res
pythonJavaScriptjavaHTMLPHPreactjsC#AndroidCSSNode.jssqlrpython-3.xMysqLjQueryc++pandasFlutterangularIOSdjangolinuxswifttypescript路由器JSON路由器设置无线路由器h3c华三华三路由器设置华三路由器电脑软件教程arraysdocker软件图文教程Cvue.jslaravelspring-boot