kafka 报错Got error produce response with correlation id on topic-partition Interaction.MainPostLike_5-2, retrying. Error: NETWORK_EXCEPTION

SONG 发表于: 2019-12-18   最后更新时间: 2019-12-18  

使用kafka-connect时偶尔会报错
报错日志先是connect中两个ERROR
[ERROR] [pool-12-thread-1] connect.runtime.WorkerSourceTask - WorkerSourceTask{id=mgsource_etl_1_20191128-0} Failed to flush, timed out while waiting for producer to flush outstanding 53 messages
[ERROR] [pool-12-thread-1] connect.runtime.SourceTaskOffsetCommitter - WorkerSourceTask{id=mgsource_etl_1_20191128-0} Failed to commit offsets

然后是kafka中两条WARN
[WARN ] [kafka-producer-network-thread | producer-5] producer.internals.Sender - [Producer clientId=producer-5] Received invalid metadata error in produce request on partition Interaction.MainPostLike_6-1 due to org.apache.kafka.common.errors.NetworkException: The server disconnected before a response was received.. Going to request metadata update now
[WARN ] [kafka-producer-network-thread | producer-5] producer.internals.Sender - [Producer clientId=producer-5] Got error produce response with correlation id 1861556 on topic-partition Interaction.MainPostLike_5-2, retrying (9 attempts left). Error: NETWORK_EXCEPTION

集群没问题,各种压测也做了,很难复现,偶尔几天会有一两次报错。其实数据并没有丢,因为kafka和connect的重试机制,在报错后下次都flush出去了,找不出原因



您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: kafka connector 从源系统数据库同步数据到目的数据库,如何知道已经同步完成了
下一条: kafka手动管理offset,poll 请求拉取失败,一直轮询 ?