producer 不能正常生产数据到kafka集群?

V COME AGAIN 发表于: 2019-08-08   最后更新时间: 2019-08-08 18:36:14   2,365 游览

存在60个java producer进程,同时生产数据到5个节点的kafka集群。一部分producer丢失数据,一部分producer完全生产不了数据,生产不了数据的producer有一下日志:

09:43:19.354 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node vm121:9092 (id: -5 rack: null) for sending metadata request
09:43:19.354 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node vm121:9092 (id: -5 rack: null)
09:43:19.370 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name node--5.bytes-sent
09:43:19.371 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name node--5.bytes-received
09:43:19.372 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name node--5.latency
09:43:19.372 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.network.Selector - [Producer clientId=producer-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -5
09:43:19.372 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Completed connection to node -5. Fetching API versions.
09:43:19.373 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating API versions fetch from node -5.
09:43:49.397 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Disconnecting from node -5 due to request timeout.
09:43:49.397 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available

这个producer所在的服务器telnet 集群每个节点的9092都是可以正常连接的,这个老师能给一点排查意见吗

发表于 2019-08-08
添加评论

你上面这个看是超时呀,你先别60个一起发了,单个我都觉得会失败呀。

V COME AGAIN -> 半兽人 4年前

有一部分producer是可以连接成功生产数据的 而且看这个日志 连接是Completed connection to node -5. Fetching API versions 先建立成功了,又Disconnecting from node -5 due to request timeout. 然后Give up sending metadata request since no node is available 这个topic不会出于某个原因主动去把producer的tcp连接给关闭吧

半兽人 -> V COME AGAIN 4年前

挨个telnet,看看broker全部通吗?

V COME AGAIN -> 半兽人 4年前

5台节点 每个都测试了 telnet 9092 都是正常的

半兽人 -> V COME AGAIN 4年前

你这都是debug日志。
你的生产者用同步的发送试试。看看报啥错。
producer.send(record).get();

V COME AGAIN -> 半兽人 4年前

老师 同步的方式 异常信息

java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
 at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:1150)
 at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:846)
 at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:784)
 at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:671)
 at org.cytec.damon.analysis.dataEncryption.EncryptionApp.main(EncryptionApp.java:29)
Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
V COME AGAIN -> 半兽人 4年前

就是搞不懂为啥一直报超时信息 端口防火墙是放开的 集群配置监听 是这种方式 listeners=PLAINTEXT://vm144:9092 producer端也是以vm144:9092的方式访问 telnet vm144:9092是通的

半兽人 -> V COME AGAIN 4年前

如果所有人都能识别 vm144,那应该是没问题的。
那你做个尝试,换成ip试试。

请问楼主问题解决了么?我也遇到了这个问题

你的答案

查看kafka相关的其他问题或提一个您自己的问题