producer 不能正常生产数据到kafka集群?

V COME AGAIN 发表于: 2019-08-08   最后更新时间: 2019-08-08  

存在60个java producer进程,同时生产数据到5个节点的kafka集群。一部分producer丢失数据,一部分producer完全生产不了数据,生产不了数据的producer有一下日志:

09:43:19.354 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initialize connection to node vm121:9092 (id: -5 rack: null) for sending metadata request
09:43:19.354 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating connection to node vm121:9092 (id: -5 rack: null)
09:43:19.370 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name node--5.bytes-sent
09:43:19.371 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name node--5.bytes-received
09:43:19.372 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.metrics.Metrics - Added sensor with name node--5.latency
09:43:19.372 [kafka-producer-network-thread | producer-1] DEBUG o.a.kafka.common.network.Selector - [Producer clientId=producer-1] Created socket with SO_RCVBUF = 32768, SO_SNDBUF = 131072, SO_TIMEOUT = 0 to node -5
09:43:19.372 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Completed connection to node -5. Fetching API versions.
09:43:19.373 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Initiating API versions fetch from node -5.
09:43:49.397 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Disconnecting from node -5 due to request timeout.
09:43:49.397 [kafka-producer-network-thread | producer-1] DEBUG o.apache.kafka.clients.NetworkClient - [Producer clientId=producer-1] Give up sending metadata request since no node is available

这个producer所在的服务器telnet 集群每个节点的9092都是可以正常连接的,这个老师能给一点排查意见吗



您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: org.apache.kafka.common.errors.NotEnoughReplicasException
下一条: java 操作kafka acl?

  • 你上面这个看是超时呀,你先别60个一起发了,单个我都觉得会失败呀。

    • 有一部分producer是可以连接成功生产数据的 而且看这个日志 连接是Completed connection to node -5. Fetching API versions 先建立成功了,又Disconnecting from node -5 due to request timeout. 然后Give up sending metadata request since no node is available 这个topic不会出于某个原因主动去把producer的tcp连接给关闭吧

        • 老师 同步的方式 异常信息
          java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.
          at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.(KafkaProducer.java:1150)
          at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:846)
          at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:784)
          at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:671)
          at org.cytec.damon.analysis.dataEncryption.EncryptionApp.main(EncryptionApp.java:29)
          Caused by: org.apache.kafka.common.errors.TimeoutException: Failed to update metadata after 60000 ms.

            • 就是搞不懂为啥一直报超时信息 端口防火墙是放开的 集群配置监听 是这种方式 listeners=PLAINTEXT://vm144:9092 producer端也是以vm144:9092的方式访问 telnet vm144:9092是通的