kafka 集群consumer 收不到消息

芣杺~汸蔠 发表于: 2019-09-20   最后更新时间: 2019-09-21 12:58:57   3,374 游览

搭建了 3个zookeeper 伪分布式集群,kafka 也是三个broker,现在查看 _consumer_offset 是这样的,消费者收不到消息:

 Topic:__consumer_offsets    PartitionCount:50    ReplicationFactor:1    Configs:segment.bytes=104857600,cleanup.policy=compact,compression.type=producer
    Topic: __consumer_offsets    Partition: 0    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 1    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 2    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 3    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 4    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 5    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 6    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 7    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 8    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 9    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 10    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 11    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 12    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 13    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 14    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 15    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 16    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 17    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 18    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 19    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 20    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 21    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 22    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 23    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 24    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 25    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 26    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 27    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 28    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 29    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 30    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 31    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 32    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 33    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 34    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 35    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 36    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 37    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 38    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 39    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 40    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 41    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 42    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 43    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 44    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 45    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 46    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 47    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 48    Leader: 0    Replicas: 0    Isr: 0
    Topic: __consumer_offsets    Partition: 49    Leader: 0    Replicas: 0    Isr: 0
发表于 2019-09-20
添加评论

看这个是正常的,您可以先通过命令同时生产和消费,确认是否正常消费。
可参考:https://www.orchome.com/454

另外,如果还不行,得详细描述下你的操作步骤,不然我没办法帮你分析。

但是我明明搭建的是3个kafka的集群,但isr副本为什么只有一个,replica也是一个,不是会有另外两个kafka吗
现在我看只是broker=0的kafka服务起作用
另外我关掉一台kafak服务后,kafka消费端一直报服务不能用

另外我开起了kafka的SASL/PLIAN鉴权,单机可以正常消费
集群就是收不到消息,查看了consumer_offset.
希望您能给看看,是什么原因

1、3个kafka集群,但你需要在创建主题的时候,指定副本的数量,默认是1个。(你可以通过上面命令大全增加副本数)
2、__consumer_offsets 这个主题一定要扩展,这个是存储offset的,不扩展,0宕机消费者就全不能用了。

但现在是消费者收不到消息,跟这个有关吗?

没有,副本数是耐用性的保障,如果你3个集群都正常,是没有问题的。

那可能是哪儿出现问题了呢?单个就可以收到消息的

朋友,你这个问题描述太少了,我都不知道你的具体操作步骤,我没办法帮你分析额。

换个方式问吧!如果我想搭建kafka集群,相应的搭建 zookeeper 集群 来管理这个kafka 集群,则在配置过程中哪一项参数,保证了副本的耐用性,并且保证kafka集群的正常通信

zk集群5个节点,kafka集群3个以上。
topic分区的副本数 = 耐用性。

你的答案

查看kafka相关的其他问题或提一个您自己的问题