kafka做了topic的rebalance,感觉影响到了集群,消费延迟

lq0317。 发表于: 2021-01-25   最后更新时间: 2021-01-25  


您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: kafka消息生产速度快于发送速度,buffer.memory会生效吗?
下一条: server.log日志中发现异常,EndOfStreamException: Unable to read additional data from client sessionid 0x17684ce651a0055, likely client has closed socket

  • 你的带宽可能被平衡给占用了。
    可以考虑限流:https://www.orchome.com/510

    • 如果不是带宽的问题,会不会是我在均衡的时候,消费者组中有一个消费者没有找到leader,然后触发了消费者rebalance,所以引起了消费延迟

        • 你是做了分区迁移是吧?
          1、已经迁移成功率吗?
          2、你查看一下该topic的描述,贴一下出来。
          3、如果还在迁移,就是之前说的,可能是这个把流量占用了。

            • {"version":1,"partitions":[{"topic":"ods_be_monitor_item_detail","partition":4,"replicas":[156,157,111]},{"topic":"ods_be_monitor_item_detail","partition":2,"replicas":[154,155,156]},{"topic":"ods_be_monitor_item_detail","partition":14,"replicas":[154,157,111]},{"topic":"ods_be_monitor_item_detail","partition":1,"replicas":[153,154,155]},{"topic":"ods_be_monitor_item_detail","partition":5,"replicas":[157,111,153]},{"topic":"ods_be_monitor_item_detail","partition":9,"replicas":[155,157,111]},{"topic":"ods_be_monitor_item_detail","partition":12,"replicas":[111,155,156]},{"topic":"ods_be_monitor_item_detail","partition":8,"replicas":[154,156,157]},{"topic":"ods_be_monitor_item_detail","partition":7,"replicas":[153,155,156]},{"topic":"ods_be_monitor_item_detail","partition":0,"replicas":[111,153,154]},{"topic":"ods_be_monitor_item_detail","partition":3,"replicas":[155,156,157]},{"topic":"ods_be_monitor_item_detail","partition":6,"replicas":[111,154,155]},{"topic":"ods_be_monitor_item_detail","partition":11,"replicas":[157,153,154]},{"topic":"ods_be_monitor_item_detail","partition":13,"replicas":[153,156,157]},{"topic":"ods_be_monitor_item_detail","partition":10,"replicas":[156,111,153]}]}

                • 迁移完成了大概花了2个小时的时间,一个分区大概45g数据,15个分区,三副本,是因为流量导致影响到集群了吗,具体什么原因能帮忙分析一下吗

                    • 比如这种,明明我有五台,然后leader都重复,现在生产消息有时候超时,设置了5s的超时时间

                      [root@prd-ic-kafka-06 kafka]# /opt/kafka_2.10-0.10.2.2/bin/kafka-topics.sh --zookeeper 172.19.39.156:2181 --topic ods_meituan_crawler_flow_analysis --describe
                      Topic:ods_meituan_crawler_flow_analysis PartitionCount:5        ReplicationFactor:3     Configs:
                              Topic: ods_meituan_crawler_flow_analysis        Partition: 0    Leader: 154     Replicas: 157,154,155   Isr: 155,154,157
                              Topic: ods_meituan_crawler_flow_analysis        Partition: 1    Leader: 155     Replicas: 153,155,156   Isr: 155,153,156
                              Topic: ods_meituan_crawler_flow_analysis        Partition: 2    Leader: 154     Replicas: 154,156,157   Isr: 154,157,156
                              Topic: ods_meituan_crawler_flow_analysis        Partition: 3    Leader: 155     Replicas: 155,157,153   Isr: 155,153,157
                              Topic: ods_meituan_crawler_flow_analysis        Partition: 4    Leader: 153     Replicas: 156,153,154   Isr: 154,153,156
                      
                        25 Jan 2021 02:25:09,183 INFO  [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:25:13,512 INFO  [PollableSourceRunner-KafkaSource-kafka74] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group kadoo
                        p_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:25:14,285 INFO  [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:25:14,285 INFO  [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:35:12,738 INFO  [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:35:12,809 INFO  [PollableSourceRunner-KafkaSource-kafka74_04] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:45:12,334 INFO  [PollableSourceRunner-KafkaSource-kafka74_01] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:45:12,378 INFO  [PollableSourceRunner-KafkaSource-kafka74] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group kadoo
                        p_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        25 Jan 2021 02:45:12,396 INFO  [PollableSourceRunner-KafkaSource-kafka74_02] (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator$OffsetCommitResponseHandler.handle:542)  - Offset commit for group ka
                        doop_ods_eleme_monitor_item_detail_external failed due to REQUEST_TIMED_OUT, will find new coordinator and retry
                        

                        这个和消费者有什么关系吗。别的topic也被影响了