kafka拥有不同副本数量的集群,应该如何升级?

開開新新 发表于: 2020-11-19   最后更新时间: 2020-11-19  

我有3个broker,原先所有topic大部分副本因子都是1

后来我设置了某几个topic的副本因子为3,昨晚滚动升级了broker集群,因为新版的配置文件写了min.insync.replicas:3,然后server.log日志里就不断出现以下错误。

[2020-11-18 21:23:46,095] ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-7 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-7
[2020-11-18 21:23:46,095] INFO [GroupCoordinator 2]: Preparing to rebalance group node-red-rdkafka in state PreparingRebalance with old generation 260 (__consumer_offsets-7) (reason: error when storing group assignment during SyncGroup (member: iiot-logstash01-0-c29833bf-57b0-41f5-bebe-b355c76131d9)) (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,109] INFO [GroupCoordinator 2]: Stabilized group ingress-01 generation 368 (__consumer_offsets-43) (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,112] INFO [GroupCoordinator 2]: Assignment received from leader for group ingress-01 for generation 368 (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,112] ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-43 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-43
[2020-11-18 21:23:46,112] INFO [GroupCoordinator 2]: Preparing to rebalance group ingress-01 in state PreparingRebalance with old generation 368 (__consumer_offsets-43) (reason: error when storing group assignment during SyncGroup (member: ingress-01-0-4eb86627-d15c-4e35-b8ff-1f87a139a309)) (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,156] INFO [GroupCoordinator 2]: Assignment received from leader for group kafka-nginx-error-dev for generation 231 (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,157] ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-28 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-28

我关闭kafka,然后每个broker设置min.insync.replicas:1

日志显示开始删除多余的副本

[2020-11-18 21:31:47,437] INFO Deleted log /home/kfkdata/abp_auditlogs-1/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,442] INFO Deleted offset index /home/kfkdata/abp_auditlogs-1/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,443] INFO Deleted time index /home/kfkdata/abp_auditlogs-1/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,664] INFO [Log partition=__consumer_offsets-7, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=0, lastModifiedTime=1605699398000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:31:47,664] INFO Deleted log /home/kfkdata/__consumer_offsets-7/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,665] INFO [Log partition=__consumer_offsets-49, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=0, lastModifiedTime=1605699398000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:31:47,666] INFO Deleted offset index /home/kfkdata/__consumer_offsets-7/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,666] INFO Deleted log /home/kfkdata/__consumer_offsets-49/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,666] INFO Deleted time index /home/kfkdata/__consumer_offsets-7/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,667] INFO Deleted offset index /home/kfkdata/__consumer_offsets-49/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,667] INFO Deleted time index /home/kfkdata/__consumer_offsets-49/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,719] INFO [Log partition=__consumer_offsets-43, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=0, lastModifiedTime=1605699398000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:31:47,719] INFO Deleted log /home/kfkdata/__consumer_offsets-43/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,720] INFO Deleted offset index /home/kfkdata/__consumer_offsets-43/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,721] INFO Deleted time index /home/kfkdata/__consumer_offsets-43/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,255] INFO [Log partition=__consumer_offsets-22, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=41781, lastModifiedTime=1605705201000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:32:01,256] INFO Deleted log /home/kfkdata/__consumer_offsets-22/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,256] INFO Deleted offset index /home/kfkdata/__consumer_offsets-22/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,256] INFO Deleted time index /home/kfkdata/__consumer_offsets-22/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,316] INFO [Log partition=__consumer_offsets-28, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=56549, lastModifiedTime=1605705201000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:32:01,316] INFO Deleted log /home/kfkdata/__consumer_offsets-28/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,316] INFO Deleted offset index /home/kfkdata/__consumer_offsets-28/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,317] INFO Deleted time index /home/kfkdata/__consumer_offsets-28/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)

对于集群内拥有副本数量不同的节点,应该怎么做升级?



您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: kafka讨论下最佳配置
下一条: kafka报 1 partitions have leader brokers without a matching listener

  • 配置要保持一致的。