kafka拥有不同副本数量的集群,应该如何升级?

開開新新 发表于: 2020-11-19   最后更新时间: 2020-11-19 14:03:23   1,407 游览

我有3个broker,原先所有topic大部分副本因子都是1

后来我设置了某几个topic的副本因子为3,昨晚滚动升级了broker集群,因为新版的配置文件写了min.insync.replicas:3,然后server.log日志里就不断出现以下错误。

[2020-11-18 21:23:46,095] ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-7 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-7
[2020-11-18 21:23:46,095] INFO [GroupCoordinator 2]: Preparing to rebalance group node-red-rdkafka in state PreparingRebalance with old generation 260 (__consumer_offsets-7) (reason: error when storing group assignment during SyncGroup (member: iiot-logstash01-0-c29833bf-57b0-41f5-bebe-b355c76131d9)) (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,109] INFO [GroupCoordinator 2]: Stabilized group ingress-01 generation 368 (__consumer_offsets-43) (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,112] INFO [GroupCoordinator 2]: Assignment received from leader for group ingress-01 for generation 368 (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,112] ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-43 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-43
[2020-11-18 21:23:46,112] INFO [GroupCoordinator 2]: Preparing to rebalance group ingress-01 in state PreparingRebalance with old generation 368 (__consumer_offsets-43) (reason: error when storing group assignment during SyncGroup (member: ingress-01-0-4eb86627-d15c-4e35-b8ff-1f87a139a309)) (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,156] INFO [GroupCoordinator 2]: Assignment received from leader for group kafka-nginx-error-dev for generation 231 (kafka.coordinator.group.GroupCoordinator)
[2020-11-18 21:23:46,157] ERROR [ReplicaManager broker=2] Error processing append operation on partition __consumer_offsets-28 (kafka.server.ReplicaManager)
org.apache.kafka.common.errors.NotEnoughReplicasException: The size of the current ISR Set(2) is insufficient to satisfy the min.isr requirement of 2 for partition __consumer_offsets-28

我关闭kafka,然后每个broker设置min.insync.replicas:1

日志显示开始删除多余的副本

[2020-11-18 21:31:47,437] INFO Deleted log /home/kfkdata/abp_auditlogs-1/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,442] INFO Deleted offset index /home/kfkdata/abp_auditlogs-1/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,443] INFO Deleted time index /home/kfkdata/abp_auditlogs-1/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,664] INFO [Log partition=__consumer_offsets-7, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=0, lastModifiedTime=1605699398000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:31:47,664] INFO Deleted log /home/kfkdata/__consumer_offsets-7/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,665] INFO [Log partition=__consumer_offsets-49, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=0, lastModifiedTime=1605699398000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:31:47,666] INFO Deleted offset index /home/kfkdata/__consumer_offsets-7/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,666] INFO Deleted log /home/kfkdata/__consumer_offsets-49/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,666] INFO Deleted time index /home/kfkdata/__consumer_offsets-7/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,667] INFO Deleted offset index /home/kfkdata/__consumer_offsets-49/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,667] INFO Deleted time index /home/kfkdata/__consumer_offsets-49/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,719] INFO [Log partition=__consumer_offsets-43, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=0, lastModifiedTime=1605699398000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:31:47,719] INFO Deleted log /home/kfkdata/__consumer_offsets-43/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,720] INFO Deleted offset index /home/kfkdata/__consumer_offsets-43/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:31:47,721] INFO Deleted time index /home/kfkdata/__consumer_offsets-43/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,255] INFO [Log partition=__consumer_offsets-22, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=41781, lastModifiedTime=1605705201000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:32:01,256] INFO Deleted log /home/kfkdata/__consumer_offsets-22/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,256] INFO Deleted offset index /home/kfkdata/__consumer_offsets-22/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,256] INFO Deleted time index /home/kfkdata/__consumer_offsets-22/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,316] INFO [Log partition=__consumer_offsets-28, dir=/home/kfkdata] Deleting segments LogSegment(baseOffset=0, size=56549, lastModifiedTime=1605705201000, largestTime=0) (kafka.log.Log)
[2020-11-18 21:32:01,316] INFO Deleted log /home/kfkdata/__consumer_offsets-28/00000000000000000000.log.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,316] INFO Deleted offset index /home/kfkdata/__consumer_offsets-28/00000000000000000000.index.deleted. (kafka.log.LogSegment)
[2020-11-18 21:32:01,317] INFO Deleted time index /home/kfkdata/__consumer_offsets-28/00000000000000000000.timeindex.deleted. (kafka.log.LogSegment)

对于集群内拥有副本数量不同的节点,应该怎么做升级?

发表于 2020-11-19
添加评论

配置要保持一致的。

開開新新 -> 半兽人 3年前

如果保持一致 就是min.insync.replicas:1,那么有部分设置了副本因子3的,是不是会缩减到1?

半兽人 -> 開開新新 3年前

topic一旦创建,就不会在改变了。

開開新新 -> 半兽人 3年前

既然不会改变 ,那min.insync.replicas:3怎么会有影响?

半兽人 -> 開開新新 3年前

是broker级别的错,不是topic的错。

你的答案

查看kafka相关的其他问题或提一个您自己的问题