日志警告无法消除 : Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch

肥宁小短腿✌ 发表于: 2023-11-23   最后更新时间: 2023-11-23 17:10:25   353 游览

kafkaServer.out文件有大量以下告警,严重影响存储空间


[2023-11-22 11:23:44,497] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304816}, Current: {epoch:10, offset232258847} for Partition: loginfoTopic-17 (kafka.server.epoch.LeaderEpochFileCache)
[2023-11-22 11:23:44,621] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304771}, Current: {epoch:10, offset232258814} for Partition: loginfoTopic-11 (kafka.server.epoch.LeaderEpochFileCache)
[2023-11-22 11:23:44,622] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304795}, Current: {epoch:10, offset232258833} for Partition: loginfoTopic-8 (kafka.server.epoch.LeaderEpochFileCache)
[2023-11-22 11:23:44,650] WARN Received a PartitionLeaderEpoch assignment for an epoch < latestEpoch. This implies messages have arrived out of order. New: {epoch:0, offset:232304838}, Current: {epoch:10, offset232258887} for Partition: loginfoTopic-2 (kafka.server.epoch.LeaderEpochFileCache)

topic情况

[root@abc kafka]# bin/kafka-topics.sh --zookeeper localhost:2181 --describe --topic loginfoTopic
Topic:loginfoTopic      PartitionCount:20       ReplicationFactor:1     Configs:
        Topic: loginfoTopic     Partition: 0    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 1    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 2    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 3    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 4    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 5    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 6    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 7    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 8    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 9    Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 10   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 11   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 12   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 13   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 14   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 15   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 16   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 17   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 18   Leader: 0       Replicas: 0     Isr: 0
        Topic: loginfoTopic     Partition: 19   Leader: 0       Replicas: 0     Isr: 0

kafka的部分配置是

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

系统的日常数据采集是正常的,就是这个告警会不断重复出现,消耗大量的存储空间,导致系统无法正常运行。想通过更改kafkaServer.out的日志记录策略来减少日志的记录量,没找到可以设置的地方。

只是一个运维小白,希望各位大佬能详说,感谢

发表于 2023-11-23

重新分配分区的要求是怎么样的,我不太理解
broker只有1个的话,replica能是1吗

是kafka的bug导致的,你就1个节点,是不能的。

那是不是需要先多创建一个节点还是两个

broker.id=0,1

然后再进行分区?分区前要先创建副本吗
这三个参数需要改吗

offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

我完全不会kafka,请大佬见谅QAQ

1、kafka版本有bug,你需要先升级。
2、你现在只有一个kafka节点是0,所以你没地方可迁移呀。

请问需要升级到哪个版本才没有这个bug呀

当前版本为

org.apache.kafka.kafka_2.11-0.10.0.1.jar

除了

  • 1.1.0
  • 2.0.0

其他版本都可以。

好的。感谢大佬耐心解答!

我把版本更新到了1.0.1还是有这个问题,请问还有什么可能性呀

1.0.1是什么版本?我没见过这个版本。

我在官网下载的安装包是Scala 2.11 - kafka_2.11-1.0.1.tgz (asc, sha512)

下这个吧

kafka_2.11-1.1.1.tgz

要大于以上的小版本。

好嘞,感谢感谢

升级到该版本还是存在这个问题TAT

磁盘清理了吗?是新集群,还是用的老数据?

没有清理磁盘,需要清理哪些?
用的老数据

kafka数据:
server.properties中,找到log.dirs,全删,然后重启集群。

另外注意:数据不要放在'/tmp'下,会被系统自动回收

zookeeper数据:

/var/lib/zookeeper

问题已解决,感谢大佬耐心指点和解答!:)

你的答案

查看kafka相关的其他问题或提一个您自己的问题