kafka消费者订阅topic为什么重复消费?

開開新新 发表于: 2020-09-08   最后更新时间: 2020-09-09  

提问说明

kibana 上展示的有重复

kafkadrop上缺少2号记录

  1. 我用程序往kafka里提交了5条消息(0-4编号),但是使用不同的工具查看的message不一致
  2. kafka 版本2.2.0 (Commit:05fcfde8f69b0349)
  3. 之前有部分broker节点存在问题,后面修复了。在logstash和kafka日志里没有发现明显的报错


您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: kafka poll records数据为空?
下一条: kafka优化问题

  • 我找了另一个开发重新写了一段代码,确实是从1到5写入的,应该是之前的那位代码逻辑有问题。

    不过有点 奇怪 为什么 Test1和Test2 这条发了两次

    [{"SourceType":"IntegralMallLog","LogLevel":"info","ApiName":"Test0","ApiUrl":"/Order/Test0","Token":"","Data":{"RequestDto":"testdatadto"}}]
    [{"SourceType":"IntegralMallLog","LogLevel":"info","ApiName":"Test1","ApiUrl":"/Order/Test1","Token":"","Data":{"RequestURL":"axtest1","RequestDto":"testdatadto1"}},{"SourceType":"IntegralMallLog","LogLevel":"info","ApiName":"Test2","ApiUrl":"/Order/Test2","Token":"","Data":{"RequestURL":"axtest2","RequestDto":"testdatadto2"}}]
    [{"SourceType":"IntegralMallLog","LogLevel":"info","ApiName":"Test1","ApiUrl":"/Order/Test1","Token":"","Data":{"RequestURL":"axtest1","RequestDto":"testdatadto1"}},{"SourceType":"IntegralMallLog","LogLevel":"info","ApiName":"Test2","ApiUrl":"/Order/Test2","Token":"","Data":{"RequestURL":"axtest2","RequestDto":"testdatadto2"}}]
    [{"SourceType":"IntegralMallLog","LogLevel":"info","ApiName":"test3","ApiUrl":"test3","Token":"","Data":{"requestdto":{"custAccount":null,"salesCode":null,"dlvModeId":null,"description":null,"address":null,"sourceType":4,"bpmSerialNum":null,"salesPoolId":null,"confirmed":0,"salesLine":null},"responsedto":{"result":null,"targetUrl":null,"success":false,"error":null,"unAuthorizedRequest":false,"__abp":false}}}]
    [{"SourceType":"IntegralMallLog","LogLevel":"error","ApiName":"test4","ApiUrl":"test4","Token":"","Data":{"requestdto":{"custAccount":null,"salesCode":null,"dlvModeId":null,"description":null,"address":null,"sourceType":4,"bpmSerialNum":null,"salesPoolId":null,"confirmed":0,"salesLine":null},"exception":"exception test"}}]
    

    我有三个节点 每个消费者收到的消息都是这几个。 kafkatool 显示的也是这么多。 我想可能是代码的问题,另外logstash的问题我进一步排查下

    1、什么是不同的工具,看到的message不一致?
    2、logstash把采集的日志发送到es的消息都打印出来看看。
    3、起一个命令行,监听这topic的消息,是否就是5条(如果topic就是5条,可以排除它,把问题定位到消费者)