kafka消费者控制台不停打印一条消息(bin/kafka-console-consumer.sh)

木易 发表于: 2020-09-14   最后更新时间: 2020-09-15  

1、kafka消费者控制台不停打印一条消息
2、kafka版本 kafka_2.13-2.4.0.tgz
3、现象:只修改一次数据,kafka一直打印这条消息

 {"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}
{"data":[{"id":"45320","user_id":"524341983699795968","task_id":"101","task_name":"完成语境中学习15分钟","kind":"1","kind_name":"everyday","number":"1","total_number":"1","score":"0","insert_time":"1600071504","update_time":"1600071504","date_time":"1600012800","is_had":"1"}],"database":"world","es":1600071626000,"id":247,"isDdl":false,"mysqlType":{"id":"int(11)","user_id":"varchar(32)","task_id":"int(11)","task_name":"varchar(32)","kind":"int(11)","kind_name":"varchar(32)","number":"int(11)","total_number":"int(11)","score":"int(11)","insert_time":"int(10) unsigned","update_time":"int(10) unsigned","date_time":"int(10) unsigned","is_had":"tinyint(3)"},"old":[{"update_time":"1600071500"}],"pkNames":["id"],"sql":"","sqlType":{"id":4,"user_id":12,"task_id":4,"task_name":12,"kind":4,"kind_name":12,"number":4,"total_number":4,"score":4,"insert_time":4,"update_time":4,"date_time":4,"is_had":-6},"table":"tt_user_task","ts":1600071627111,"type":"UPDATE"}^C

4、kafka为集群环境(3台服务器三个实例),具体业务为:canal同步mysql,==》 kafka ==》logstash ==》es
5、已尝试删除topic,新建topic再写时依然出现这种状况
6、又遇见过的小伙伴吗,急需解决办法,邮箱:15661685094@163.com



您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: kafka topic个别分区出现leader为-1时,会导致消费者发生reblance吗
下一条: kafka poll records数据为空?

  • 跟kafka topic无关。
    mysql怎么把消费给kafka的?问题应该在这块逻辑上

    • 大佬,重启了下kafka集群

      tt_video        world_tt_video  6          0               61869           61869           logstash-2-839c8767-bed8-48f5-9750-e32f18c7e05f /172.19.167.102 logstash-2
      tt_video        world_tt_video  7          0               56376           56376           logstash-2-839c8767-bed8-48f5-9750-e32f18c7e05f /172.19.167.102 logstash-2
      tt_video        world_tt_video  8          0               58132           58132           logstash-2-839c8767-bed8-48f5-9750-e32f18c7e05f /172.19.167.102 logstash-2
      tt_video        world_tt_video  12         0               58475           58475           logstash-4-1c02cb6c-e634-4843-8ef1-a83264701973 /172.19.167.102 logstash-4
      tt_video        world_tt_video  13         0               55608           55608           logstash-4-1c02cb6c-e634-4843-8ef1-a83264701973 /172.19.167.102 logstash-4
      tt_video        world_tt_video  14         4               57689           57685           logstash-4-1c02cb6c-e634-4843-8ef1-a83264701973 /172.19.167.102 logstash-4
      

      怎么 CURRENT-OFFSET的数量都跑到 LAG 了

        • GROUP           TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                                     HOST            CLIENT-ID
          tt_video world_tt_video   6                      0                            61869                         61869 logstash-2-839c8767-bed8-48f5-9750-e32f18c7e05f /172.19.167.102 logstash-2
          tt_video world_tt_video   7                      0                            56376                         56376 logstash-2-839c8767-bed8-48f5-9750-e32f18c7e05f /172.19.167.102 logstash-2
          tt_video world_tt_video   8                      0                            58132                         58132 logstash-2-839c8767-bed8-48f5-9750-e32f18c7e05f /172.19.167.102 logstash-2
          tt_video world_tt_video   12                    0                            58475                         58475 logstash-4-1c02cb6c-e634-4843-8ef1-a83264701973 /172.19.167.102 logstash-4
          tt_video world_tt_video   13                    0                            55608                         55608 logstash-4-1c02cb6c-e634-4843-8ef1-a83264701973 /172.19.167.102 logstash-4
          tt_video world_tt_video   14                    4                            57689                         57685 logstash-4-1c02cb6c-e634-4843-8ef1-a83264701973 /172.19.167.102 logstash-4
          
            • Fetch offset 4255503097 is out of range for partition master-tt-user-task-13, resetting offset
              Resetting offset for partition master-tt-user-task-13 to offset 104.
              Fetch offset 4253098153 is out of range for partition master-tt-user-task-14, resetting offset
              Resetting offset for partition master-tt-user-task-14 to offset 107.
              我这边使用logstash做的消费者,现在日志里总是包这些消息是正常的吗(我感觉Fetch offset 4255503097 is out…… 是不是有问题 ),我感觉有点问题

                • 大佬,kafka报了个错
                  [2020-09-18 18:05:11,532] ERROR [ReplicaFetcher replicaId=1, leaderId=0, fetcherId=0] Unexpected error occurred while processing data for partition topic-playlist-words-12 at offset 7081 (kafka.server.ReplicaFetcherThread)
                  kafka.common.OffsetsOutOfOrderException: Out of order offsets found in append to topic-playlist-words-12: ArrayBuffer(4256941968, 4255223704, 4283001280, 4283075360……)
                  是怎么回事

                    • 大佬出现这种问题,是啥情况呢(还是上边那个同步mysql数据的事,kafka跑一会就这样了,然后查看kafka的是时候就一直在刷消息,突然变成了大的数字“ 4076763905 375 -4076763530”)

                      logstash_playlist_pr6 topic-playlist-words6 1          4076763905      375             -4076763530     logstash_playlist_p6-1-cdd8b4d9-e9d0-4a2f-939c-ebd110ec0475 /172.19.167.97  logstash_playlist_p6-1
                      logstash_playlist_pr6 topic-playlist6       1          235             235             0               logstash_playlist_p6-1-cdd8b4d9-e9d0-4a2f-939c-ebd110ec0475 /172.19.167.97  logstash_playlist_p6-1
                      logstash_playlist_pr6 topic-playlist-words6 3          369             369             0               logstash_playlist_p6-3-f25e1500-cae9-4b23-94e5-5286bf5c18fb /172.19.167.97  logstash_playlist_p6-3
                      logstash_playlist_pr6 topic-playlist6       3          4062253257      137             -4062253120     logstash_playlist_p6-3-f25e1500-cae9-4b23-94e5-5286bf5c18fb /172.19.167.97  logstash_playlist_p6-3
                      logstash_playlist_pr6 topic-playlist-words6 2          4062453161      375             -4062452786     logstash_playlist_p6-2-93da2c58-734f-41b2-a820-f14049516712 /172.19.167.97  logstash_playlist_p6-2
                      logstash_playlist_pr6 topic-playlist6       2          326             326             0               logstash_playlist_p6-2-93da2c58-734f-41b2-a820-f14049516712 /172.19.167.97  logstash_playlist_p6-2
                      logstash_playlist_pr6 topic-playlist6       4          4067926729      769             -4067925960     logstash_playlist_p6-4-4e234e6e-badb-4147-99b2-d2770b0d6c8c /172.19.167.97  logstash_playlist_p6-4
                      logstash_playlist_pr6 topic-playlist-words6 4          396             396             0               logstash_playlist_p6-4-4e234e6e-badb-4147-99b2-d2770b0d6c8c /172.19.167.97  logstash_playlist_p6-4
                      logstash_playlist_pr6 topic-playlist-words6 0          4062556329      346             -4062555983     logstash_playlist_p6-0-7bbafac3-35ec-4f98-bee1-9f9847705670 /172.19.167.97  logstash_playlist_p6-0
                      logstash_playlist_pr6 topic-playlist6       0          135             135             0               logstash_playlist_p6-0-7bbafac3-35ec-4f98-bee1-9f9847705670 /172.19.167.97  logstash_playlist_p6-0
                      
                        • 一般出现负数 是因为topic分区的消息和统计的消息不匹配,这种现象常见是在kakfa集群某个分区故障,数据是新的。
                          如果是kafka第三方监控统计的。重启下监控,也会恢复正常(第三方监控会缓存一些统计信息)。

                            • 不是监控,是(./kafka-consumer-groups.sh --bootstrap-server 172.19.167.197:9092 --describe --group logstash_playlist_pr6)这个命令看到的,不正常的那几个分区就是一直在刷一条消息(就是我最初提的问题:kafka消费者控制台不停打印一条消息?(用这个监控的:./kafka-console-consumer.sh --bootstrap-server 172.19.167.197:9092 --from-beginning --topic topic-playlist-words6 ))

                                • 这条消费命令,每次会从头开始消费topic的消息,--from-beginning控制的。
                                  另外,不说从头开始消费的问题,如果你一直持续收1条消息,说明你的生产者(logstash)持续发送同一条消息。

                                    • 还有,负数还有一种是你的log删除策略,kafka会默认删除7天前的日志,如果你的消费者很长时间没起,数据已经被清理了,也会导致你看到负数。