kafka集群在zookeeper中注册的/borkers/ids 自动删除了，但kafka进程存在

痕☆ 发表于: 2022-06-22   最后更新时间: 2024-05-23 10:00:12   1,413 游览

请教一个关于kafka集群的问题，困扰了好几天了，仍未解决。

之前使用阿里云的经典网络服务器，部署了kafka服务（3个节点）和zookeeper服务（节点），应阿里云迁移要求，从经典网络迁移到他们的专有网络。

迁移完成后重启服务，kafka集群在zookeeper中注册的/borkers/ids经常自动运行一段时间后，/borkers/ids注册信息自动就没了，但是节点服务器上kakfa进程都在，3个节点中有时2个节点注册信息都会掉。

之前kafka挺稳定的，就是迁移以后不稳定，节点一直掉，导致服务异常。

kafka版本：0.8.2.1
zookeeper版本：3.4.6

我这边做过如下处理：

  1. 修改过kafka配置文件中的zookeeper相关超时时间；

  2. 之前谷歌发现有人有出现过相同问题，原因是说0.8.2.1存在controller相关的bug，在0.9版本以后修复，于是在6月21日对kafka进行了版本升级，升级到0.10.0.1；

  3. 查看服务器相关监控指标，基本都在控制范围内，都是内网交互，偶尔内网带宽会高一点；

以上处理方式并没有解决问题，今日发现异常仍然存在。

以下是日志中的错误日志

java.io.IOException: Connection to 0 was disconnected before the response was read
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
  at scala.Option.foreach(Option.scala:257)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
  at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
  at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
  at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-22 04:29:24,946] WARN [ReplicaFetcherThread-0-0], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5d9da6c4 (kafka.server.ReplicaFetcherThread)
java.net.SocketTimeoutException: Failed to connect within 30000 ms
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:240)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-22 04:29:36,017] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

以下是某个节点的配置，当前版本是0.10.0.1：

broker.id=0
host.name=DC-Kafka-1
port=9092
num.network.threads=4
num.io.threads=8
socket.send.buffer.bytes=1024000
socket.receive.buffer.bytes=1024000
socket.request.max.bytes=104857600
log.dirs=/home/kafka/kafka-logs/broker-0
num.partitions=4
num.recovery.threads.per.data.dir=1
#log.flush.interval.ms=1000
log.retention.hours=168
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka
zookeeper.connection.timeout.ms=60000
zookeeper.session.timeout.ms=120000

3个节点服务器，kafka日志存储目录大小均在23G左右。
会不会是因为kafka日志数据大，节点之间数据同步导致超时，这只是猜测。

已经出现好几天了，没有排查头绪，请大佬帮忙梳理下。

非常感谢！

以下是服务器监控信息和日志。

screenshot

[2022-06-23 00:05:20,718] INFO KafkaConfig values: 
  advertised.host.name = null
  metric.reporters = []
  quota.producer.default = 9223372036854775807
  offsets.topic.num.partitions = 50
  log.flush.interval.messages = 9223372036854775807
  auto.create.topics.enable = true
  controller.socket.timeout.ms = 30000
  log.flush.interval.ms = null
  principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
  replica.socket.receive.buffer.bytes = 65536
  min.insync.replicas = 1
  replica.fetch.wait.max.ms = 500
  num.recovery.threads.per.data.dir = 1
  ssl.keystore.type = JKS
  sasl.mechanism.inter.broker.protocol = GSSAPI
  default.replication.factor = 3
  ssl.truststore.password = null
  log.preallocate = false
  sasl.kerberos.principal.to.local.rules = [DEFAULT]
  fetch.purgatory.purge.interval.requests = 1000
  ssl.endpoint.identification.algorithm = null
  replica.socket.timeout.ms = 30000
  message.max.bytes = 2048000
  num.io.threads = 10
  offsets.commit.required.acks = -1
  log.flush.offset.checkpoint.interval.ms = 60000
  delete.topic.enable = false
  quota.window.size.seconds = 1
  ssl.truststore.type = JKS
  offsets.commit.timeout.ms = 5000
  quota.window.num = 11
  zookeeper.connect = DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka
  authorizer.class.name = 
  num.replica.fetchers = 1
  log.retention.ms = null
  log.roll.jitter.hours = 0
  log.cleaner.enable = false
  offsets.load.buffer.size = 5242880
  log.cleaner.delete.retention.ms = 86400000
  ssl.client.auth = none
  controlled.shutdown.max.retries = 3
  queued.max.requests = 500
  offsets.topic.replication.factor = 1
  log.cleaner.threads = 1
  sasl.kerberos.service.name = null
  sasl.kerberos.ticket.renew.jitter = 0.05
  socket.request.max.bytes = 104857600
  ssl.trustmanager.algorithm = PKIX
  zookeeper.session.timeout.ms = 90000
  log.retention.bytes = -1
  log.message.timestamp.type = CreateTime
  sasl.kerberos.min.time.before.relogin = 60000
  zookeeper.set.acl = false
  connections.max.idle.ms = 600000
  offsets.retention.minutes = 1440
  replica.fetch.backoff.ms = 1000
  inter.broker.protocol.version = 0.10.0-IV1
  log.retention.hours = 72
  num.partitions = 6
  broker.id.generation.enable = true
  listeners = null
  ssl.provider = null
  ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
  log.roll.ms = null
  log.flush.scheduler.interval.ms = 9223372036854775807
  ssl.cipher.suites = null
  log.index.size.max.bytes = 10485760
  ssl.keymanager.algorithm = SunX509
  security.inter.broker.protocol = PLAINTEXT
  replica.fetch.max.bytes = 2048576
  advertised.port = null
  log.cleaner.dedupe.buffer.size = 134217728
  replica.high.watermark.checkpoint.interval.ms = 5000
  log.cleaner.io.buffer.size = 524288
  sasl.kerberos.ticket.renew.window.factor = 0.8
  zookeeper.connection.timeout.ms = 60000
  controlled.shutdown.retry.backoff.ms = 5000
  log.roll.hours = 168
  log.cleanup.policy = delete
  host.name = DC-Kafka-1
  log.roll.jitter.ms = null
  max.connections.per.ip = 2147483647
  offsets.topic.segment.bytes = 104857600
  background.threads = 10
  quota.consumer.default = 9223372036854775807
  request.timeout.ms = 30000
  log.message.format.version = 0.10.0-IV1
  log.index.interval.bytes = 4096
  log.dir = /tmp/kafka-logs
  log.segment.bytes = 1073741824
  log.cleaner.backoff.ms = 15000
  offset.metadata.max.bytes = 4096
  ssl.truststore.location = null
  group.max.session.timeout.ms = 300000
  ssl.keystore.password = null
  zookeeper.sync.time.ms = 2000
  port = 9092
  log.retention.minutes = null
  log.segment.delete.delay.ms = 60000
  log.dirs = /home/kafka/kafka-logs
  controlled.shutdown.enable = true
  compression.type = producer
  max.connections.per.ip.overrides = 
  log.message.timestamp.difference.max.ms = 9223372036854775807
  sasl.kerberos.kinit.cmd = /usr/bin/kinit
  log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
  auto.leader.rebalance.enable = true
  leader.imbalance.check.interval.seconds = 300
  log.cleaner.min.cleanable.ratio = 0.5
  replica.lag.time.max.ms = 10000
  num.network.threads = 3
  ssl.key.password = null
  reserved.broker.max.id = 1000
  metrics.num.samples = 2
  socket.send.buffer.bytes = 102400
  ssl.protocol = TLS
  socket.receive.buffer.bytes = 102400
  ssl.keystore.location = null
  replica.fetch.min.bytes = 1
  broker.rack = null
  unclean.leader.election.enable = true
  sasl.enabled.mechanisms = [GSSAPI]
  group.min.session.timeout.ms = 6000
  log.cleaner.io.buffer.load.factor = 0.9
  offsets.retention.check.interval.ms = 600000
  producer.purgatory.purge.interval.requests = 1000
  metrics.sample.window.ms = 30000
  broker.id = 0
  offsets.topic.compression.codec = 0
  log.retention.check.interval.ms = 300000
  advertised.listeners = null
  leader.imbalance.per.broker.percentage = 10
 (kafka.server.KafkaConfig)
[2022-06-23 00:05:20,796] INFO starting (kafka.server.KafkaServer)
[2022-06-23 00:05:20,803] INFO Connecting to zookeeper on DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka (kafka.server.KafkaServer)
[2022-06-23 00:05:20,818] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:host.name=iZx3rbxxxxxx (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.version=1.8.0_321 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.home=/usr/java/jdk1.8.0_321/jre (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.class.path=.:/usr/java/jdk1.8.0_321/lib:/usr/java/jdk1.8.0_321/jre/lib:.:/usr/java/jdk1.8.0_321/lib:/usr/java/jdk1.8.0_321/jre/lib::/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/argparse4j-0.5.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-api-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-file-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-json-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-runtime-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/guava-18.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/hk2-api-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/hk2-locator-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/hk2-utils-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-annotations-2.6.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-core-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-databind-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javassist-3.18.2-GA.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-client-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-common-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-guava-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-media-jaxb-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-server-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jopt-simple-4.9.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-sources.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-test-sources.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-clients-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-log4j-appender-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-examples-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-tools-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/log4j-1.2.17.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/lz4-1.3.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/reflections-0.9.10.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/rocksdbjni-4.8.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/scala-library-2.11.8.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/slf4j-api-1.7.21.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/slf4j-log4j12-1.7.21.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/snappy-java-1.1.2.6.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/validation-api-1.1.0.Final.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/zkclient-0.8.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/zookeeper-3.4.6.jar (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:os.version=3.10.0-123.9.3.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:user.dir=/home/kafka/kafka_2.11-0.10.0.1/config (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,819] INFO Initiating client connection, connectString=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181 sessionTimeout=90000 watcher=org.I0Itec.zkclient.ZkClient@5119fb47 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,826] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2022-06-23 00:05:20,859] INFO Opening socket connection to server DC-Ice-2/10.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,862] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:20,862] INFO Socket connection established to DC-Ice-2/10.x.x.x:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,874] INFO Session establishment complete on server DC-Ice-2/10.x.x.x:2181, sessionid = 0x2818b2b837c0139, negotiated timeout = 90000 (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,875] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:20,884] INFO Created zookeeper path /kafka (kafka.server.KafkaServer)
[2022-06-23 00:05:20,884] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2022-06-23 00:05:20,889] INFO Session: 0x2818b2b837c0139 closed (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,889] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,898] INFO Initiating client connection, connectString=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka sessionTimeout=90000 watcher=org.I0Itec.zkclient.ZkClient@6356695f (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,898] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2022-06-23 00:05:20,905] INFO Opening socket connection to server DC-Storm-3/10.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,906] INFO Socket connection established to DC-Storm-3/10.x.x.x:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,908] INFO Session establishment complete on server DC-Storm-3/10.x.x.x:2181, sessionid = 0x3818b2b83980100, negotiated timeout = 90000 (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,910] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:20,910] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:21,142] INFO Loading logs. (kafka.log.LogManager)
[2022-06-23 00:05:21,207] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,222] INFO Recovering unflushed segment 0 in log Gxky_rec-2. (kafka.log.Log)
[2022-06-23 00:05:21,228] INFO Completed load of log Gxky_rec-2 with log end offset 17 (kafka.log.Log)
[2022-06-23 00:05:21,239] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,305] INFO Recovering unflushed segment 0 in log Gqxrmt_data-1. (kafka.log.Log)
[2022-06-23 00:05:21,314] INFO Completed load of log Gqxrmt_data-1 with log end offset 1677 (kafka.log.Log)
[2022-06-23 00:05:21,320] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,320] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-3. (kafka.log.Log)
[2022-06-23 00:05:21,323] INFO Completed load of log Gqxrmt_rec-3 with log end offset 0 (kafka.log.Log)
[2022-06-23 00:05:33,696] INFO Logs loading complete. (kafka.log.LogManager)
[2022-06-23 00:05:33,696] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2022-06-23 00:05:33,700] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2022-06-23 00:05:33,751] INFO Awaiting socket connections on DC-Kafka-1:9092. (kafka.network.Acceptor)
[2022-06-23 00:05:33,755] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2022-06-23 00:05:33,798] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,798] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,842] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,843] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,863] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2022-06-23 00:05:33,863] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2022-06-23 00:05:33,889] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 16 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:05:33,908] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2022-06-23 00:05:33,913] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2022-06-23 00:05:33,913] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2022-06-23 00:05:33,930] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:05:33,936] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:05:33,937] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(DC-Kafka-1,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2022-06-23 00:05:33,946] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2022-06-23 00:05:33,946] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2022-06-23 00:05:33,947] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2022-06-23 00:05:34,336] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gqxrmt_data,0],[Gxky_event,5],[Gqxrmt_event,0],[Gqxrmt_rec,5],[Gxky_data,3],[Gdsjb_data,5],[Gdsjb_event,1],[Gdsjb_data,4],[Gxky_data,1],[Gdsjb_data,1],[Gdsjb_rec,1],[Gxky_event,1],[Gxky_data,4],[Gxky_rec,0],[Gxky_rec,4],[Gdsjb_rec,2],[Gxky_data,0],[Gdsjb_data,2],[Gqxrmt_event,2],[Gdsjb_rec,3],[Gqxrmt_data,2],[Gqxrmt_event,1],[Gqxrmt_data,4],[Gdsjb_rec,0],[Gqxrmt_data,3],[Gqxrmt_rec,1],[Gdsjb_event,5],[Gdsjb_rec,4],[Gdsjb_event,0],[Gqxrmt_rec,3],[Gxky_rec,3],[Gqxrmt_event,3],[Gdsjb_data,0],[Gxky_data,2],[Gxky_event,0],[Gdsjb_event,3],[Gxky_rec,5],[Gxky_data,5],[Gqxrmt_rec,2],[Gxky_event,2],[Gxky_event,3],[Gqxrmt_event,4],[Gxky_rec,1],[Gqxrmt_rec,0],[Gqxrmt_data,1],[Gdsjb_event,4],[Gqxrmt_rec,4],[Gqxrmt_data,5],[Gxky_event,4],[Gqxrmt_event,5],[Gxky_rec,2],[Gdsjb_event,2],[Gdsjb_rec,5],[Gdsjb_data,3] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:05:34,348] INFO Truncating log Gxky_event-5 to offset 48113. (kafka.log.Log)
[2022-06-23 00:05:34,352] INFO Truncating log Gxky_event-4 to offset 47766. (kafka.log.Log)
[2022-06-23 00:05:34,352] INFO Truncating log Gxky_rec-0 to offset 16. (kafka.log.Log)
[2022-06-23 00:05:34,353] INFO Truncating log Gdsjb_rec-3 to offset 179. (kafka.log.Log)
[2022-06-23 00:05:34,353] INFO Truncating log Gxky_data-4 to offset 294. (kafka.log.Log)
[2022-06-23 00:05:34,353] INFO Truncating log Gxky_data-1 to offset 323. (kafka.log.Log)
[2022-06-23 00:05:34,354] INFO Truncating log Gdsjb_rec-1 to offset 165. (kafka.log.Log)
[2022-06-23 00:05:34,360] INFO Truncating log Gxky_rec-5 to offset 13. (kafka.log.Log)
[2022-06-23 00:05:34,360] INFO Truncating log Gqxrmt_rec-2 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,361] INFO Truncating log Gqxrmt_data-0 to offset 1625. (kafka.log.Log)
[2022-06-23 00:05:34,361] INFO Truncating log Gqxrmt_rec-0 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,362] INFO Truncating log Gxky_rec-3 to offset 16. (kafka.log.Log)
[2022-06-23 00:05:34,362] INFO Truncating log Gxky_event-3 to offset 48270. (kafka.log.Log)
[2022-06-23 00:05:34,363] INFO Truncating log Gxky_data-5 to offset 316. (kafka.log.Log)
[2022-06-23 00:05:34,363] INFO Truncating log Gdsjb_event-5 to offset 462449. (kafka.log.Log)
[2022-06-23 00:05:34,364] INFO Truncating log Gqxrmt_data-1 to offset 1677. (kafka.log.Log)
[2022-06-23 00:05:34,364] INFO Truncating log Gdsjb_data-4 to offset 172. (kafka.log.Log)
[2022-06-23 00:05:34,365] INFO Truncating log Gxky_data-2 to offset 403. (kafka.log.Log)
[2022-06-23 00:05:34,365] INFO Truncating log Gqxrmt_rec-3 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,365] INFO Truncating log Gqxrmt_data-2 to offset 1626. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gqxrmt_rec-1 to offset 1. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gdsjb_event-0 to offset 461377. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gqxrmt_rec-4 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gqxrmt_event-5 to offset 76927. (kafka.log.Log)
[2022-06-23 00:05:34,367] INFO Truncating log Gqxrmt_data-4 to offset 1579. (kafka.log.Log)
[2022-06-23 00:05:34,367] INFO Truncating log Gqxrmt_rec-5 to offset 1. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gxky_event-0 to offset 48069. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gqxrmt_event-3 to offset 76895. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gqxrmt_event-0 to offset 76454. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gdsjb_rec-2 to offset 173. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gxky_rec-1 to offset 13. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gqxrmt_data-5 to offset 1673. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gxky_rec-2 to offset 17. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gdsjb_rec-0 to offset 160. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gdsjb_event-4 to offset 459323. (kafka.log.Log)
[2022-06-23 00:05:34,370] INFO Truncating log Gdsjb_data-2 to offset 180. (kafka.log.Log)
[2022-06-23 00:05:34,370] INFO Truncating log Gdsjb_data-3 to offset 158. (kafka.log.Log)
[2022-06-23 00:05:34,370] INFO Truncating log Gdsjb_data-0 to offset 140. (kafka.log.Log)
[2022-06-23 00:05:34,371] INFO Truncating log Gdsjb_data-1 to offset 138. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gqxrmt_event-4 to offset 76771. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gdsjb_rec-4 to offset 149. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gqxrmt_event-1 to offset 76290. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gdsjb_event-2 to offset 477694. (kafka.log.Log)
[2022-06-23 00:05:34,375] INFO Truncating log Gdsjb_data-5 to offset 155. (kafka.log.Log)
[2022-06-23 00:05:34,375] INFO Truncating log Gxky_data-3 to offset 332. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gqxrmt_data-3 to offset 1542. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gdsjb_event-1 to offset 479871. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gqxrmt_event-2 to offset 77283. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gdsjb_rec-5 to offset 168. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gxky_event-2 to offset 47985. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gxky_data-0 to offset 284. (kafka.log.Log)
[2022-06-23 00:05:34,377] INFO Truncating log Gxky_event-1 to offset 48289. (kafka.log.Log)
[2022-06-23 00:05:34,377] INFO Truncating log Gxky_rec-4 to offset 27. (kafka.log.Log)
[2022-06-23 00:05:34,378] INFO Truncating log Gdsjb_event-3 to offset 460706. (kafka.log.Log)
[2022-06-23 00:05:34,440] INFO [ReplicaFetcherThread-0-1], Starting  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:05:34,443] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List([[Gxky_event,5], initOffset 48113 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,4], initOffset 47766 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,0], initOffset 16 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,3], initOffset 179 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,4], initOffset 294 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,1], initOffset 323 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,1], initOffset 165 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,5], initOffset 13 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,2], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,0], initOffset 1625 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,0], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,3], initOffset 16 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,3], initOffset 48270 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,5], initOffset 316 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,5], initOffset 462449 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,1], initOffset 1677 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,4], initOffset 172 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,2], initOffset 403 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,3], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,2], initOffset 1626 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,1], initOffset 1 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,0], initOffset 461377 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,4], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,5], initOffset 76927 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,4], initOffset 1579 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,5], initOffset 1 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,0], initOffset 48069 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,3], initOffset 76895 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,0], initOffset 76454 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,2], initOffset 173 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,1], initOffset 13 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,5], initOffset 1673 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,2], initOffset 17 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,0], initOffset 160 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,4], initOffset 459323 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,2], initOffset 180 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,3], initOffset 158 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,0], initOffset 140 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,1], initOffset 138 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,4], initOffset 76771 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,4], initOffset 149 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,1], initOffset 76290 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,2], initOffset 477694 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,5], initOffset 155 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,3], initOffset 332 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,3], initOffset 1542 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,1], initOffset 479871 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,2], initOffset 77283 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,5], initOffset 168 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,2], initOffset 47985 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,0], initOffset 284 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,1], initOffset 48289 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,4], initOffset 27 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,3], initOffset 460706 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] ) (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:08:55,823] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@1de06723 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1 was disconnected before the response was read
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
  at scala.Option.foreach(Option.scala:257)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
  at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
  at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
  at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:08:57,832] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@7613510c (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
  at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
  at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
  at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:24,012] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2022-06-23 00:10:24,499] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gxky_event,5],[Gxky_event,4],[Gdsjb_rec,3],[Gxky_data,4],[Gxky_rec,5],[Gqxrmt_data,1],[Gdsjb_data,4],[Gqxrmt_data,2],[Gqxrmt_rec,1],[Gqxrmt_rec,4],[Gqxrmt_data,4],[Gqxrmt_rec,5],[Gqxrmt_event,3],[Gqxrmt_event,0],[Gdsjb_rec,2],[Gxky_rec,1],[Gdsjb_event,4],[Gdsjb_data,2],[Gdsjb_data,1],[Gqxrmt_event,4],[Gdsjb_event,2],[Gxky_data,3],[Gdsjb_event,1],[Gdsjb_rec,5],[Gxky_data,0],[Gxky_event,1],[Gxky_rec,4] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:10:24,511] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gqxrmt_data,0],[Gdsjb_event,5],[Gdsjb_rec,4],[Gdsjb_data,5],[Gdsjb_event,0],[Gqxrmt_rec,3],[Gxky_data,1],[Gxky_data,2],[Gdsjb_event,3],[Gxky_rec,3],[Gdsjb_rec,1],[Gxky_event,0],[Gdsjb_data,0],[Gxky_data,5],[Gqxrmt_rec,2],[Gxky_event,2],[Gxky_event,3],[Gxky_rec,0],[Gqxrmt_rec,0],[Gqxrmt_data,5],[Gqxrmt_event,2],[Gqxrmt_event,5],[Gqxrmt_event,1],[Gdsjb_rec,0],[Gxky_rec,2],[Gqxrmt_data,3],[Gdsjb_data,3] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_rec-0 to offset 17. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_data-1 to offset 332. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gdsjb_rec-1 to offset 176. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_rec-2 to offset 0. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_data-0 to offset 1650. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_rec-0 to offset 0. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_rec-3 to offset 16. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_event-3 to offset 52068. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_data-5 to offset 333. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gdsjb_event-5 to offset 490190. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_data-2 to offset 414. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_rec-3 to offset 0. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gdsjb_event-0 to offset 489031. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_event-5 to offset 83926. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gxky_event-0 to offset 52018. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_data-5 to offset 1699. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gxky_rec-2 to offset 17. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_rec-0 to offset 171. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_data-3 to offset 159. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_data-0 to offset 140. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_rec-4 to offset 157. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_event-1 to offset 83328. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_data-5 to offset 156. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_data-3 to offset 1561. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_event-2 to offset 84475. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gxky_event-2 to offset 51758. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_event-3 to offset 488286. (kafka.log.Log)
[2022-06-23 00:10:24,522] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List([[Gxky_rec,0], initOffset 17 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_data,1], initOffset 332 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_rec,1], initOffset 176 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_rec,2], initOffset 0 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_data,0], initOffset 1650 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_rec,0], initOffset 0 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_rec,3], initOffset 16 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_event,3], initOffset 52068 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_data,5], initOffset 333 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_event,5], initOffset 490190 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_data,2], initOffset 414 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_rec,3], initOffset 0 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_event,0], initOffset 489031 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_event,5], initOffset 83926 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_event,0], initOffset 52018 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_data,5], initOffset 1699 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_rec,2], initOffset 17 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_rec,0], initOffset 171 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_data,3], initOffset 159 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_data,0], initOffset 140 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_rec,4], initOffset 157 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_event,1], initOffset 83328 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_data,5], initOffset 156 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_data,3], initOffset 1561 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_event,2], initOffset 84475 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_event,2], initOffset 51758 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_event,3], initOffset 488286 to broker BrokerEndPoint(2,dc-solr-1,9092)] ) (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:10:24,523] INFO [ReplicaFetcherThread-0-1], Shutting down (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:10:24,525] INFO [ReplicaFetcherThread-0-1], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:10:24,525] INFO [ReplicaFetcherThread-0-1], Stopped  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:10:24,530] INFO [ReplicaFetcherThread-0-2], Starting  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:15:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:25:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:26:15,288] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@7cab2945 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
  at scala.Option.foreach(Option.scala:257)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
  at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
  at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
  at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:26:23,785] INFO Partition [Gdsjb_data,4] on broker 0: Shrinking ISR for partition [Gdsjb_data,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,792] INFO Partition [Gdsjb_event,1] on broker 0: Shrinking ISR for partition [Gdsjb_event,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,794] INFO Partition [Gqxrmt_rec,5] on broker 0: Shrinking ISR for partition [Gqxrmt_rec,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,796] INFO Partition [Gxky_event,4] on broker 0: Shrinking ISR for partition [Gxky_event,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,798] INFO Partition [Gqxrmt_rec,1] on broker 0: Shrinking ISR for partition [Gqxrmt_rec,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,800] INFO Partition [Gdsjb_data,1] on broker 0: Shrinking ISR for partition [Gdsjb_data,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,802] INFO Partition [Gqxrmt_event,4] on broker 0: Shrinking ISR for partition [Gqxrmt_event,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,805] INFO Partition [Gdsjb_event,4] on broker 0: Shrinking ISR for partition [Gdsjb_event,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,808] INFO Partition [Gxky_data,4] on broker 0: Shrinking ISR for partition [Gxky_data,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,810] INFO Partition [Gqxrmt_data,4] on broker 0: Shrinking ISR for partition [Gqxrmt_data,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,811] INFO Partition [Gdsjb_event,2] on broker 0: Shrinking ISR for partition [Gdsjb_event,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,813] INFO Partition [Gdsjb_rec,2] on broker 0: Shrinking ISR for partition [Gdsjb_rec,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,815] INFO Partition [Gqxrmt_data,1] on broker 0: Shrinking ISR for partition [Gqxrmt_data,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,817] INFO Partition [Gqxrmt_data,2] on broker 0: Shrinking ISR for partition [Gqxrmt_data,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,819] INFO Partition [Gqxrmt_rec,4] on broker 0: Shrinking ISR for partition [Gqxrmt_rec,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,821] INFO Partition [Gxky_data,3] on broker 0: Shrinking ISR for partition [Gxky_data,3] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,822] INFO Partition [Gdsjb_rec,3] on broker 0: Shrinking ISR for partition [Gdsjb_rec,3] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,824] INFO Partition [Gxky_data,0] on broker 0: Shrinking ISR for partition [Gxky_data,0] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,826] INFO Partition [Gxky_rec,5] on broker 0: Shrinking ISR for partition [Gxky_rec,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,828] INFO Partition [Gxky_event,1] on broker 0: Shrinking ISR for partition [Gxky_event,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,830] INFO Partition [Gxky_rec,1] on broker 0: Shrinking ISR for partition [Gxky_rec,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,832] INFO Partition [Gqxrmt_event,3] on broker 0: Shrinking ISR for partition [Gqxrmt_event,3] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,834] INFO Partition [Gdsjb_rec,5] on broker 0: Shrinking ISR for partition [Gdsjb_rec,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,836] INFO Partition [Gdsjb_data,2] on broker 0: Shrinking ISR for partition [Gdsjb_data,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,837] INFO Partition [Gxky_event,5] on broker 0: Shrinking ISR for partition [Gxky_event,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,840] INFO Partition [Gxky_rec,4] on broker 0: Shrinking ISR for partition [Gxky_rec,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,841] INFO Partition [Gqxrmt_event,0] on broker 0: Shrinking ISR for partition [Gqxrmt_event,0] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:47,318] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@2d1e26f4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
  at scala.Option.foreach(Option.scala:257)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
  at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
  at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
  at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:27:19,349] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@361a4a63 (kafka.server.ReplicaFetcherThread)
java.net.SocketTimeoutException: Failed to connect within 30000 ms
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:240)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:27:35,533] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@2cb8ae5b (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
  at scala.Option.foreach(Option.scala:257)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
  at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
  at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
  at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
  at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
  at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
  at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
  at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
  at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
  at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:27:44,004] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:27:44,006] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:27:44,006] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2022-06-23 00:27:44,365] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gxky_rec,0],[Gxky_data,1],[Gdsjb_rec,1],[Gqxrmt_rec,2],[Gqxrmt_data,0],[Gqxrmt_rec,0],[Gxky_rec,3],[Gxky_event,3],[Gxky_data,5],[Gdsjb_event,5],[Gxky_data,2],[Gqxrmt_rec,3],[Gdsjb_event,0],[Gqxrmt_event,5],[Gxky_event,0],[Gqxrmt_data,5],[Gxky_rec,2],[Gdsjb_rec,0],[Gdsjb_data,3],[Gdsjb_data,0],[Gdsjb_rec,4],[Gqxrmt_event,1],[Gdsjb_data,5],[Gqxrmt_data,3],[Gqxrmt_event,2],[Gxky_event,2],[Gdsjb_event,3] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:27:44,367] INFO [ReplicaFetcherThread-0-2], Shutting down (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:27:44,383] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2022-06-23 00:27:58,733] INFO [ReplicaFetcherThread-0-2], Stopped  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:27:58,733] INFO [ReplicaFetcherThread-0-2], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:35:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:45:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:55:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 01:05:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 01:15:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 01:25:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
kafka
发表于 2022-06-22
痕☆
添加评论

这种超时的错误，一般2种情况会引起

  1. 本身就无法到达（确保所有节点（hostname）均可互相识别和网络畅通）
  2. 某些资源占用率过高，导致一直在等待而导致的超时（例如迁移分区会产生大量的网络io需要进行限流）。

另外：brokers/ids 来是退出，还有种情况是broker.id误写重复了，导致数据混乱，kafka是基于broker.id来确认节点身份的，这种只能清理掉该节点kafka的物理数据，重新加入。

回复
回答于 3年前
半兽人
痕☆ -> 半兽人 3年前
  1. 节点之间的hostname均可识别互通，这个是没问题的；
  2. /borkers/ids 自动删除后，没有自动恢复；

【例如迁移分区会产生大量的网络io需要进行限流】这个限流指的是kafka消费和生产的数据流吗？
请问下，一般通过什么配置可以达到效果？

0 0 回复
半兽人 -> 痕☆ 3年前

参考：kafka在数据迁移期间限制带宽的使用

0 0 回复
痕☆ -> 半兽人 3年前

您好，这边因为业务数据不太重要，昨天这边把kafka数据和zookeeper中的kafka相关配置，全部清理了，重启后，依然存在相关问题。
现在有个现象，这边kafka的下游是storm集群实时消费。
看了下kafka和storm的内网带宽监控趋势，
kafka平均出入内网带宽在20M/s, 峰值在70~80M/s，之后就趋于较低值100K/s，应该是自动停掉了。
同时storm也不再消费了。

不太清楚是storm消费流量太大导致kafka停掉，还是因为kafka自身流量问题被停掉。

0 0 回复
半兽人 -> 痕☆ 3年前

看看系统日志，看看那个时间点kafka节点是不是被kill了

/var/log/message
0 0 回复
痕☆ -> 半兽人 3年前

您好，查了一下3台机器的 /var/log/message 没有看见被 killed的记录

0 0 回复
半兽人 -> 痕☆ 3年前

就奇怪了，进程还在，timeout的kafka节点是会自愈。

1、资源（带宽）一直被占用？
2、系统负载很高？

或者你是否有更多可疑的日志提供。

0 0 回复
痕☆ -> 半兽人 3年前

这边kafka集群是和solr集群共用的服务器资源，相关监控指标都在范围内，不知是否有影响？
已将监控信息和日志贴到主贴后方，麻烦您看下。多谢

0 0 回复
痕☆ -> 半兽人 3年前

大佬，您好。
刚看了下kafka，正在运行中，三台占用的cpu都在80%左右。
想考虑升级kafka和zookeeper版本，请问有推荐的稳定版本吗？

0 0 回复
痕☆ -> 半兽人 3年前

您好，今天将kafka和zookeeper的版本进行了升级，分别升级到2.3.1和3.4.14，刚才偶发性的又掉了一个节点，现象和之前一样，进程在，brokers/ids 掉了,真是奇怪了。

以下是某个节点的配置，麻烦您帮忙看看配置是否有误。

broker.id=3
host.name=DC-kafka-3
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/kafka/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
#log.flush.interval.ms=1000
log.retention.hours=72
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
offsets.topic.replication.factor=1
zookeeper.connect=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka
zookeeper.connection.timeout.ms=60000
zookeeper.session.timeout.ms=90000

message.max.bytes=2048000
replica.fetch.max.bytes=2048576
default.replication.factor=3
0 0 回复
半兽人 -> 痕☆ 3年前

你这个新环境也掉，网络是不是防火墙或者什么白名单。
或者你的数据没有清理干净，比如zk多节点没有全停了之后清理？

0 0 回复
痕☆ -> 半兽人 3年前
  1. 防火墙都关闭了，不然启动就会报错，一般都是跑一会才会掉；
  2. 清理数据，是先停kafka，然后删掉zookeper中的kafka节点信息，然后清理kafka日志目录；

和客户端，比如生产者的客户端版本有关系吗？比如springboot中，这个版本没有升级，因为我看数据是正常写入到kafka中的。

0 0 回复
半兽人 -> 痕☆ 3年前
防火墙

我是基于这个报错怀疑的：

Connection to 2 was disconnected before the response was read
在读取响应之前，断开了对2的连接

跑一会儿会掉，也有一种情况是防火墙杀长连接的策略，关闭了就不关注这个了。

生产者的客户端版本

这不会的，客户端发送的数据，不会影响到kafka集群的，只会影响消费者。

0 0 回复
痕☆ -> 半兽人 3年前

防火墙都是关闭了的。
刚和同事沟通，说是再升级一下kafka版本。

之前版本和现在版本都出现问题，感觉和版本关系不大，挺奇怪的。

我的配置文件里配置的是host.name,我看很多都是配置listeners和advertised.listeners。不知是否有影响

0 0 回复
半兽人 -> 痕☆ 3年前

你hostname映射没问题的情况下是ok，建议换掉。

host.name介绍：

已弃用：当advertised.listeners或listeners没设置时候才使用。请改用advertised.listeners。Hostname发布到Zookeeper供客户端使用。在IaaS环境中，Broker可能需要绑定不同的接口。如果没有设置，将会使用host.name（如果配置了）。否则将从java.net.InetAddress.getCanonicalHostName()获取。

参考来自：Kafka Broker配置

0 0 回复
痕☆ -> 半兽人 3年前

  1. 我设置的机器别名格式为：dc-kafka-N ，配置的也是这个，我看启动kafka的时候，获取的host.name 如下：
    获取的是阿里云自定义的主机名，这个主机名只在本机hosts里配置了

    [2022-06-25 09:19:31,965] INFO Client environment:host.name=iZ23p29i***** (org.apache.zookeeper.ZooKeeper)
[2022-06-25 09:19:31,965] INFO Client environment:java.version=1.8.0_321 (org.apache.zookeeper.ZooKeeper)

  2. 今天又发现个现象。zookeeper三个节点其中一个节点挂了，通过可视化工具客户端无法访问，但是在该节点服务器通过./zkCli.sh status 查看正常，并且可以进去命令行查看数据。

0 0 回复
半兽人 -> 痕☆ 3年前

换成具体的ip吗？现在怎么样了。

0 0 回复
痕☆ -> 半兽人 3年前

您好。做了如下一些调整：

  1. 升级了相关kafka的版本（目前是2.8.1）、zookeeper的版本（目前是3.5.9）、生产者springboot中的相关客户端版本；
  2. 原来kafka是和其他集群共用资源，目前是独享资源；

比较遗憾，上述操作还是没能解决kafka集群节点假死的现象，但是有所缓解，目前不影响生产者写入数据。

近两天发现个问题，由于使用zookeeper的服务比较多，包括kafka、solr、hadoop、spark、storm等。
运行一段时间后，zookeeper就会持续报超时等问题，重启后就恢复。

猜想，会不会是因为使用zookeeper的服务较多，某个服务GC时间较长，使zookeeper不稳定，从而导致kafka节点注册自行消失。
请问下，从哪些日志内容或异常能看出问题来?有没有什么排查手段

0 0 回复
半兽人 -> 痕☆ 3年前

持续报超时，是别人连接zk报的，还是zk自己报的？

0 0 回复
痕☆ -> 半兽人 3年前

是zookeeper自己报的。

0 0 回复

这边kafka集群是和solr集群共用的服务器资源，相关监控指标都在范围内，不知是否有影响？
迁移阿里云之前挺稳定的，从经典网络变为专有网络后，问题不断。

回复
回答于 3年前
痕☆
你的答案

查看kafka相关的其他问题或提一个您自己的问题。
提问

找不到想要的答案？提一个您自己的问题。