kafka集群在zookeeper中注册的/borkers/ids 自动删除了,但kafka进程存在

痕☆ 发表于: 2022-06-22   最后更新时间: 2022-06-23 11:05:01   243 游览

请教一个关于kafka集群的问题,困扰了好几天了,仍未解决。

之前使用阿里云的经典网络服务器,部署了kafka服务(3个节点)和zookeeper服务(节点),应阿里云迁移要求,从经典网络迁移到他们的专有网络。

迁移完成后重启服务,kafka集群在zookeeper中注册的/borkers/ids经常自动运行一段时间后,/borkers/ids注册信息自动就没了,但是节点服务器上kakfa进程都在,3个节点中有时2个节点注册信息都会掉。

之前kafka挺稳定的,就是迁移以后不稳定,节点一直掉,导致服务异常。

kafka版本:0.8.2.1
zookeeper版本:3.4.6

我这边做过如下处理:

  1. 修改过kafka配置文件中的zookeeper相关超时时间;

  2. 之前谷歌发现有人有出现过相同问题,原因是说0.8.2.1存在controller相关的bug,在0.9版本以后修复,于是在6月21日对kafka进行了版本升级,升级到0.10.0.1;

  3. 查看服务器相关监控指标,基本都在控制范围内,都是内网交互,偶尔内网带宽会高一点;

以上处理方式并没有解决问题,今日发现异常仍然存在。

以下是日志中的错误日志

java.io.IOException: Connection to 0 was disconnected before the response was read
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
    at scala.Option.foreach(Option.scala:257)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
    at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-22 04:29:24,946] WARN [ReplicaFetcherThread-0-0], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5d9da6c4 (kafka.server.ReplicaFetcherThread)
java.net.SocketTimeoutException: Failed to connect within 30000 ms
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:240)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-22 04:29:36,017] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)

以下是某个节点的配置,当前版本是0.10.0.1:

broker.id=0
host.name=DC-Kafka-1
port=9092
num.network.threads=4
num.io.threads=8
socket.send.buffer.bytes=1024000
socket.receive.buffer.bytes=1024000
socket.request.max.bytes=104857600
log.dirs=/home/kafka/kafka-logs/broker-0
num.partitions=4
num.recovery.threads.per.data.dir=1
#log.flush.interval.ms=1000
log.retention.hours=168
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
zookeeper.connect=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka
zookeeper.connection.timeout.ms=60000
zookeeper.session.timeout.ms=120000

3个节点服务器,kafka日志存储目录大小均在23G左右。
会不会是因为kafka日志数据大,节点之间数据同步导致超时,这只是猜测。

已经出现好几天了,没有排查头绪,请大佬帮忙梳理下。

非常感谢!

以下是服务器监控信息和日志。
screenshot

[2022-06-23 00:05:20,718] INFO KafkaConfig values: 
    advertised.host.name = null
    metric.reporters = []
    quota.producer.default = 9223372036854775807
    offsets.topic.num.partitions = 50
    log.flush.interval.messages = 9223372036854775807
    auto.create.topics.enable = true
    controller.socket.timeout.ms = 30000
    log.flush.interval.ms = null
    principal.builder.class = class org.apache.kafka.common.security.auth.DefaultPrincipalBuilder
    replica.socket.receive.buffer.bytes = 65536
    min.insync.replicas = 1
    replica.fetch.wait.max.ms = 500
    num.recovery.threads.per.data.dir = 1
    ssl.keystore.type = JKS
    sasl.mechanism.inter.broker.protocol = GSSAPI
    default.replication.factor = 3
    ssl.truststore.password = null
    log.preallocate = false
    sasl.kerberos.principal.to.local.rules = [DEFAULT]
    fetch.purgatory.purge.interval.requests = 1000
    ssl.endpoint.identification.algorithm = null
    replica.socket.timeout.ms = 30000
    message.max.bytes = 2048000
    num.io.threads = 10
    offsets.commit.required.acks = -1
    log.flush.offset.checkpoint.interval.ms = 60000
    delete.topic.enable = false
    quota.window.size.seconds = 1
    ssl.truststore.type = JKS
    offsets.commit.timeout.ms = 5000
    quota.window.num = 11
    zookeeper.connect = DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka
    authorizer.class.name = 
    num.replica.fetchers = 1
    log.retention.ms = null
    log.roll.jitter.hours = 0
    log.cleaner.enable = false
    offsets.load.buffer.size = 5242880
    log.cleaner.delete.retention.ms = 86400000
    ssl.client.auth = none
    controlled.shutdown.max.retries = 3
    queued.max.requests = 500
    offsets.topic.replication.factor = 1
    log.cleaner.threads = 1
    sasl.kerberos.service.name = null
    sasl.kerberos.ticket.renew.jitter = 0.05
    socket.request.max.bytes = 104857600
    ssl.trustmanager.algorithm = PKIX
    zookeeper.session.timeout.ms = 90000
    log.retention.bytes = -1
    log.message.timestamp.type = CreateTime
    sasl.kerberos.min.time.before.relogin = 60000
    zookeeper.set.acl = false
    connections.max.idle.ms = 600000
    offsets.retention.minutes = 1440
    replica.fetch.backoff.ms = 1000
    inter.broker.protocol.version = 0.10.0-IV1
    log.retention.hours = 72
    num.partitions = 6
    broker.id.generation.enable = true
    listeners = null
    ssl.provider = null
    ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
    log.roll.ms = null
    log.flush.scheduler.interval.ms = 9223372036854775807
    ssl.cipher.suites = null
    log.index.size.max.bytes = 10485760
    ssl.keymanager.algorithm = SunX509
    security.inter.broker.protocol = PLAINTEXT
    replica.fetch.max.bytes = 2048576
    advertised.port = null
    log.cleaner.dedupe.buffer.size = 134217728
    replica.high.watermark.checkpoint.interval.ms = 5000
    log.cleaner.io.buffer.size = 524288
    sasl.kerberos.ticket.renew.window.factor = 0.8
    zookeeper.connection.timeout.ms = 60000
    controlled.shutdown.retry.backoff.ms = 5000
    log.roll.hours = 168
    log.cleanup.policy = delete
    host.name = DC-Kafka-1
    log.roll.jitter.ms = null
    max.connections.per.ip = 2147483647
    offsets.topic.segment.bytes = 104857600
    background.threads = 10
    quota.consumer.default = 9223372036854775807
    request.timeout.ms = 30000
    log.message.format.version = 0.10.0-IV1
    log.index.interval.bytes = 4096
    log.dir = /tmp/kafka-logs
    log.segment.bytes = 1073741824
    log.cleaner.backoff.ms = 15000
    offset.metadata.max.bytes = 4096
    ssl.truststore.location = null
    group.max.session.timeout.ms = 300000
    ssl.keystore.password = null
    zookeeper.sync.time.ms = 2000
    port = 9092
    log.retention.minutes = null
    log.segment.delete.delay.ms = 60000
    log.dirs = /home/kafka/kafka-logs
    controlled.shutdown.enable = true
    compression.type = producer
    max.connections.per.ip.overrides = 
    log.message.timestamp.difference.max.ms = 9223372036854775807
    sasl.kerberos.kinit.cmd = /usr/bin/kinit
    log.cleaner.io.max.bytes.per.second = 1.7976931348623157E308
    auto.leader.rebalance.enable = true
    leader.imbalance.check.interval.seconds = 300
    log.cleaner.min.cleanable.ratio = 0.5
    replica.lag.time.max.ms = 10000
    num.network.threads = 3
    ssl.key.password = null
    reserved.broker.max.id = 1000
    metrics.num.samples = 2
    socket.send.buffer.bytes = 102400
    ssl.protocol = TLS
    socket.receive.buffer.bytes = 102400
    ssl.keystore.location = null
    replica.fetch.min.bytes = 1
    broker.rack = null
    unclean.leader.election.enable = true
    sasl.enabled.mechanisms = [GSSAPI]
    group.min.session.timeout.ms = 6000
    log.cleaner.io.buffer.load.factor = 0.9
    offsets.retention.check.interval.ms = 600000
    producer.purgatory.purge.interval.requests = 1000
    metrics.sample.window.ms = 30000
    broker.id = 0
    offsets.topic.compression.codec = 0
    log.retention.check.interval.ms = 300000
    advertised.listeners = null
    leader.imbalance.per.broker.percentage = 10
 (kafka.server.KafkaConfig)
[2022-06-23 00:05:20,796] INFO starting (kafka.server.KafkaServer)
[2022-06-23 00:05:20,803] INFO Connecting to zookeeper on DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka (kafka.server.KafkaServer)
[2022-06-23 00:05:20,818] INFO Client environment:zookeeper.version=3.4.6-1569965, built on 02/20/2014 09:09 GMT (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:host.name=iZx3rbxxxxxx (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.version=1.8.0_321 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.home=/usr/java/jdk1.8.0_321/jre (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.class.path=.:/usr/java/jdk1.8.0_321/lib:/usr/java/jdk1.8.0_321/jre/lib:.:/usr/java/jdk1.8.0_321/lib:/usr/java/jdk1.8.0_321/jre/lib::/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/aopalliance-repackaged-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/argparse4j-0.5.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-api-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-file-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-json-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/connect-runtime-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/guava-18.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/hk2-api-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/hk2-locator-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/hk2-utils-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-annotations-2.6.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-core-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-databind-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-base-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-jaxrs-json-provider-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jackson-module-jaxb-annotations-2.6.3.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javassist-3.18.2-GA.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.annotation-api-1.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.inject-2.4.0-b34.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.servlet-api-3.1.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/javax.ws.rs-api-2.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-client-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-common-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-container-servlet-core-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-guava-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-media-jaxb-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jersey-server-2.22.2.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-continuation-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-http-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-io-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-security-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-server-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlet-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-servlets-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jetty-util-9.2.15.v20160210.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/jopt-simple-4.9.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-sources.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka_2.11-0.10.0.1-test-sources.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-clients-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-log4j-appender-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-streams-examples-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/kafka-tools-0.10.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/log4j-1.2.17.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/lz4-1.3.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/metrics-core-2.2.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/osgi-resource-locator-1.0.1.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/reflections-0.9.10.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/rocksdbjni-4.8.0.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/scala-library-2.11.8.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/scala-parser-combinators_2.11-1.0.4.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/slf4j-api-1.7.21.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/slf4j-log4j12-1.7.21.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/snappy-java-1.1.2.6.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/validation-api-1.1.0.Final.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/zkclient-0.8.jar:/home/kafka/kafka_2.11-0.10.0.1/bin/../libs/zookeeper-3.4.6.jar (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.io.tmpdir=/tmp (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:os.name=Linux (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:os.version=3.10.0-123.9.3.el7.x86_64 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:user.name=kafka (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:user.home=/home/kafka (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,818] INFO Client environment:user.dir=/home/kafka/kafka_2.11-0.10.0.1/config (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,819] INFO Initiating client connection, connectString=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181 sessionTimeout=90000 watcher=org.I0Itec.zkclient.ZkClient@5119fb47 (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,826] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2022-06-23 00:05:20,859] INFO Opening socket connection to server DC-Ice-2/10.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,862] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:20,862] INFO Socket connection established to DC-Ice-2/10.x.x.x:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,874] INFO Session establishment complete on server DC-Ice-2/10.x.x.x:2181, sessionid = 0x2818b2b837c0139, negotiated timeout = 90000 (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,875] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:20,884] INFO Created zookeeper path /kafka (kafka.server.KafkaServer)
[2022-06-23 00:05:20,884] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2022-06-23 00:05:20,889] INFO Session: 0x2818b2b837c0139 closed (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,889] INFO EventThread shut down (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,898] INFO Initiating client connection, connectString=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka sessionTimeout=90000 watcher=org.I0Itec.zkclient.ZkClient@6356695f (org.apache.zookeeper.ZooKeeper)
[2022-06-23 00:05:20,898] INFO Starting ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[2022-06-23 00:05:20,905] INFO Opening socket connection to server DC-Storm-3/10.x.x.x:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,906] INFO Socket connection established to DC-Storm-3/10.x.x.x:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,908] INFO Session establishment complete on server DC-Storm-3/10.x.x.x:2181, sessionid = 0x3818b2b83980100, negotiated timeout = 90000 (org.apache.zookeeper.ClientCnxn)
[2022-06-23 00:05:20,910] INFO Waiting for keeper state SyncConnected (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:20,910] INFO zookeeper state changed (SyncConnected) (org.I0Itec.zkclient.ZkClient)
[2022-06-23 00:05:21,142] INFO Loading logs. (kafka.log.LogManager)
[2022-06-23 00:05:21,207] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,222] INFO Recovering unflushed segment 0 in log Gxky_rec-2. (kafka.log.Log)
[2022-06-23 00:05:21,228] INFO Completed load of log Gxky_rec-2 with log end offset 17 (kafka.log.Log)
[2022-06-23 00:05:21,239] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,305] INFO Recovering unflushed segment 0 in log Gqxrmt_data-1. (kafka.log.Log)
[2022-06-23 00:05:21,314] INFO Completed load of log Gqxrmt_data-1 with log end offset 1677 (kafka.log.Log)
[2022-06-23 00:05:21,320] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,320] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-3. (kafka.log.Log)
[2022-06-23 00:05:21,323] INFO Completed load of log Gqxrmt_rec-3 with log end offset 0 (kafka.log.Log)
[2022-06-23 00:05:21,327] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_data-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,336] INFO Recovering unflushed segment 0 in log Gxky_data-3. (kafka.log.Log)
[2022-06-23 00:05:21,340] INFO Completed load of log Gxky_data-3 with log end offset 332 (kafka.log.Log)
[2022-06-23 00:05:21,347] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_event-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,634] INFO Recovering unflushed segment 0 in log Gqxrmt_event-5. (kafka.log.Log)
[2022-06-23 00:05:21,858] INFO Completed load of log Gqxrmt_event-5 with log end offset 76930 (kafka.log.Log)
[2022-06-23 00:05:21,866] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_rec-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:21,868] INFO Recovering unflushed segment 0 in log Gdsjb_rec-2. (kafka.log.Log)
[2022-06-23 00:05:21,869] INFO Completed load of log Gdsjb_rec-2 with log end offset 173 (kafka.log.Log)
[2022-06-23 00:05:21,878] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_event-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:22,670] INFO Recovering unflushed segment 0 in log Gdsjb_event-3. (kafka.log.Log)
[2022-06-23 00:05:23,395] INFO Completed load of log Gdsjb_event-3 with log end offset 460728 (kafka.log.Log)
[2022-06-23 00:05:23,405] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:23,408] INFO Recovering unflushed segment 0 in log Gxky_rec-1. (kafka.log.Log)
[2022-06-23 00:05:23,409] INFO Completed load of log Gxky_rec-1 with log end offset 13 (kafka.log.Log)
[2022-06-23 00:05:23,411] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:23,412] INFO Recovering unflushed segment 0 in log Gxky_rec-4. (kafka.log.Log)
[2022-06-23 00:05:23,412] INFO Completed load of log Gxky_rec-4 with log end offset 27 (kafka.log.Log)
[2022-06-23 00:05:23,416] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_event-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:23,624] INFO Recovering unflushed segment 0 in log Gqxrmt_event-1. (kafka.log.Log)
[2022-06-23 00:05:23,765] INFO Completed load of log Gqxrmt_event-1 with log end offset 76297 (kafka.log.Log)
[2022-06-23 00:05:23,774] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_data-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:23,776] INFO Recovering unflushed segment 0 in log Gdsjb_data-5. (kafka.log.Log)
[2022-06-23 00:05:23,778] INFO Completed load of log Gdsjb_data-5 with log end offset 155 (kafka.log.Log)
[2022-06-23 00:05:23,781] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_event-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:23,925] INFO Recovering unflushed segment 0 in log Gqxrmt_event-4. (kafka.log.Log)
[2022-06-23 00:05:24,042] INFO Completed load of log Gqxrmt_event-4 with log end offset 76775 (kafka.log.Log)
[2022-06-23 00:05:24,049] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:24,053] INFO Recovering unflushed segment 0 in log Gqxrmt_data-0. (kafka.log.Log)
[2022-06-23 00:05:24,058] INFO Completed load of log Gqxrmt_data-0 with log end offset 1625 (kafka.log.Log)
[2022-06-23 00:05:24,073] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_data-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:24,075] INFO Recovering unflushed segment 0 in log Gdsjb_data-3. (kafka.log.Log)
[2022-06-23 00:05:24,077] INFO Completed load of log Gdsjb_data-3 with log end offset 158 (kafka.log.Log)
[2022-06-23 00:05:24,085] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_rec-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:24,088] INFO Recovering unflushed segment 0 in log Gdsjb_rec-1. (kafka.log.Log)
[2022-06-23 00:05:24,089] INFO Completed load of log Gdsjb_rec-1 with log end offset 165 (kafka.log.Log)
[2022-06-23 00:05:24,092] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_data-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:24,093] INFO Recovering unflushed segment 0 in log Gxky_data-0. (kafka.log.Log)
[2022-06-23 00:05:24,094] INFO Completed load of log Gxky_data-0 with log end offset 284 (kafka.log.Log)
[2022-06-23 00:05:24,101] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_event-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:25,060] INFO Recovering unflushed segment 0 in log Gdsjb_event-1. (kafka.log.Log)
[2022-06-23 00:05:25,766] INFO Completed load of log Gdsjb_event-1 with log end offset 479871 (kafka.log.Log)
[2022-06-23 00:05:25,770] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_data-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:25,771] INFO Recovering unflushed segment 0 in log Gdsjb_data-2. (kafka.log.Log)
[2022-06-23 00:05:25,772] INFO Completed load of log Gdsjb_data-2 with log end offset 180 (kafka.log.Log)
[2022-06-23 00:05:25,774] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_rec-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:25,775] INFO Recovering unflushed segment 0 in log Gdsjb_rec-4. (kafka.log.Log)
[2022-06-23 00:05:25,776] INFO Completed load of log Gdsjb_rec-4 with log end offset 149 (kafka.log.Log)
[2022-06-23 00:05:25,785] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:25,790] INFO Recovering unflushed segment 0 in log Gqxrmt_data-4. (kafka.log.Log)
[2022-06-23 00:05:25,795] INFO Completed load of log Gqxrmt_data-4 with log end offset 1579 (kafka.log.Log)
[2022-06-23 00:05:25,799] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_data-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:25,800] INFO Recovering unflushed segment 0 in log Gdsjb_data-1. (kafka.log.Log)
[2022-06-23 00:05:25,800] INFO Completed load of log Gdsjb_data-1 with log end offset 138 (kafka.log.Log)
[2022-06-23 00:05:25,805] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_event-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:25,917] INFO Recovering unflushed segment 0 in log Gqxrmt_event-3. (kafka.log.Log)
[2022-06-23 00:05:26,048] INFO Completed load of log Gqxrmt_event-3 with log end offset 76895 (kafka.log.Log)
[2022-06-23 00:05:26,052] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,052] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-1. (kafka.log.Log)
[2022-06-23 00:05:26,053] INFO Completed load of log Gqxrmt_rec-1 with log end offset 1 (kafka.log.Log)
[2022-06-23 00:05:26,062] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,067] INFO Recovering unflushed segment 0 in log Gqxrmt_data-3. (kafka.log.Log)
[2022-06-23 00:05:26,072] INFO Completed load of log Gqxrmt_data-3 with log end offset 1542 (kafka.log.Log)
[2022-06-23 00:05:26,094] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_event-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,234] INFO Recovering unflushed segment 0 in log Gxky_event-5. (kafka.log.Log)
[2022-06-23 00:05:26,330] INFO Completed load of log Gxky_event-5 with log end offset 48118 (kafka.log.Log)
[2022-06-23 00:05:26,336] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_data-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,337] INFO Recovering unflushed segment 0 in log Gdsjb_data-4. (kafka.log.Log)
[2022-06-23 00:05:26,338] INFO Completed load of log Gdsjb_data-4 with log end offset 172 (kafka.log.Log)
[2022-06-23 00:05:26,342] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_data-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,344] INFO Recovering unflushed segment 0 in log Gxky_data-2. (kafka.log.Log)
[2022-06-23 00:05:26,345] INFO Completed load of log Gxky_data-2 with log end offset 403 (kafka.log.Log)
[2022-06-23 00:05:26,347] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_data-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,348] INFO Recovering unflushed segment 0 in log Gxky_data-1. (kafka.log.Log)
[2022-06-23 00:05:26,349] INFO Completed load of log Gxky_data-1 with log end offset 323 (kafka.log.Log)
[2022-06-23 00:05:26,351] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,358] INFO Recovering unflushed segment 0 in log Gqxrmt_data-5. (kafka.log.Log)
[2022-06-23 00:05:26,363] INFO Completed load of log Gqxrmt_data-5 with log end offset 1673 (kafka.log.Log)
[2022-06-23 00:05:26,370] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,370] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-2. (kafka.log.Log)
[2022-06-23 00:05:26,371] INFO Completed load of log Gqxrmt_rec-2 with log end offset 0 (kafka.log.Log)
[2022-06-23 00:05:26,373] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_rec-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,375] INFO Recovering unflushed segment 0 in log Gdsjb_rec-3. (kafka.log.Log)
[2022-06-23 00:05:26,376] INFO Completed load of log Gdsjb_rec-3 with log end offset 179 (kafka.log.Log)
[2022-06-23 00:05:26,378] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_event-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,476] INFO Recovering unflushed segment 0 in log Gxky_event-4. (kafka.log.Log)
[2022-06-23 00:05:26,597] INFO Completed load of log Gxky_event-4 with log end offset 47766 (kafka.log.Log)
[2022-06-23 00:05:26,604] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_event-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,687] INFO Recovering unflushed segment 0 in log Gxky_event-2. (kafka.log.Log)
[2022-06-23 00:05:26,764] INFO Completed load of log Gxky_event-2 with log end offset 47991 (kafka.log.Log)
[2022-06-23 00:05:26,766] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_event-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,876] INFO Recovering unflushed segment 0 in log Gqxrmt_event-0. (kafka.log.Log)
[2022-06-23 00:05:26,987] INFO Completed load of log Gqxrmt_event-0 with log end offset 76454 (kafka.log.Log)
[2022-06-23 00:05:26,992] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:26,993] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-0. (kafka.log.Log)
[2022-06-23 00:05:26,999] INFO Completed load of log Gqxrmt_rec-0 with log end offset 0 (kafka.log.Log)
[2022-06-23 00:05:27,000] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:27,001] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-5. (kafka.log.Log)
[2022-06-23 00:05:27,001] INFO Completed load of log Gqxrmt_rec-5 with log end offset 1 (kafka.log.Log)
[2022-06-23 00:05:27,003] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_data-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:27,004] INFO Recovering unflushed segment 0 in log Gxky_data-5. (kafka.log.Log)
[2022-06-23 00:05:27,005] INFO Completed load of log Gxky_data-5 with log end offset 316 (kafka.log.Log)
[2022-06-23 00:05:27,006] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:27,007] INFO Recovering unflushed segment 0 in log Gxky_rec-3. (kafka.log.Log)
[2022-06-23 00:05:27,007] INFO Completed load of log Gxky_rec-3 with log end offset 16 (kafka.log.Log)
[2022-06-23 00:05:27,014] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_event-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:27,670] INFO Recovering unflushed segment 0 in log Gdsjb_event-4. (kafka.log.Log)
[2022-06-23 00:05:28,379] INFO Completed load of log Gdsjb_event-4 with log end offset 459323 (kafka.log.Log)
[2022-06-23 00:05:28,381] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_rec-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:28,382] INFO Recovering unflushed segment 0 in log Gdsjb_rec-5. (kafka.log.Log)
[2022-06-23 00:05:28,383] INFO Completed load of log Gdsjb_rec-5 with log end offset 168 (kafka.log.Log)
[2022-06-23 00:05:28,384] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_event-1/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:28,462] INFO Recovering unflushed segment 0 in log Gxky_event-1. (kafka.log.Log)
[2022-06-23 00:05:28,540] INFO Completed load of log Gxky_event-1 with log end offset 48289 (kafka.log.Log)
[2022-06-23 00:05:28,541] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_event-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:29,470] INFO Recovering unflushed segment 0 in log Gdsjb_event-2. (kafka.log.Log)
[2022-06-23 00:05:30,158] INFO Completed load of log Gdsjb_event-2 with log end offset 477717 (kafka.log.Log)
[2022-06-23 00:05:30,160] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_data-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,161] INFO Recovering unflushed segment 0 in log Gdsjb_data-0. (kafka.log.Log)
[2022-06-23 00:05:30,162] INFO Completed load of log Gdsjb_data-0 with log end offset 140 (kafka.log.Log)
[2022-06-23 00:05:30,163] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,164] INFO Recovering unflushed segment 0 in log Gxky_rec-0. (kafka.log.Log)
[2022-06-23 00:05:30,171] INFO Completed load of log Gxky_rec-0 with log end offset 16 (kafka.log.Log)
[2022-06-23 00:05:30,174] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_event-3/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,263] INFO Recovering unflushed segment 0 in log Gxky_event-3. (kafka.log.Log)
[2022-06-23 00:05:30,338] INFO Completed load of log Gxky_event-3 with log end offset 48274 (kafka.log.Log)
[2022-06-23 00:05:30,339] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_rec-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,341] INFO Recovering unflushed segment 0 in log Gqxrmt_rec-4. (kafka.log.Log)
[2022-06-23 00:05:30,341] INFO Completed load of log Gqxrmt_rec-4 with log end offset 0 (kafka.log.Log)
[2022-06-23 00:05:30,343] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_rec-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,344] INFO Recovering unflushed segment 0 in log Gxky_rec-5. (kafka.log.Log)
[2022-06-23 00:05:30,344] INFO Completed load of log Gxky_rec-5 with log end offset 13 (kafka.log.Log)
[2022-06-23 00:05:30,345] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_event-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,460] INFO Recovering unflushed segment 0 in log Gqxrmt_event-2. (kafka.log.Log)
[2022-06-23 00:05:30,573] INFO Completed load of log Gqxrmt_event-2 with log end offset 77284 (kafka.log.Log)
[2022-06-23 00:05:30,575] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_event-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,654] INFO Recovering unflushed segment 0 in log Gxky_event-0. (kafka.log.Log)
[2022-06-23 00:05:30,732] INFO Completed load of log Gxky_event-0 with log end offset 48075 (kafka.log.Log)
[2022-06-23 00:05:30,733] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_rec-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:30,734] INFO Recovering unflushed segment 0 in log Gdsjb_rec-0. (kafka.log.Log)
[2022-06-23 00:05:30,734] INFO Completed load of log Gdsjb_rec-0 with log end offset 160 (kafka.log.Log)
[2022-06-23 00:05:30,735] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_event-5/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:31,505] INFO Recovering unflushed segment 0 in log Gdsjb_event-5. (kafka.log.Log)
[2022-06-23 00:05:32,168] INFO Completed load of log Gdsjb_event-5 with log end offset 462478 (kafka.log.Log)
[2022-06-23 00:05:32,170] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gqxrmt_data-2/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:32,180] INFO Recovering unflushed segment 0 in log Gqxrmt_data-2. (kafka.log.Log)
[2022-06-23 00:05:32,184] INFO Completed load of log Gqxrmt_data-2 with log end offset 1626 (kafka.log.Log)
[2022-06-23 00:05:32,185] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gdsjb_event-0/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:32,932] INFO Recovering unflushed segment 0 in log Gdsjb_event-0. (kafka.log.Log)
[2022-06-23 00:05:33,681] INFO Completed load of log Gdsjb_event-0 with log end offset 461404 (kafka.log.Log)
[2022-06-23 00:05:33,689] WARN Found a corrupted index file, /home/kafka/kafka-logs/Gxky_data-4/00000000000000000000.index, deleting and rebuilding index... (kafka.log.Log)
[2022-06-23 00:05:33,690] INFO Recovering unflushed segment 0 in log Gxky_data-4. (kafka.log.Log)
[2022-06-23 00:05:33,691] INFO Completed load of log Gxky_data-4 with log end offset 294 (kafka.log.Log)
[2022-06-23 00:05:33,696] INFO Logs loading complete. (kafka.log.LogManager)
[2022-06-23 00:05:33,696] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2022-06-23 00:05:33,700] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2022-06-23 00:05:33,751] INFO Awaiting socket connections on DC-Kafka-1:9092. (kafka.network.Acceptor)
[2022-06-23 00:05:33,755] INFO [Socket Server on Broker 0], Started 1 acceptor threads (kafka.network.SocketServer)
[2022-06-23 00:05:33,798] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,798] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,842] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,843] INFO [ExpirationReaper-0], Starting  (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2022-06-23 00:05:33,863] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.GroupCoordinator)
[2022-06-23 00:05:33,863] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.GroupCoordinator)
[2022-06-23 00:05:33,889] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 16 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:05:33,908] INFO Will not load MX4J, mx4j-tools.jar is not in the classpath (kafka.utils.Mx4jLoader$)
[2022-06-23 00:05:33,913] INFO [ThrottledRequestReaper-Produce], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2022-06-23 00:05:33,913] INFO [ThrottledRequestReaper-Fetch], Starting  (kafka.server.ClientQuotaManager$ThrottledRequestReaper)
[2022-06-23 00:05:33,930] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:05:33,936] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:05:33,937] INFO Registered broker 0 at path /brokers/ids/0 with addresses: PLAINTEXT -> EndPoint(DC-Kafka-1,9092,PLAINTEXT) (kafka.utils.ZkUtils)
[2022-06-23 00:05:33,946] INFO Kafka version : 0.10.0.1 (org.apache.kafka.common.utils.AppInfoParser)
[2022-06-23 00:05:33,946] INFO Kafka commitId : a7a17cdec9eaa6c5 (org.apache.kafka.common.utils.AppInfoParser)
[2022-06-23 00:05:33,947] INFO [Kafka Server 0], started (kafka.server.KafkaServer)
[2022-06-23 00:05:34,336] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gqxrmt_data,0],[Gxky_event,5],[Gqxrmt_event,0],[Gqxrmt_rec,5],[Gxky_data,3],[Gdsjb_data,5],[Gdsjb_event,1],[Gdsjb_data,4],[Gxky_data,1],[Gdsjb_data,1],[Gdsjb_rec,1],[Gxky_event,1],[Gxky_data,4],[Gxky_rec,0],[Gxky_rec,4],[Gdsjb_rec,2],[Gxky_data,0],[Gdsjb_data,2],[Gqxrmt_event,2],[Gdsjb_rec,3],[Gqxrmt_data,2],[Gqxrmt_event,1],[Gqxrmt_data,4],[Gdsjb_rec,0],[Gqxrmt_data,3],[Gqxrmt_rec,1],[Gdsjb_event,5],[Gdsjb_rec,4],[Gdsjb_event,0],[Gqxrmt_rec,3],[Gxky_rec,3],[Gqxrmt_event,3],[Gdsjb_data,0],[Gxky_data,2],[Gxky_event,0],[Gdsjb_event,3],[Gxky_rec,5],[Gxky_data,5],[Gqxrmt_rec,2],[Gxky_event,2],[Gxky_event,3],[Gqxrmt_event,4],[Gxky_rec,1],[Gqxrmt_rec,0],[Gqxrmt_data,1],[Gdsjb_event,4],[Gqxrmt_rec,4],[Gqxrmt_data,5],[Gxky_event,4],[Gqxrmt_event,5],[Gxky_rec,2],[Gdsjb_event,2],[Gdsjb_rec,5],[Gdsjb_data,3] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:05:34,348] INFO Truncating log Gxky_event-5 to offset 48113. (kafka.log.Log)
[2022-06-23 00:05:34,352] INFO Truncating log Gxky_event-4 to offset 47766. (kafka.log.Log)
[2022-06-23 00:05:34,352] INFO Truncating log Gxky_rec-0 to offset 16. (kafka.log.Log)
[2022-06-23 00:05:34,353] INFO Truncating log Gdsjb_rec-3 to offset 179. (kafka.log.Log)
[2022-06-23 00:05:34,353] INFO Truncating log Gxky_data-4 to offset 294. (kafka.log.Log)
[2022-06-23 00:05:34,353] INFO Truncating log Gxky_data-1 to offset 323. (kafka.log.Log)
[2022-06-23 00:05:34,354] INFO Truncating log Gdsjb_rec-1 to offset 165. (kafka.log.Log)
[2022-06-23 00:05:34,360] INFO Truncating log Gxky_rec-5 to offset 13. (kafka.log.Log)
[2022-06-23 00:05:34,360] INFO Truncating log Gqxrmt_rec-2 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,361] INFO Truncating log Gqxrmt_data-0 to offset 1625. (kafka.log.Log)
[2022-06-23 00:05:34,361] INFO Truncating log Gqxrmt_rec-0 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,362] INFO Truncating log Gxky_rec-3 to offset 16. (kafka.log.Log)
[2022-06-23 00:05:34,362] INFO Truncating log Gxky_event-3 to offset 48270. (kafka.log.Log)
[2022-06-23 00:05:34,363] INFO Truncating log Gxky_data-5 to offset 316. (kafka.log.Log)
[2022-06-23 00:05:34,363] INFO Truncating log Gdsjb_event-5 to offset 462449. (kafka.log.Log)
[2022-06-23 00:05:34,364] INFO Truncating log Gqxrmt_data-1 to offset 1677. (kafka.log.Log)
[2022-06-23 00:05:34,364] INFO Truncating log Gdsjb_data-4 to offset 172. (kafka.log.Log)
[2022-06-23 00:05:34,365] INFO Truncating log Gxky_data-2 to offset 403. (kafka.log.Log)
[2022-06-23 00:05:34,365] INFO Truncating log Gqxrmt_rec-3 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,365] INFO Truncating log Gqxrmt_data-2 to offset 1626. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gqxrmt_rec-1 to offset 1. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gdsjb_event-0 to offset 461377. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gqxrmt_rec-4 to offset 0. (kafka.log.Log)
[2022-06-23 00:05:34,366] INFO Truncating log Gqxrmt_event-5 to offset 76927. (kafka.log.Log)
[2022-06-23 00:05:34,367] INFO Truncating log Gqxrmt_data-4 to offset 1579. (kafka.log.Log)
[2022-06-23 00:05:34,367] INFO Truncating log Gqxrmt_rec-5 to offset 1. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gxky_event-0 to offset 48069. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gqxrmt_event-3 to offset 76895. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gqxrmt_event-0 to offset 76454. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gdsjb_rec-2 to offset 173. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gxky_rec-1 to offset 13. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gqxrmt_data-5 to offset 1673. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gxky_rec-2 to offset 17. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gdsjb_rec-0 to offset 160. (kafka.log.Log)
[2022-06-23 00:05:34,368] INFO Truncating log Gdsjb_event-4 to offset 459323. (kafka.log.Log)
[2022-06-23 00:05:34,370] INFO Truncating log Gdsjb_data-2 to offset 180. (kafka.log.Log)
[2022-06-23 00:05:34,370] INFO Truncating log Gdsjb_data-3 to offset 158. (kafka.log.Log)
[2022-06-23 00:05:34,370] INFO Truncating log Gdsjb_data-0 to offset 140. (kafka.log.Log)
[2022-06-23 00:05:34,371] INFO Truncating log Gdsjb_data-1 to offset 138. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gqxrmt_event-4 to offset 76771. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gdsjb_rec-4 to offset 149. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gqxrmt_event-1 to offset 76290. (kafka.log.Log)
[2022-06-23 00:05:34,372] INFO Truncating log Gdsjb_event-2 to offset 477694. (kafka.log.Log)
[2022-06-23 00:05:34,375] INFO Truncating log Gdsjb_data-5 to offset 155. (kafka.log.Log)
[2022-06-23 00:05:34,375] INFO Truncating log Gxky_data-3 to offset 332. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gqxrmt_data-3 to offset 1542. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gdsjb_event-1 to offset 479871. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gqxrmt_event-2 to offset 77283. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gdsjb_rec-5 to offset 168. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gxky_event-2 to offset 47985. (kafka.log.Log)
[2022-06-23 00:05:34,376] INFO Truncating log Gxky_data-0 to offset 284. (kafka.log.Log)
[2022-06-23 00:05:34,377] INFO Truncating log Gxky_event-1 to offset 48289. (kafka.log.Log)
[2022-06-23 00:05:34,377] INFO Truncating log Gxky_rec-4 to offset 27. (kafka.log.Log)
[2022-06-23 00:05:34,378] INFO Truncating log Gdsjb_event-3 to offset 460706. (kafka.log.Log)
[2022-06-23 00:05:34,440] INFO [ReplicaFetcherThread-0-1], Starting  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:05:34,443] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List([[Gxky_event,5], initOffset 48113 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,4], initOffset 47766 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,0], initOffset 16 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,3], initOffset 179 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,4], initOffset 294 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,1], initOffset 323 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,1], initOffset 165 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,5], initOffset 13 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,2], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,0], initOffset 1625 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,0], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,3], initOffset 16 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,3], initOffset 48270 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,5], initOffset 316 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,5], initOffset 462449 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,1], initOffset 1677 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,4], initOffset 172 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,2], initOffset 403 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,3], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,2], initOffset 1626 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,1], initOffset 1 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,0], initOffset 461377 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,4], initOffset 0 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,5], initOffset 76927 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,4], initOffset 1579 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_rec,5], initOffset 1 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,0], initOffset 48069 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,3], initOffset 76895 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,0], initOffset 76454 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,2], initOffset 173 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,1], initOffset 13 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,5], initOffset 1673 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,2], initOffset 17 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,0], initOffset 160 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,4], initOffset 459323 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,2], initOffset 180 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,3], initOffset 158 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,0], initOffset 140 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,1], initOffset 138 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,4], initOffset 76771 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,4], initOffset 149 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,1], initOffset 76290 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,2], initOffset 477694 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_data,5], initOffset 155 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,3], initOffset 332 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_data,3], initOffset 1542 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,1], initOffset 479871 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gqxrmt_event,2], initOffset 77283 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_rec,5], initOffset 168 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,2], initOffset 47985 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_data,0], initOffset 284 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_event,1], initOffset 48289 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gxky_rec,4], initOffset 27 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] , [[Gdsjb_event,3], initOffset 460706 to broker BrokerEndPoint(1,DC-Kafka-2,9092)] ) (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:08:55,823] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@1de06723 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 1 was disconnected before the response was read
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
    at scala.Option.foreach(Option.scala:257)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
    at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:08:57,832] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@7613510c (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:08:59,833] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@29f8cd66 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:01,834] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@57118356 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:03,835] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4e1db3b4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:05,836] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@3a876b7d (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:07,838] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@7535ef7 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:09,839] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@750ade1c (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:11,840] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@c310d8f (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:13,841] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@f07dc28 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:15,842] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@50ccb09e (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:17,843] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@1af05b5c (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:19,844] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5fae1294 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:21,845] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@66a495d9 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:23,846] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@520387eb (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:25,847] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@585a57f6 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:27,849] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6cca8cc3 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:29,850] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@47427995 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:31,852] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4c474fa8 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:33,853] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4b64f2ae (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:35,855] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@537e0f91 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:37,856] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5b428657 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:39,857] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5ca4fe93 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:41,858] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@3d19569f (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:43,859] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@12df045f (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:45,860] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@294db8d6 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:47,861] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@d3d92f0 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:49,862] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@55b208d4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:51,863] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@651b7977 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:53,864] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@6f185982 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:55,865] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@48f48162 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:57,866] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@1165d906 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:09:59,867] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5d7ba0e4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:01,868] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@7ac35f90 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:03,869] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@425a9e00 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:05,870] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@305350de (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:07,871] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@64422cc8 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:09,872] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@36aa3132 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:11,873] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4f34873e (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:13,874] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5d689fab (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:15,876] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5f501790 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:17,877] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@5fb1edf4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:19,878] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@4b76979c (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:21,879] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@37cc1ff6 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:23,880] WARN [ReplicaFetcherThread-0-1], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@8951a88 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to DC-Kafka-2:9092 (id: 1 rack: null) failed
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:63)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingReady$extension$2.apply(NetworkClientBlockingOps.scala:59)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$1(NetworkClientBlockingOps.scala:112)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollUntil$extension(NetworkClientBlockingOps.scala:120)
    at kafka.utils.NetworkClientBlockingOps$.blockingReady$extension(NetworkClientBlockingOps.scala:59)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:239)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:10:24,012] INFO New leader is 2 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2022-06-23 00:10:24,499] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gxky_event,5],[Gxky_event,4],[Gdsjb_rec,3],[Gxky_data,4],[Gxky_rec,5],[Gqxrmt_data,1],[Gdsjb_data,4],[Gqxrmt_data,2],[Gqxrmt_rec,1],[Gqxrmt_rec,4],[Gqxrmt_data,4],[Gqxrmt_rec,5],[Gqxrmt_event,3],[Gqxrmt_event,0],[Gdsjb_rec,2],[Gxky_rec,1],[Gdsjb_event,4],[Gdsjb_data,2],[Gdsjb_data,1],[Gqxrmt_event,4],[Gdsjb_event,2],[Gxky_data,3],[Gdsjb_event,1],[Gdsjb_rec,5],[Gxky_data,0],[Gxky_event,1],[Gxky_rec,4] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:10:24,511] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gqxrmt_data,0],[Gdsjb_event,5],[Gdsjb_rec,4],[Gdsjb_data,5],[Gdsjb_event,0],[Gqxrmt_rec,3],[Gxky_data,1],[Gxky_data,2],[Gdsjb_event,3],[Gxky_rec,3],[Gdsjb_rec,1],[Gxky_event,0],[Gdsjb_data,0],[Gxky_data,5],[Gqxrmt_rec,2],[Gxky_event,2],[Gxky_event,3],[Gxky_rec,0],[Gqxrmt_rec,0],[Gqxrmt_data,5],[Gqxrmt_event,2],[Gqxrmt_event,5],[Gqxrmt_event,1],[Gdsjb_rec,0],[Gxky_rec,2],[Gqxrmt_data,3],[Gdsjb_data,3] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_rec-0 to offset 17. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_data-1 to offset 332. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gdsjb_rec-1 to offset 176. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_rec-2 to offset 0. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_data-0 to offset 1650. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_rec-0 to offset 0. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_rec-3 to offset 16. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_event-3 to offset 52068. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_data-5 to offset 333. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gdsjb_event-5 to offset 490190. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gxky_data-2 to offset 414. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_rec-3 to offset 0. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gdsjb_event-0 to offset 489031. (kafka.log.Log)
[2022-06-23 00:10:24,512] INFO Truncating log Gqxrmt_event-5 to offset 83926. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gxky_event-0 to offset 52018. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_data-5 to offset 1699. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gxky_rec-2 to offset 17. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_rec-0 to offset 171. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_data-3 to offset 159. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_data-0 to offset 140. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_rec-4 to offset 157. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_event-1 to offset 83328. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_data-5 to offset 156. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_data-3 to offset 1561. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gqxrmt_event-2 to offset 84475. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gxky_event-2 to offset 51758. (kafka.log.Log)
[2022-06-23 00:10:24,513] INFO Truncating log Gdsjb_event-3 to offset 488286. (kafka.log.Log)
[2022-06-23 00:10:24,522] INFO [ReplicaFetcherManager on broker 0] Added fetcher for partitions List([[Gxky_rec,0], initOffset 17 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_data,1], initOffset 332 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_rec,1], initOffset 176 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_rec,2], initOffset 0 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_data,0], initOffset 1650 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_rec,0], initOffset 0 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_rec,3], initOffset 16 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_event,3], initOffset 52068 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_data,5], initOffset 333 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_event,5], initOffset 490190 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_data,2], initOffset 414 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_rec,3], initOffset 0 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_event,0], initOffset 489031 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_event,5], initOffset 83926 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_event,0], initOffset 52018 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_data,5], initOffset 1699 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_rec,2], initOffset 17 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_rec,0], initOffset 171 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_data,3], initOffset 159 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_data,0], initOffset 140 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_rec,4], initOffset 157 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_event,1], initOffset 83328 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_data,5], initOffset 156 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_data,3], initOffset 1561 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gqxrmt_event,2], initOffset 84475 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gxky_event,2], initOffset 51758 to broker BrokerEndPoint(2,dc-solr-1,9092)] , [[Gdsjb_event,3], initOffset 488286 to broker BrokerEndPoint(2,dc-solr-1,9092)] ) (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:10:24,523] INFO [ReplicaFetcherThread-0-1], Shutting down (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:10:24,525] INFO [ReplicaFetcherThread-0-1], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:10:24,525] INFO [ReplicaFetcherThread-0-1], Stopped  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:10:24,530] INFO [ReplicaFetcherThread-0-2], Starting  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:15:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:25:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:26:15,288] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@7cab2945 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
    at scala.Option.foreach(Option.scala:257)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
    at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:26:23,785] INFO Partition [Gdsjb_data,4] on broker 0: Shrinking ISR for partition [Gdsjb_data,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,792] INFO Partition [Gdsjb_event,1] on broker 0: Shrinking ISR for partition [Gdsjb_event,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,794] INFO Partition [Gqxrmt_rec,5] on broker 0: Shrinking ISR for partition [Gqxrmt_rec,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,796] INFO Partition [Gxky_event,4] on broker 0: Shrinking ISR for partition [Gxky_event,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,798] INFO Partition [Gqxrmt_rec,1] on broker 0: Shrinking ISR for partition [Gqxrmt_rec,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,800] INFO Partition [Gdsjb_data,1] on broker 0: Shrinking ISR for partition [Gdsjb_data,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,802] INFO Partition [Gqxrmt_event,4] on broker 0: Shrinking ISR for partition [Gqxrmt_event,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,805] INFO Partition [Gdsjb_event,4] on broker 0: Shrinking ISR for partition [Gdsjb_event,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,808] INFO Partition [Gxky_data,4] on broker 0: Shrinking ISR for partition [Gxky_data,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,810] INFO Partition [Gqxrmt_data,4] on broker 0: Shrinking ISR for partition [Gqxrmt_data,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,811] INFO Partition [Gdsjb_event,2] on broker 0: Shrinking ISR for partition [Gdsjb_event,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,813] INFO Partition [Gdsjb_rec,2] on broker 0: Shrinking ISR for partition [Gdsjb_rec,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,815] INFO Partition [Gqxrmt_data,1] on broker 0: Shrinking ISR for partition [Gqxrmt_data,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,817] INFO Partition [Gqxrmt_data,2] on broker 0: Shrinking ISR for partition [Gqxrmt_data,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,819] INFO Partition [Gqxrmt_rec,4] on broker 0: Shrinking ISR for partition [Gqxrmt_rec,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,821] INFO Partition [Gxky_data,3] on broker 0: Shrinking ISR for partition [Gxky_data,3] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,822] INFO Partition [Gdsjb_rec,3] on broker 0: Shrinking ISR for partition [Gdsjb_rec,3] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,824] INFO Partition [Gxky_data,0] on broker 0: Shrinking ISR for partition [Gxky_data,0] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,826] INFO Partition [Gxky_rec,5] on broker 0: Shrinking ISR for partition [Gxky_rec,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,828] INFO Partition [Gxky_event,1] on broker 0: Shrinking ISR for partition [Gxky_event,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,830] INFO Partition [Gxky_rec,1] on broker 0: Shrinking ISR for partition [Gxky_rec,1] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,832] INFO Partition [Gqxrmt_event,3] on broker 0: Shrinking ISR for partition [Gqxrmt_event,3] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,834] INFO Partition [Gdsjb_rec,5] on broker 0: Shrinking ISR for partition [Gdsjb_rec,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,836] INFO Partition [Gdsjb_data,2] on broker 0: Shrinking ISR for partition [Gdsjb_data,2] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,837] INFO Partition [Gxky_event,5] on broker 0: Shrinking ISR for partition [Gxky_event,5] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,840] INFO Partition [Gxky_rec,4] on broker 0: Shrinking ISR for partition [Gxky_rec,4] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:23,841] INFO Partition [Gqxrmt_event,0] on broker 0: Shrinking ISR for partition [Gqxrmt_event,0] from 0,2 to 0 (kafka.cluster.Partition)
[2022-06-23 00:26:47,318] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@2d1e26f4 (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
    at scala.Option.foreach(Option.scala:257)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
    at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:27:19,349] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@361a4a63 (kafka.server.ReplicaFetcherThread)
java.net.SocketTimeoutException: Failed to connect within 30000 ms
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:240)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:27:35,533] WARN [ReplicaFetcherThread-0-2], Error in fetch kafka.server.ReplicaFetcherThread$FetchRequest@2cb8ae5b (kafka.server.ReplicaFetcherThread)
java.io.IOException: Connection to 2 was disconnected before the response was read
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:87)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1$$anonfun$apply$1.apply(NetworkClientBlockingOps.scala:84)
    at scala.Option.foreach(Option.scala:257)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:84)
    at kafka.utils.NetworkClientBlockingOps$$anonfun$blockingSendAndReceive$extension$1.apply(NetworkClientBlockingOps.scala:80)
    at kafka.utils.NetworkClientBlockingOps$.recursivePoll$2(NetworkClientBlockingOps.scala:137)
    at kafka.utils.NetworkClientBlockingOps$.kafka$utils$NetworkClientBlockingOps$$pollContinuously$extension(NetworkClientBlockingOps.scala:143)
    at kafka.utils.NetworkClientBlockingOps$.blockingSendAndReceive$extension(NetworkClientBlockingOps.scala:80)
    at kafka.server.ReplicaFetcherThread.sendRequest(ReplicaFetcherThread.scala:244)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:229)
    at kafka.server.ReplicaFetcherThread.fetch(ReplicaFetcherThread.scala:42)
    at kafka.server.AbstractFetcherThread.processFetchRequest(AbstractFetcherThread.scala:107)
    at kafka.server.AbstractFetcherThread.doWork(AbstractFetcherThread.scala:98)
    at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2022-06-23 00:27:44,004] INFO Creating /controller (is it secure? false) (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:27:44,006] INFO Result of znode creation is: OK (kafka.utils.ZKCheckedEphemeral)
[2022-06-23 00:27:44,006] INFO 0 successfully elected as leader (kafka.server.ZookeeperLeaderElector)
[2022-06-23 00:27:44,365] INFO [ReplicaFetcherManager on broker 0] Removed fetcher for partitions [Gxky_rec,0],[Gxky_data,1],[Gdsjb_rec,1],[Gqxrmt_rec,2],[Gqxrmt_data,0],[Gqxrmt_rec,0],[Gxky_rec,3],[Gxky_event,3],[Gxky_data,5],[Gdsjb_event,5],[Gxky_data,2],[Gqxrmt_rec,3],[Gdsjb_event,0],[Gqxrmt_event,5],[Gxky_event,0],[Gqxrmt_data,5],[Gxky_rec,2],[Gdsjb_rec,0],[Gdsjb_data,3],[Gdsjb_data,0],[Gdsjb_rec,4],[Gqxrmt_event,1],[Gdsjb_data,5],[Gqxrmt_data,3],[Gqxrmt_event,2],[Gxky_event,2],[Gdsjb_event,3] (kafka.server.ReplicaFetcherManager)
[2022-06-23 00:27:44,367] INFO [ReplicaFetcherThread-0-2], Shutting down (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:27:44,383] INFO New leader is 0 (kafka.server.ZookeeperLeaderElector$LeaderChangeListener)
[2022-06-23 00:27:58,733] INFO [ReplicaFetcherThread-0-2], Stopped  (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:27:58,733] INFO [ReplicaFetcherThread-0-2], Shutdown completed (kafka.server.ReplicaFetcherThread)
[2022-06-23 00:35:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:45:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 00:55:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 01:05:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 01:15:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)
[2022-06-23 01:25:33,855] INFO [Group Metadata Manager on Broker 0]: Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.GroupMetadataManager)


发表于 2022-06-22
添加评论

这种超时的错误,一般2种情况会引起

  1. 本身就无法到达(确保所有节点(hostname)均可互相识别和网络畅通)
  2. 某些资源占用率过高,导致一直在等待而导致的超时(例如迁移分区会产生大量的网络io需要进行限流)。

另外:brokers/ids 来是退出,还有种情况是broker.id误写重复了,导致数据混乱,kafka是基于broker.id来确认节点身份的,这种只能清理掉该节点kafka的物理数据,重新加入。

痕☆ -> 半兽人 1月前
  1. 节点之间的hostname均可识别互通,这个是没问题的;
  2. /borkers/ids 自动删除后,没有自动恢复;

【例如迁移分区会产生大量的网络io需要进行限流】这个限流指的是kafka消费和生产的数据流吗?
请问下,一般通过什么配置可以达到效果?

痕☆ -> 半兽人 1月前

您好,这边因为业务数据不太重要,昨天这边把kafka数据和zookeeper中的kafka相关配置,全部清理了,重启后,依然存在相关问题。
现在有个现象,这边kafka的下游是storm集群实时消费。
看了下kafka和storm的内网带宽监控趋势,
kafka平均出入内网带宽在20M/s, 峰值在70~80M/s,之后就趋于较低值100K/s,应该是自动停掉了。
同时storm也不再消费了。

不太清楚是storm消费流量太大导致kafka停掉,还是因为kafka自身流量问题被停掉。

半兽人 -> 痕☆ 1月前

看看系统日志,看看那个时间点kafka节点是不是被kill了

/var/log/message
痕☆ -> 半兽人 1月前

您好,查了一下3台机器的 /var/log/message 没有看见被 killed的记录

半兽人 -> 痕☆ 1月前

就奇怪了,进程还在,timeout的kafka节点是会自愈。

1、资源(带宽)一直被占用?
2、系统负载很高?

或者你是否有更多可疑的日志提供。

痕☆ -> 半兽人 1月前

这边kafka集群是和solr集群共用的服务器资源,相关监控指标都在范围内,不知是否有影响?
已将监控信息和日志贴到主贴后方,麻烦您看下。多谢

痕☆ -> 半兽人 1月前

大佬,您好。
刚看了下kafka,正在运行中,三台占用的cpu都在80%左右。
想考虑升级kafka和zookeeper版本,请问有推荐的稳定版本吗?

痕☆ -> 半兽人 1月前

您好,今天将kafka和zookeeper的版本进行了升级,分别升级到2.3.1和3.4.14,刚才偶发性的又掉了一个节点,现象和之前一样,进程在,brokers/ids 掉了,真是奇怪了。

以下是某个节点的配置,麻烦您帮忙看看配置是否有误。

broker.id=3
host.name=DC-kafka-3
port=9092
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/home/kafka/kafka-logs
num.partitions=3
num.recovery.threads.per.data.dir=1
#log.flush.interval.ms=1000
log.retention.hours=72
#log.retention.bytes=1073741824
log.segment.bytes=1073741824
log.retention.check.interval.ms=300000
log.cleaner.enable=false
offsets.topic.replication.factor=1
zookeeper.connect=DC-Zk-1:2181,DC-Zk-2:2181,DC-Zk-3:2181/kafka
zookeeper.connection.timeout.ms=60000
zookeeper.session.timeout.ms=90000

message.max.bytes=2048000
replica.fetch.max.bytes=2048576
default.replication.factor=3
半兽人 -> 痕☆ 1月前

你这个新环境也掉,网络是不是防火墙或者什么白名单。
或者你的数据没有清理干净,比如zk多节点没有全停了之后清理?

痕☆ -> 半兽人 1月前
  1. 防火墙都关闭了,不然启动就会报错,一般都是跑一会才会掉;
  2. 清理数据,是先停kafka,然后删掉zookeper中的kafka节点信息,然后清理kafka日志目录;

和客户端,比如生产者的客户端版本有关系吗?比如springboot中,这个版本没有升级,因为我看数据是正常写入到kafka中的。

半兽人 -> 痕☆ 1月前
防火墙

我是基于这个报错怀疑的:

Connection to 2 was disconnected before the response was read
在读取响应之前,断开了对2的连接

跑一会儿会掉,也有一种情况是防火墙杀长连接的策略,关闭了就不关注这个了。

生产者的客户端版本

这不会的,客户端发送的数据,不会影响到kafka集群的,只会影响消费者。

痕☆ -> 半兽人 1月前

防火墙都是关闭了的。
刚和同事沟通,说是再升级一下kafka版本。

之前版本和现在版本都出现问题,感觉和版本关系不大,挺奇怪的。

我的配置文件里配置的是host.name,我看很多都是配置listeners和advertised.listeners。不知是否有影响

半兽人 -> 痕☆ 1月前

你hostname映射没问题的情况下是ok,建议换掉。

host.name介绍:

已弃用:当advertised.listeners或listeners没设置时候才使用。请改用advertised.listeners。Hostname发布到Zookeeper供客户端使用。在IaaS环境中,Broker可能需要绑定不同的接口。如果没有设置,将会使用host.name(如果配置了)。否则将从java.net.InetAddress.getCanonicalHostName()获取。

参考来自:Kafka Broker配置

痕☆ -> 半兽人 1月前
  1. 我设置的机器别名格式为:dc-kafka-N ,配置的也是这个,我看启动kafka的时候,获取的host.name 如下:
    获取的是阿里云自定义的主机名,这个主机名只在本机hosts里配置了

    [2022-06-25 09:19:31,965] INFO Client environment:host.name=iZ23p29i***** (org.apache.zookeeper.ZooKeeper)
    [2022-06-25 09:19:31,965] INFO Client environment:java.version=1.8.0_321 (org.apache.zookeeper.ZooKeeper)
    
  2. 今天又发现个现象。zookeeper三个节点其中一个节点挂了,通过可视化工具客户端无法访问,但是在该节点服务器通过./zkCli.sh status 查看正常,并且可以进去命令行查看数据。

半兽人 -> 痕☆ 1月前

换成具体的ip吗?现在怎么样了。

痕☆ -> 半兽人 1月前

您好。做了如下一些调整:

  1. 升级了相关kafka的版本(目前是2.8.1)、zookeeper的版本(目前是3.5.9)、生产者springboot中的相关客户端版本;
  2. 原来kafka是和其他集群共用资源,目前是独享资源;

比较遗憾,上述操作还是没能解决kafka集群节点假死的现象,但是有所缓解,目前不影响生产者写入数据。

近两天发现个问题,由于使用zookeeper的服务比较多,包括kafka、solr、hadoop、spark、storm等。
运行一段时间后,zookeeper就会持续报超时等问题,重启后就恢复。

猜想,会不会是因为使用zookeeper的服务较多,某个服务GC时间较长,使zookeeper不稳定,从而导致kafka节点注册自行消失。
请问下,从哪些日志内容或异常能看出问题来?有没有什么排查手段

半兽人 -> 痕☆ 1月前

持续报超时,是别人连接zk报的,还是zk自己报的?

痕☆ -> 半兽人 1月前

是zookeeper自己报的。

这边kafka集群是和solr集群共用的服务器资源,相关监控指标都在范围内,不知是否有影响?
迁移阿里云之前挺稳定的,从经典网络变为专有网络后,问题不断。

你的答案

查看kafka相关的其他问题或提一个您自己的问题
提问