kafka 2.11-2.1.1 java.lang.OutOfMemoryError: Direct buffer memory 如何处理

giant 发表于: 2019-07-30   最后更新时间: 2019-07-30  

提问说明

 java.lang.OutOfMemoryError: Direct buffer memory
  1. 已改为:ExplicitGCInvokesConcurrent:
KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"

2:

KAFKA_HEAP_OPTS="-Xmx2048M -Xms2048M"

3: product.propertis

buffer.memory=1024M

Direct buffer memory



您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: Kafka多线程多分区消费顺序错乱问题
下一条: kafka消息消费慢的解决方案

  • 你的留言太混乱了,现在到哪一步了。

    • 没集群,就一台,我半路来的,不清楚安装后干了啥, 现在是启动可以, 但是后面会报如下错误。

      [2019-07-30 17:17:21,586] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
      [2019-07-30 17:17:21,587] INFO Shutting down. (kafka.log.LogManager)
      [2019-07-30 17:17:21,605] INFO [ProducerStateManager partition=__consumer_offsets-21] Writing producer snapshot at offset 425 (kafka.log.ProducerStateManager)
      [2019-07-30 17:17:21,607] INFO [ProducerStateManager partition=D_GENERATE_ADDRESS_TOPIC-0] Writing producer snapshot at offset 2 (kafka.log.ProducerStateManager)
      [2019-07-30 17:17:21,615] INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 171 (kafka.log.ProducerStateManager)
      [2019-07-30 17:17:21,630] INFO Shutdown complete. (kafka.log.LogManager)
      [2019-07-30 17:17:21,639] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
      [2019-07-30 17:17:21,642] INFO Session: 0x10157e437850000 closed (org.apache.zookeeper.ZooKeeper)
      [2019-07-30 17:17:21,643] INFO EventThread shut down for session: 0x10157e437850000 (org.apache.zookeeper.ClientCnxn)
      [2019-07-30 17:17:21,644] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
      [2019-07-30 17:17:21,644] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
      [2019-07-30 17:17:23,794] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
      [2019-07-30 17:17:23,813] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
      [2019-07-30 17:17:23,817] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)
      
        [2019-07-30 14:59:43,139] ERROR [GroupMetadataManager brokerId=0] Error loading offsets from __consumer_offsets-30 (kafka.coordinator.group.GroupMetadataManager)
        org.apache.kafka.common.KafkaException: Unknown group metadata version 3
            at kafka.coordinator.group.GroupMetadataManager$.schemaForGroupValue(GroupMetadataManager.scala:1077)
            at kafka.coordinator.group.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:1309)
            at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:602)
            at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:574)
            at scala.collection.Iterator$class.foreach(Iterator.scala:891)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
            at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
            at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
            at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:574)
            at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:555)
            at scala.collection.Iterator$class.foreach(Iterator.scala:891)
            at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
            at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
            at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
            at kafka.coordinator.group.GroupMetadataManager.doLoadGroupsAndOffsets(GroupMetadataManager.scala:555)
            at kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:500)
            at kafka.coordinator.group.GroupMetadataManager$$anonfun$scheduleLoadGroupAndOffsets$1.apply$mcV$sp(GroupMetadataManager.scala:491)
            at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:114)
            at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
            at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
            at java.util.concurrent.FutureTask.run(FutureTask.java:266)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
            at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
            at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
            at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
            at java.lang.Thread.run(Thread.java:748)
        [2019-07-30 14:59:43,140] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-36 in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 14:59:43,140] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-42 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 14:59:43,140] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-48 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 14:59:43,279] INFO [GroupCoordinator 0]: Preparing to rebalance group CALLBACK.PROCESSOR in state PreparingRebalance with old generation 0 (__consumer_offsets-21) (reason: Adding new member consumer-1-86dbbaa7-ed76-495b-af22-17186d8e284b) (kafka.coordinator.group.GroupCoordinator)
        [2019-07-30 14:59:43,287] INFO [GroupCoordinator 0]: Stabilized group CALLBACK.PROCESSOR generation 1 (__consumer_offsets-21) (kafka.coordinator.group.GroupCoordinator)
        [2019-07-30 14:59:43,297] INFO [GroupCoordinator 0]: Assignment received from leader for group CALLBACK.PROCESSOR for generation 1 (kafka.coordinator.group.GroupCoordinator)
        kill -9 29786[2019-07-30 15:09:42,389] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 2 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 15:19:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 15:29:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 15:39:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        [2019-07-30 15:48:25,840] ERROR Processor got uncaught exception. (kafka.network.Processor)
        java.lang.OutOfMemoryError: Direct buffer memory
            at java.nio.Bits.reserveMemory(Bits.java:695)
            at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
            at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
            at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
            at sun.nio.ch.IOUtil.read(IOUtil.java:195)
            at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
            at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:103)
            at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:117)
            at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
            at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
            at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
            at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
            at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
            at kafka.network.Processor.poll(SocketServer.scala:689)
            at kafka.network.Processor.run(SocketServer.scala:594)
            at java.lang.Thread.run(Thread.java:748)
        [2019-07-30 15:48:26,412] ERROR Processor got uncaught exception. (kafka.network.Processor)
        java.lang.OutOfMemoryError: Direct buffer memory
            at java.nio.Bits.reserveMemory(Bits.java:695)
            at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:123)
            at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:311)
            at sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:241)
            at sun.nio.ch.IOUtil.read(IOUtil.java:195)
            at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
            at org.apache.kafka.common.network.PlaintextTransportLayer.read(PlaintextTransportLayer.java:103)
            at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:117)
            at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:381)
            at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:342)
            at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:609)
            at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:541)
            at org.apache.kafka.common.network.Selector.poll(Selector.java:467)
            at kafka.network.Processor.poll(SocketServer.scala:689)
            at kafka.network.Processor.run(SocketServer.scala:594)
            at java.lang.Thread.run(Thread.java:748)
        [2019-07-30 15:49:42,387] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
        

        仍然报这个错, 哪位朋友帮看看

        • 找到/binkafka-run-class.sh,加大-XX:MaxDirectMemorySize,该参数默认是64M,可以根据需求调大
          并且检查JVM参数里面有无:-XX:+DisableExplicitGC,如果有就去掉.

            • -XX:MaxDirectMemorySize=512M ,-XX:+DisableExplicitGC之前就没有,
              KAFKA_JVM_PERFORMANCE_OPTS="-server -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:+ExplicitGCInvokesConcurrent -Djava.awt.headless=true"

                total               used           free           shared         buff/cache       available
                Mem:          16047        9413         534              3               6099               6264
                Swap:             0           0                 0
                
                • 是的 ,M。 我在前面参数的基础上, 在kafka-run-class.sh文件加了个-XX:MaxDirectMemorySize=512M, 启动 报错:

                  [2019-07-30 14:59:43,134] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-35 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                  [2019-07-30 14:59:43,134] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-0 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                  [2019-07-30 14:59:43,135] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-6 in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                  [2019-07-30 14:59:43,135] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-12 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                  [2019-07-30 14:59:43,135] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-18 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                  [2019-07-30 14:59:43,136] INFO [GroupMetadataManager brokerId=0] Finished loading offsets and group metadata from __consumer_offsets-24 in 1 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                  [2019-07-30 14:59:43,139] ERROR [GroupMetadataManager brokerId=0] Error loading offsets from __consumer_offsets-30 (kafka.coordinator.group.GroupMetadataManager)
                  org.apache.kafka.common.KafkaException: Unknown group metadata version 3
                      at kafka.coordinator.group.GroupMetadataManager$.schemaForGroupValue(GroupMetadataManager.scala:1077)
                      at kafka.coordinator.group.GroupMetadataManager$.readGroupMessageValue(GroupMetadataManager.scala:1309)
                      at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:602)
                      at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2$$anonfun$apply$13.apply(GroupMetadataManager.scala:574)
                      at scala.collection.Iterator$class.foreach(Iterator.scala:891)
                      at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
                      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
                      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
                      at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:574)
                      at kafka.coordinator.group.GroupMetadataManager$$anonfun$doLoadGroupsAndOffsets$2.apply(GroupMetadataManager.scala:555)
                      at scala.collection.Iterator$class.foreach(Iterator.scala:891)
                      at scala.collection.AbstractIterator.foreach(Iterator.scala:1334)
                      at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
                      at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
                      at kafka.coordinator.group.GroupMetadataManager.doLoadGroupsAndOffsets(GroupMetadataManager.scala:555)
                      at kafka.coordinator.group.GroupMetadataManager.loadGroupsAndOffsets(GroupMetadataManager.scala:500)
                      at kafka.coordinator.group.GroupMetadataManager$$anonfun$scheduleLoadGroupAndOffsets$1.apply$mcV$sp(GroupMetadataManager.scala:491)
                      at kafka.utils.KafkaScheduler$$anonfun$1.apply$mcV$sp(KafkaScheduler.scala:114)
                      at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:63)
                      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
                      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
                      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
                      at java.lang.Thread.run(Thread.java:748)
                  
                    • 删掉 log.dirs 下的 文件,重启zk和kafka 后, 第一次使用,可以, 几分钟后就不行了。

                      2019-07-30 17:53:58.539  WARN 19940 --- [ntainer#0-0-C-1] org.apache.kafka.clients.NetworkClient 
                        : [Consumer clientId=consumer-1, groupId=ADDRESS.PROCESSOR] 
                      Connection to node -1 could not be established. Broker may not be available.
                      
                      [2019-07-30 17:17:21,588] INFO Shutting down the log cleaner. (kafka.log.LogCleaner)
                      [2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Shutting down (kafka.log.LogCleaner)
                      [2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Stopped (kafka.log.LogCleaner)
                      [2019-07-30 17:17:21,589] INFO [kafka-log-cleaner-thread-0]: Shutdown completed (kafka.log.LogCleaner)
                      
                      
                      [2019-07-30 17:17:21,640] INFO Processed session termination for sessionid: 0x10157e437850000 (org.apache.zookeeper.server.PrepRequestProcessor)
                      [2019-07-30 17:17:21,643] INFO Closed socket connection for client / which had sessionid 0x10157e437850000 
                      (org.apache.zookeeper.server.NIOServerCnxn)
                      
                      [2019-07-30 17:07:00,588] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
                      [2019-07-30 17:12:00,588] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
                      [2019-07-30 17:12:00,590] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
                      [2019-07-30 17:17:00,590] TRACE [Controller id=0] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
                      [2019-07-30 17:17:00,591] TRACE [Controller id=0] Leader imbalance ratio for broker 0 is 0.0 (kafka.controller.KafkaController)
                      [2019-07-30 17:17:20,985] INFO [Controller id=0] Shutting down broker 0 (kafka.controller.KafkaController)
                      [2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Shutting down (kafka.controller.ControllerEventManager$ControllerEventThread)
                      [2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Stopped (kafka.controller.ControllerEventManager$ControllerEventThread)
                      [2019-07-30 17:17:21,630] INFO [ControllerEventThread controllerId=0] Shutdown completed (kafka.controller.ControllerEventManager$ControllerEventThread)
                      [2019-07-30 17:17:21,633] INFO [PartitionStateMachine controllerId=0] Stopped partition state machine (kafka.controller.PartitionStateMachine)
                      [2019-07-30 17:17:21,633] INFO [ReplicaStateMachine controllerId=0] Stopped replica state machine (kafka.controller.ReplicaStateMachine)
                      [2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Shutting down (kafka.controller.RequestSendThread)
                      [2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Stopped (kafka.controller.RequestSendThread)
                      [2019-07-30 17:17:21,636] INFO [RequestSendThread controllerId=0] Shutdown completed (kafka.controller.RequestSendThread)
                      [2019-07-30 17:17:21,638] INFO [Controller id=0] Resigned (kafka.controller.KafkaController)
                      

                      server.log

                       Terminating process due to signal SIGHUP (org.apache.kafka.common.utils.LoggingSignalHandler)
                      [2019-07-30 17:17:20,969] INFO [KafkaServer id=0] shutting down (kafka.server.KafkaServer)
                      [2019-07-30 17:17:20,969] INFO [KafkaServer id=0] Starting controlled shutdown (kafka.server.KafkaServer)
                      [2019-07-30 17:17:20,996] INFO [KafkaServer id=0] Controlled shutdown succeeded (kafka.server.KafkaServer)
                      [2019-07-30 17:17:20,999] INFO [/config/changes-event-process-thread]: Shutting down (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
                      [2019-07-30 17:17:21,000] INFO [/config/changes-event-process-thread]: Shutdown completed (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
                      [2019-07-30 17:17:21,000] INFO [/config/changes-event-process-thread]: Stopped (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
                      [2019-07-30 17:17:21,000] INFO [SocketServer brokerId=0] Stopping socket server request processors (kafka.network.SocketServer)
                      [2019-07-30 17:17:21,011] INFO [SocketServer brokerId=0] Stopped socket server request processors (kafka.network.SocketServer)
                      [2019-07-30 17:17:21,012] INFO [Kafka Request Handler on Broker 0], shutting down (kafka.server.KafkaRequestHandlerPool)
                      [2019-07-30 17:17:21,014] INFO [Kafka Request Handler on Broker 0], shut down completely (kafka.server.KafkaRequestHandlerPool)
                      [2019-07-30 17:17:21,017] INFO [KafkaApi-0] Shutdown complete. (kafka.server.KafkaApis)
                      [2019-07-30 17:17:21,017] INFO [ExpirationReaper-0-topic]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,095] INFO [ExpirationReaper-0-topic]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,096] INFO [ExpirationReaper-0-topic]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,098] INFO [TransactionCoordinator id=0] Shutting down. (kafka.coordinator.transaction.TransactionCoordinator)
                      [2019-07-30 17:17:21,098] INFO [ProducerId Manager 0]: Shutdown complete: last producerId assigned 159000 (kafka.coordinator.transaction.ProducerIdManager)
                      [2019-07-30 17:17:21,099] INFO [Transaction State Manager 0]: Shutdown complete (kafka.coordinator.transaction.TransactionStateManager)
                      [2019-07-30 17:17:21,099] INFO [Transaction Marker Channel Manager 0]: Shutting down (kafka.coordinator.transaction.TransactionMarkerChannelManager)
                      [2019-07-30 17:17:21,099] INFO [Transaction Marker Channel Manager 0]: Stopped (kafka.coordinator.transaction.TransactionMarkerChannelManager)
                      [2019-07-30 17:17:21,100] INFO [Transaction Marker Channel Manager 0]: Shutdown completed (kafka.coordinator.transaction.TransactionMarkerChannelManager)
                      [2019-07-30 17:17:21,100] INFO [TransactionCoordinator id=0] Shutdown complete. (kafka.coordinator.transaction.TransactionCoordinator)
                      [2019-07-30 17:17:21,100] INFO [GroupCoordinator 0]: Shutting down. (kafka.coordinator.group.GroupCoordinator)
                      [2019-07-30 17:17:21,101] INFO [ExpirationReaper-0-Heartbeat]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,298] INFO [ExpirationReaper-0-Heartbeat]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,298] INFO [ExpirationReaper-0-Heartbeat]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,299] INFO [ExpirationReaper-0-Rebalance]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,301] INFO [ExpirationReaper-0-Rebalance]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,301] INFO [ExpirationReaper-0-Rebalance]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,302] INFO [GroupCoordinator 0]: Shutdown complete. (kafka.coordinator.group.GroupCoordinator)
                      [2019-07-30 17:17:21,303] INFO [ReplicaManager broker=0] Shutting down (kafka.server.ReplicaManager)
                      [2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Shutting down (kafka.server.ReplicaManager$LogDirFailureHandler)
                      [2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Stopped (kafka.server.ReplicaManager$LogDirFailureHandler)
                      [2019-07-30 17:17:21,303] INFO [LogDirFailureHandler]: Shutdown completed (kafka.server.ReplicaManager$LogDirFailureHandler)
                      [2019-07-30 17:17:21,304] INFO [ReplicaFetcherManager on broker 0] shutting down (kafka.server.ReplicaFetcherManager)
                      [2019-07-30 17:17:21,305] INFO [ReplicaFetcherManager on broker 0] shutdown completed (kafka.server.ReplicaFetcherManager)
                      [2019-07-30 17:17:21,305] INFO [ReplicaAlterLogDirsManager on broker 0] shutting down (kafka.server.ReplicaAlterLogDirsManager)
                      [2019-07-30 17:17:21,305] INFO [ReplicaAlterLogDirsManager on broker 0] shutdown completed (kafka.server.ReplicaAlterLogDirsManager)
                      [2019-07-30 17:17:21,305] INFO [ExpirationReaper-0-Fetch]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,344] INFO [ExpirationReaper-0-Fetch]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,344] INFO [ExpirationReaper-0-Fetch]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,345] INFO [ExpirationReaper-0-Produce]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-Produce]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-Produce]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,381] INFO [ExpirationReaper-0-DeleteRecords]: Shutting down (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,582] INFO [ExpirationReaper-0-DeleteRecords]: Stopped (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,583] INFO [ExpirationReaper-0-DeleteRecords]: Shutdown completed (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
                      [2019-07-30 17:17:21,586] INFO [ReplicaManager broker=0] Shut down completely (kafka.server.ReplicaManager)
                      [2019-07-30 17:17:21,587] INFO Shutting down. (kafka.log.LogManager)
                      [2019-07-30 17:17:21,605] INFO [ProducerStateManager partition=__consumer_offsets-21] Writing producer snapshot at offset 425 (kafka.log.ProducerStateManager)
                      [2019-07-30 17:17:21,607] INFO [ProducerStateManager partition=D_GENERATE_ADDRESS_TOPIC-0] Writing producer snapshot at offset 2 (kafka.log.ProducerStateManager)
                      [2019-07-30 17:17:21,615] INFO [ProducerStateManager partition=__consumer_offsets-44] Writing producer snapshot at offset 171 (kafka.log.ProducerStateManager)
                      [2019-07-30 17:17:21,630] INFO Shutdown complete. (kafka.log.LogManager)
                      [2019-07-30 17:17:21,639] INFO [ZooKeeperClient] Closing. (kafka.zookeeper.ZooKeeperClient)
                      [2019-07-30 17:17:21,642] INFO Session: 0x10157e437850000 closed (org.apache.zookeeper.ZooKeeper)
                      [2019-07-30 17:17:21,643] INFO EventThread shut down for session: 0x10157e437850000 (org.apache.zookeeper.ClientCnxn)
                      [2019-07-30 17:17:21,644] INFO [ZooKeeperClient] Closed. (kafka.zookeeper.ZooKeeperClient)
                      [2019-07-30 17:17:21,644] INFO [ThrottledChannelReaper-Fetch]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Fetch]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:21,793] INFO [ThrottledChannelReaper-Produce]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Produce]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:22,793] INFO [ThrottledChannelReaper-Request]: Shutting down (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Stopped (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:23,793] INFO [ThrottledChannelReaper-Request]: Shutdown completed (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
                      [2019-07-30 17:17:23,794] INFO [SocketServer brokerId=0] Shutting down socket server (kafka.network.SocketServer)
                      [2019-07-30 17:17:23,813] INFO [SocketServer brokerId=0] Shutdown completed (kafka.network.SocketServer)
                      [2019-07-30 17:17:23,817] INFO [KafkaServer id=0] shut down completed (kafka.server.KafkaServer)