kafka-consumer-perf-test.sh脚本遇到OutOfMemoryError

一如乞人不需要形象 发表于: 2021-12-29   最后更新时间: 2021-12-29  

大佬,我在使用kafka-consumer-perf-test.sh做消费者performance 测试时报java.lang.OutOfMemoryError,想问是什么原因,怎么解决?

 kafka-consumer-perf-test.sh --broker-list localhost:8083 --messages 100000 --topic test
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.nio.HeapByteBuffer.<init>(HeapByteBuffer.java:57)
    at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
    at org.apache.kafka.common.memory.MemoryPool$1.tryAllocate(MemoryPool.java:30)
    at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:113)
    at org.apache.kafka.common.network.KafkaChannel.receive(KafkaChannel.java:448)
    at org.apache.kafka.common.network.KafkaChannel.read(KafkaChannel.java:398)
    at org.apache.kafka.common.network.Selector.attemptRead(Selector.java:678)
    at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:580)
    at org.apache.kafka.common.network.Selector.poll(Selector.java:485)
    at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:549)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:262)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:233)
    at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:212)
    at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:236)
    at org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.poll(ConsumerCoordinator.java:463)
    at org.apache.kafka.clients.consumer.KafkaConsumer.updateAssignmentMetadataIfNeeded(KafkaConsumer.java:1275)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1241)
    at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1216)
    at kafka.tools.ConsumerPerformance$.consume(ConsumerPerformance.scala:126)
    at kafka.tools.ConsumerPerformance$.main(ConsumerPerformance.scala:56)
    at kafka.tools.ConsumerPerformance.main(ConsumerPerformance.scala)


您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: 请教下kafka消费速度问题
下一条: kafka mirrorMaker可以做跨版本的kafka集群数据同步吗

  • 内存不足了,到脚本里面调整一下jvm

    • 额,不是很懂脚本,改这个吗 KAFKA_HEAP_OPTS="-Xmx512M"

      #!/bin/bash
      # Licensed to the Apache Software Foundation (ASF) under one or more
      # contributor license agreements.  See the NOTICE file distributed with
      # this work for additional information regarding copyright ownership.
      # The ASF licenses this file to You under the Apache License, Version 2.0
      # (the "License"); you may not use this file except in compliance with
      # the License.  You may obtain a copy of the License at
      #
      #    http://www.apache.org/licenses/LICENSE-2.0
      #
      # Unless required by applicable law or agreed to in writing, software
      # distributed under the License is distributed on an "AS IS" BASIS,
      # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
      # See the License for the specific language governing permissions and
      # limitations under the License.
      if [ "x$KAFKA_HEAP_OPTS" = "x" ]; then
          export KAFKA_HEAP_OPTS="-Xmx512M"
      fi
      exec $(dirname $0)/kafka-run-class.sh kafka.tools.ConsumerPerformance "$@"