kafka报错 :org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 704708608 larger than 524288)

伴我同行 发表于: 2019-09-25   最后更新时间: 2019-09-26  

kafka报错 :org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 704708608 larger than 524288)

WARN Unexpected error from /172.17.0.136; closing connection (org.apache.kafka.common.network.Selector)
org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 704708608 larger than 524288)
        at org.apache.kafka.common.network.NetworkReceive.readFromReadableChannel(NetworkReceive.java:95)
        at org.apache.kafka.common.network.NetworkReceive.readFrom(NetworkReceive.java:75)
        at org.apache.kafka.common.security.authenticator.SaslServerAuthenticator.authenticate(SaslServerAuthenticator.java:216)
        at org.apache.kafka.common.network.KafkaChannel.prepare(KafkaChannel.java:76)
        at org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:376)
        at org.apache.kafka.common.network.Selector.poll(Selector.java:326)
        at kafka.network.Processor.poll(SocketServer.scala:500)
        at kafka.network.Processor.run(SocketServer.scala:435)
        at java.lang.Thread.run(Thread.java:745)

我的kafka server.properties配置文件里面是有配置

socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600

但是好像配置并没有生效

而且我查看kafka server.log 发现内部有错误

[2019-09-25 19:13:56,879] FATAL  (kafka.Kafka$)
java.io.FileNotFoundException: ./config/server.properties (No such file or directory)
        at java.io.FileInputStream.open(Native Method)
        at java.io.FileInputStream.<init>(FileInputStream.java:146)
        at java.io.FileInputStream.<init>(FileInputStream.java:101)
        at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:428)
        at kafka.Kafka$.getPropsFromArgs(Kafka.scala:41)
        at kafka.Kafka$.main(Kafka.scala:57)
        at kafka.Kafka.main(Kafka.scala)

但是我不知道我哪里有获取这个文件,而且我的kafka正常启动了,所以想咨询一下大佬,有没有碰到我这个问题,我弄了一周了还没解决



您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: kafka扩容副本后,消费者无法消费
下一条: zookeeper启动时报错空指针

  • 解决了吗?我也一样的问题

    • [2019-12-23 05:44:31,268] INFO [SocketServer brokerId=0] Failed authentication with /172.17.0.1 (Authentication failed during authentication due to invalid credentials with SASL mechanism SCRAM-SHA-256) (org.apache.kafka.common.network.Selector),
      

      我是先启动zk和kafka再创建admin用户的,因为是现成的docker镜像,只能先启动,再创建admin,再重启的,不知道这样是不是有问题

        • 1、可能试图接收到太大的请求。最大大小是socket.request.max.bytes的默认大小,为100MB。 因此,如果收到的消息大于100MB,尝试增加server.properties下此变量的值。
          2、我看你报的是认证的错误,我怀疑你漏配了server内部协议

          listeners=SASL_SSL://localhost:9093
          security.inter.broker.protocol=SASL_SSL
          sasl.mechanism.inter.broker.protocol=PLAIN
          sasl.enabled.mechanisms=PLAIN
          
            • 我只是执行了kafka-console-producer.sh 命令,还没有发送消息到kafka,kafka那边就已经这个错了,我配置的是SASL_PLAINTEXT

                这是我启动的命令:

                cd /usr/kafka/kafka_2.11-0.11.0.1 ./bin/kafka-server-start.sh -daemon ./config/server.properties 1>/dev/null 2>&1 &
                
                • 我最困惑的还是这个错误Invalid receive (size = 704708608 larger than 524288) 明明我的server.properties里面有设置但是配置没生效感觉跟这个java.io.FileNotFoundException: ./config/server.properties (No such file or directory)错误有关系

                    • 找不到目录或文件这种错,在启动时会起不来,而不是在运行中报的,第一次遇到,你只运行

                      bin/kafka-server-start.sh config/server.properties
                      

                      看看是否可以启动成功。

                        • 直接按照你的命令启动过不了多久又会出来

                          [2019-09-26 14:08:51,191] INFO [Group Metadata Manager on Broker 191]: Finished loading offsets and group metadata from __consumer_offsets-21 in 0 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
                          [2019-09-26 14:09:46,469] FATAL  (kafka.Kafka$)
                          java.io.FileNotFoundException: ./config/server.properties (No such file or directory)
                                  at java.io.FileInputStream.open(Native Method)
                                  at java.io.FileInputStream.(FileInputStream.java:146)
                                  at java.io.FileInputStream.(FileInputStream.java:101)
                                  at org.apache.kafka.common.utils.Utils.loadProps(Utils.java:428)
                                  at kafka.Kafka$.getPropsFromArgs(Kafka.scala:41)
                                  at kafka.Kafka$.main(Kafka.scala:57)
                                  at kafka.Kafka.main(Kafka.scala)
                          

                          但是进程是在的

                          [root@servername191 kafka_2.11-0.11.0.1]# jps
                          19047 -- process information unavailable
                          5039 Jps
                          13565 core
                          3430 Kafka
                          13454 QuorumPeerMain
                          14697 nimbus
                          
                            • 问题是我消费和生成的程序都可以正常运行,但是外网的就不行就会报:org.apache.kafka.common.network.InvalidReceiveException: Invalid receive (size = 704708608 larger than 524288) ,真的是没办法

                                你这么启动的kafka?命令贴一下。 是否是这样

                                bin/kafka-server-start.sh config/server.properties