虚拟机CentOS 7下的docker(CentOS 7)内的kafka消费者无法消费到来自windows下Java客户端(IDE)生产者生产的消息

ぶぽめつせ 发表于: 2020-11-17   最后更新时间: 2020-11-17  

环境

位置关系如下:

                        |----------------------------------------|
                        |                                        |
                        |   |--------------------------------|   |
                        |   |                                |   |
                        |   |    docker(CentOS) 172.18.0.2   |   |
                        |   |--------------------------------|   |
                        |                                        |
                        | Virtual Machine(CentOS) 192.168.27.143 |
                        |----------------------------------------|

                                 Windows 192.168.137.123
  1. 我在虚拟机里创建了3个docker容器并分别部署了kafka和ZooKeeper(172.18.0.2,172.18.0.3,172.18.0.4 在同一个network)

  2. windows下hosts文件已做IP映射(172.18.0.2 master,172.18.0.3 slave1,172.18.0.4 slave2)

  3. 两两可以互相ping通

  4. 虚拟机2181,9092端口已开启

  5. docker 172.18.0.2 的2181,9092端口已映射

  6. 虚拟机防火墙已关,docker没有安装防火墙

出现问题

我在windows下Java客户端用代码模拟生产者向kafka发消息,kafka里面的消费者没有收到消息。

尝试一

我按照网上说的将advertised.listeners设为PLAINTEXT://<容器的hostname或IP>:9092,依旧不行

尝试二

listeners=INTERNAL://0.0.0.0:29094,EXTERNAL://0.0.0.0:9092
advertised.listeners=INTERNAL://master:29094,EXTERNAL://192.168.27.143:9092
listener.security.protocol.map=INTERNAL:PLAINTEXT,EXTERNAL:PLAINTEXT
inter.broker.listener.name=INTERNAL

命令行

bin/kafka-server-start.sh -daemon /opt/kafka-2.6.0/config/server.properties

bin/kafka-topics.sh --create --zookeeper master:2181 --replication-factor 1 --partitions 1 --topic one-kafka

bin/kafka-console-producer.sh --broker-list master:9092 --topic one-kafka

bin/kafka-console-consumer.sh --bootstrap-server master:9092 --topic one-kafka

命令行生产者消费者没问题,Java客户端生产的消息依然不能被kafka消费

生产者代码

package com.kafka;

import java.util.Properties;
import java.util.concurrent.ExecutionException;
import java.util.concurrent.Future;

import org.apache.kafka.clients.producer.KafkaProducer;
import org.apache.kafka.clients.producer.ProducerConfig;
import org.apache.kafka.clients.producer.ProducerRecord;
import org.apache.kafka.clients.producer.RecordMetadata;
public class MyProducer {

    public static void main(String[] args) {
        Properties properties = new Properties();
        properties.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, "192.168.27.143:9092");// 这里写master也不行
        properties.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG,
                "org.apache.kafka.common.serialization.StringSerializer");
        properties.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG,
                "org.apache.kafka.common.serialization.StringSerializer");
        KafkaProducer<String, String> producer = new KafkaProducer<String, String>(properties);
        for (int i = 1; i <= 10; i++) {
            Future<RecordMetadata> future = producer.send(new ProducerRecord<>("one-kafka", "666"));
            try {
                RecordMetadata metadata = future.get();
                System.out.println(metadata.topic());
            } catch (InterruptedException e) {
                e.printStackTrace();
            } catch (ExecutionException e) {
                e.printStackTrace();
            }
        }
        producer.close();
    }

}

客户端报错

SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
java.util.concurrent.ExecutionException: org.apache.kafka.common.errors.TimeoutException: Topic one-kafka not present in metadata after 60000 ms.
    at org.apache.kafka.clients.producer.KafkaProducer$FutureFailure.<init>(KafkaProducer.java:1314)
    at org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:970)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:870)
    at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:758)
    at com.kafka.MyProducer.main(MyProducer.java:22)
Caused by: org.apache.kafka.common.errors.TimeoutException: Topic one-kafka not present in metadata after 60000 ms.


您需要解锁本帖隐藏内容请: 点击这里
本帖隐藏的内容




上一条: 今天kafka生产环境突然报错TimeoutException: Expiring 1 record(s) for TOPICNAME-1:963170 ms has passed since batch creation
下一条: kafka分配leader的时候,是依据什么条件?

  • 把listeners的0.0.0.0改成${HOSTNAME##*-}试试。我记得kafka 0.0.0.0 这种一直不可用。
    参考来自:https://www.kubebiz.com/KubeBiz/kafka

    • 错了错了,我复制错了,这个,获取当前容器的ip。

      listeners=PLAINTEXT://$(hostname -i):9093
      

      advertised.listeners已经弃用了,直接配置listeners就可以了。

        • 配置文件我写成了listeners=PLAINTEXT://master:9092
          listeners=PLAINTEXT://$(hostname -i):9092这个不行,解析错误,Error creating broker listeners from 'PLAINTEXT://$(hostname -i):909
          2': Unable to parse PLAINTEXT://$(hostname -i):9092 to a broker endpoint
          我代码也写成了"172.18.0.2:9092"
          客户端还是连不上。

            • 我telnet不了,一进去就卡住了,不过我换了个工具,测得端口是开的

              C:\Users\HP&gt;tcping 192.168.27.143 2181
              Probing 192.168.27.143:2181/tcp - Port is open - time=14.415ms
              Probing 192.168.27.143:2181/tcp - Port is open - time=77.710ms
              Probing 192.168.27.143:2181/tcp - Port is open - time=0.690ms
              Probing 192.168.27.143:2181/tcp - Port is open - time=0.541ms
              Ping statistics for 192.168.27.143:2181
                   4 probes sent.
                   4 successful, 0 failed.  (0.00% fail)
              Approximate trip times in milli-seconds:
                   Minimum = 0.541ms, Maximum = 77.710ms, Average = 23.339ms
              C:\Users\HP&gt;tcping 192.168.27.143 9092
              Probing 192.168.27.143:9092/tcp - Port is open - time=2.055ms
              Probing 192.168.27.143:9092/tcp - Port is open - time=0.686ms
              Probing 192.168.27.143:9092/tcp - Port is open - time=3.479ms
              Probing 192.168.27.143:9092/tcp - Port is open - time=0.502ms
              Ping statistics for 192.168.27.143:9092
                   4 probes sent.
                   4 successful, 0 failed.  (0.00% fail)
              Approximate trip times in milli-seconds:
                   Minimum = 0.502ms, Maximum = 3.479ms, Average = 1.681ms
              C:\Users\HP&gt;tcping 192.168.27.143 9093
              Probing 192.168.27.143:9093/tcp - No response - time=2001.959ms
              Probing 192.168.27.143:9093/tcp - No response - time=2000.190ms
              Probing 192.168.27.143:9093/tcp - No response - time=2000.843ms
              Probing 192.168.27.143:9093/tcp - No response - time=2000.144ms
              Ping statistics for 192.168.27.143:9093
                   4 probes sent.
                   0 successful, 4 failed.  (100.00% fail)
              Was unable to connect, cannot provide trip statistics.
              C:\Users\HP&gt;tcping 192.168.27.143 22
              Probing 192.168.27.143:22/tcp - Port is open - time=2.345ms
              Probing 192.168.27.143:22/tcp - Port is open - time=1.384ms
              Probing 192.168.27.143:22/tcp - Port is open - time=0.525ms
              Probing 192.168.27.143:22/tcp - Port is open - time=1.177ms
              Ping statistics for 192.168.27.143:22
                   4 probes sent.
                   4 successful, 0 failed.  (0.00% fail)
              Approximate trip times in milli-seconds:
                   Minimum = 0.525ms, Maximum = 2.345ms, Average = 1.358ms
              
                • 这是不是有了呀?br-1d17b5a61c8d是我自定义的docker网络

                  [root@docker ~]# route -n
                  Kernel IP routing table
                  Destination     Gateway         Genmask         Flags Metric Ref    Use Iface
                  0.0.0.0         192.168.27.2    0.0.0.0         UG    100    0        0 ens33
                  172.17.0.0      0.0.0.0         255.255.0.0     U     0      0        0 docker0
                  172.18.0.0      0.0.0.0         255.255.0.0     U     0      0        0 br-1d17b5a61c8d
                  192.168.27.0    0.0.0.0         255.255.255.0   U     100    0        0 ens33
                  
                    • root@docker ~]# route -n
                      Kernel IP routing table
                      Destination   |  Gateway    |     Genmask     |    Flags | Metric | Ref  |  Use | Iface
                      0.0.0.0      |   192.168.27.2 |   0.0.0.0    |     UG    |100  |  0    |    0 |  ens33
                      172.17.0.0   |   0.0.0.0      |   255.255.0.0  |   U  |   0   |   0    |    0 | docker0
                      172.18.0.0   |   0.0.0.0     |    255.255.0.0  |   U  |   0  |    0   |     0  | br-1d17b5a61c8d
                      192.168.27.0 |   0.0.0.0    |     255.255.255.0 |  U |    100  |  0    |    0 | ens33
                      
                        • 我分别在虚拟机和master容器用nc -lk 9092接收数据,用java发消息,都可以收到,路由应该没问题?

                              try {
                                  socket = new Socket("master", 9092);
                                  String message = "\n你叫搜集的备份节点开发板的百分比时刻的部分看都不看分不开的";
                                  OutputStream os = socket.getOutputStream();
                                  os.write(message.getBytes("GBK"));
                              } catch (IOException e) {
                                  e.printStackTrace();
                              } finally {
                                  try {
                                      socket.close();
                                  } catch (IOException e) {
                                      e.printStackTrace();
                                  }
                              }