kafka manager Consumers选项为空,看不到消费者

. 发表于: 2019-09-24   最后更新时间: 2019-09-24 19:45:11   8,353 游览

github下载了最新版,编译后kafka-manager-2.0.0.2.zip

启动后正常访问,topic都可以看到

Consumers选项为空,看不到消费者

日志一直输出

2019-09-24 18:05:41,759 - [INFO] k.m.a.KafkaManagerActor - Updating internal state...
2019-09-24 18:05:42,298 - [INFO] p.c.s.AkkaHttpServer - Listening for HTTP on /0.0.0.0:9095
2019-09-24 18:05:42,539 - [INFO] k.m.a.KafkaManagerActor - Adding new cluster manager for cluster : kafka-beta
2019-09-24 18:05:42,545 - [WARN] o.a.c.r.ExponentialBackoffRetry - maxRetries too large (100). Pinning to 29
2019-09-24 18:05:42,547 - [INFO] k.m.a.c.ClusterManagerActor - Starting curator...
2019-09-24 18:05:42,547 - [INFO] o.a.z.ZooKeeper - Initiating client connection, connectString=localhost:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@7e9e31e1
2019-09-24 18:05:42,551 - [WARN] o.a.c.r.ExponentialBackoffRetry - maxRetries too large (100). Pinning to 29
2019-09-24 18:05:42,552 - [INFO] k.m.a.c.ClusterManagerActor - Starting shared curator...
2019-09-24 18:05:42,554 - [INFO] o.a.z.ZooKeeper - Initiating client connection, connectString=127.0.0.1:2181 sessionTimeout=60000 watcher=org.apache.curator.ConnectionState@7695d471
2019-09-24 18:05:42,584 - [WARN] o.a.z.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/local/install/kafka23/config/kafka_manager_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-09-24 18:05:42,584 - [INFO] o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181
2019-09-24 18:05:42,584 - [INFO] o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
2019-09-24 18:05:42,585 - [WARN] o.a.z.ClientCnxn - SASL configuration failed: javax.security.auth.login.LoginException: No JAAS configuration section named 'Client' was found in specified JAAS configuration file: '/usr/local/install/kafka23/config/kafka_manager_jaas.conf'. Will continue connection to Zookeeper server without SASL authentication, if Zookeeper server allows it.
2019-09-24 18:05:42,585 - [INFO] o.a.z.ClientCnxn - Opening socket connection to server localhost/127.0.0.1:2181
2019-09-24 18:05:42,585 - [INFO] o.a.z.ClientCnxn - Socket connection established to localhost/127.0.0.1:2181, initiating session
2019-09-24 18:05:42,586 - [ERROR] o.a.c.ConnectionState - Authentication failed
2019-09-24 18:05:42,588 - [ERROR] o.a.c.ConnectionState - Authentication failed
2019-09-24 18:05:42,599 - [INFO] o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1001f2887100079, negotiated timeout = 60000
2019-09-24 18:05:42,600 - [INFO] o.a.z.ClientCnxn - Session establishment complete on server localhost/127.0.0.1:2181, sessionid = 0x1001f288710007a, negotiated timeout = 60000
2019-09-24 18:05:42,635 - [INFO] k.m.a.c.KafkaCommandActor - Started actor akka://kafka-manager-system/user/kafka-manager/kafka-beta/kafka-command
2019-09-24 18:05:42,637 - [INFO] k.m.a.c.KafkaAdminClientActor - KafkaAdminClientActorConfig(ClusterContext(ClusterFeatures(Set(KMDeleteTopicFeature)),ClusterConfig(kafka-beta,CuratorConfig(127.0.0.1:2181,100,100,1000),true,2.2.0,false,Some(admin),Some(admin-secret),false,false,false,false,false,false,Some(ClusterTuning(Some(30),Some(2),Some(100),Some(2),Some(100),Some(2),Some(100),Some(30),Some(5),Some(2),Some(1000),Some(2),Some(1000),Some(2),Some(1000),Some(30000),Some(1000000),Some(7))),SASL_PLAINTEXT,None,Some(/usr/local/install/kafka23/config/kafka_manager_jaas.conf))),LongRunningPoolConfig(2,1000),akka://kafka-manager-system/user/kafka-manager/kafka-beta/kafka-state,None)
2019-09-24 18:05:42,642 - [INFO] k.m.a.c.BrokerViewCacheActor - Started actor akka://kafka-manager-system/user/kafka-manager/kafka-beta/broker-view
2019-09-24 18:05:42,642 - [INFO] k.m.a.c.BrokerViewCacheActor - Scheduling updater for 30 seconds
2019-09-24 18:05:42,644 - [INFO] k.m.a.c.BrokerViewCacheActor - Updating broker view...
2019-09-24 18:05:42,644 - [INFO] k.m.a.c.ClusterManagerActor - Started actor akka://kafka-manager-system/user/kafka-manager/kafka-beta
2019-09-24 18:05:42,644 - [INFO] k.m.a.c.ClusterManagerActor - Starting cluster manager topics path cache...
2019-09-24 18:05:42,663 - [INFO] k.m.a.c.KafkaStateActor - KafkaStateActorConfig(org.apache.curator.framework.imps.CuratorFrameworkImpl@d8dc36c,pinned-dispatcher,ClusterContext(ClusterFeatures(Set(KMDeleteTopicFeature)),ClusterConfig(kafka-beta,CuratorConfig(127.0.0.1:2181,100,100,1000),true,2.2.0,false,Some(admin),Some(admin-secret),false,false,false,false,false,false,Some(ClusterTuning(Some(30),Some(2),Some(100),Some(2),Some(100),Some(2),Some(100),Some(30),Some(5),Some(2),Some(1000),Some(2),Some(1000),Some(2),Some(1000),Some(30000),Some(1000000),Some(7))),SASL_PLAINTEXT,None,Some(/usr/local/install/kafka23/config/kafka_manager_jaas.conf))),LongRunningPoolConfig(2,1000),LongRunningPoolConfig(2,1000),5,10000,None,KafkaManagedOffsetCacheConfig(30000,1000000,7))
2019-09-24 18:05:42,664 - [INFO] k.m.a.c.KafkaStateActor - Started actor akka://kafka-manager-system/user/kafka-manager/kafka-beta/kafka-state
2019-09-24 18:05:42,664 - [INFO] k.m.a.c.KafkaStateActor - Starting topics tree cache...
2019-09-24 18:05:42,672 - [INFO] k.m.a.c.KafkaStateActor - Starting topics config path cache...
2019-09-24 18:05:42,790 - [INFO] k.m.a.c.KafkaStateActor - Starting brokers path cache...
2019-09-24 18:05:42,798 - [INFO] k.m.a.c.KafkaStateActor - Starting admin path cache...
2019-09-24 18:05:42,801 - [INFO] k.m.a.c.KafkaStateActor - Starting delete topics path cache...
2019-09-24 18:05:42,811 - [INFO] k.m.a.c.KafkaStateActor - Adding topics tree cache listener...
2019-09-24 18:05:42,812 - [INFO] k.m.a.c.KafkaStateActor - Adding admin path cache listener...
2019-09-24 18:05:42,812 - [INFO] k.m.a.c.KafkaStateActor - Starting offset cache...
2019-09-24 18:05:42,817 - [INFO] k.m.a.c.OffsetCachePassive - Starting kafka managed offset cache ...
2019-09-24 18:05:43,090 - [INFO] k.m.a.c.OffsetCachePassive - Starting consumers path children cache...
2019-09-24 18:05:43,094 - [INFO] k.m.a.c.KafkaManagedOffsetCache - Constructing new kafka consumer client using these properties: 
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - auto.offset.reset=latest
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - bootstrap.servers=IP(手动替换了):9093,IP(手动替换了):9094
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - enable.auto.commit=false
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - group.id=KMOffsetCache-iZbp11w2fps1wt2ckilxfhZ
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - exclude.internal.topics=false
2019-09-24 18:05:43,097 - [INFO] k.m.a.c.KafkaManagedOffsetCache - security.protocol=SASL_PLAINTEXT
2019-09-24 18:05:43,170 - [INFO] k.m.a.c.OffsetCachePassive - Adding consumers path children cache listener...
2019-09-24 18:05:43,179 - [INFO] k.m.a.c.KafkaStateActor - Starting kafka managed Topic Offset Getter ...
2019-09-24 18:05:43,380 - [INFO] k.m.a.c.KafkaStateActor - SASL JAAS config=/usr/local/install/kafka23/config/kafka_manager_jaas.conf
2019-09-24 18:05:43,412 - [ERROR] k.m.a.c.KafkaStateActor - consumerProperties:{key.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, bootstrap.servers=IP(手动替换了):9093, group.id=kafka.manager.actor.cluster.KafkaStateActor.KafkaTopicOffsetGetter, sasl.jaas.config=/usr/local/install/kafka23/config/kafka_manager_jaas.conf, value.deserializer=org.apache.kafka.common.serialization.ByteArrayDeserializer, security.protocol=SASL_PLAINTEXT}
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$21(KafkaStateActor.scala:1489)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$21$adapted(KafkaStateActor.scala:1467)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:181)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$20(KafkaStateActor.scala:1467)
Caused by: java.lang.IllegalArgumentException: Login module not specified in JAAS config
        at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:66)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
        at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:124)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:112)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:726)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$21(KafkaStateActor.scala:1489)
2019-09-24 18:05:43,413 - [ERROR] k.m.a.c.KafkaStateActor - KafkaTopicOffsetGetter exception 
org.apache.kafka.common.KafkaException: Failed to construct kafka consumer
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:811)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$21(KafkaStateActor.scala:1489)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$21$adapted(KafkaStateActor.scala:1467)
        at scala.collection.Iterator.foreach(Iterator.scala:941)
        at scala.collection.Iterator.foreach$(Iterator.scala:941)
        at scala.collection.AbstractIterator.foreach(Iterator.scala:1429)
        at scala.collection.MapLike$DefaultKeySet.foreach(MapLike.scala:181)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$20(KafkaStateActor.scala:1467)
Caused by: java.lang.IllegalArgumentException: Login module not specified in JAAS config
        at org.apache.kafka.common.security.JaasConfig.<init>(JaasConfig.java:66)
        at org.apache.kafka.common.security.JaasContext.load(JaasContext.java:90)
        at org.apache.kafka.common.security.JaasContext.loadClientContext(JaasContext.java:84)
        at org.apache.kafka.common.network.ChannelBuilders.create(ChannelBuilders.java:124)
        at org.apache.kafka.common.network.ChannelBuilders.clientChannelBuilder(ChannelBuilders.java:67)
        at org.apache.kafka.clients.ClientUtils.createChannelBuilder(ClientUtils.java:112)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:726)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:659)
        at org.apache.kafka.clients.consumer.KafkaConsumer.<init>(KafkaConsumer.java:639)
        at kafka.manager.actor.cluster.KafkaStateActor$KafkaTopicOffsetGetter.$anonfun$run$21(KafkaStateActor.scala:1489)
2019-09-24 18:05:43,569 - [INFO] k.m.a.c.KafkaManagedOffsetCache - Consumer created for kafka offset topic consumption for cluster kafka-beta
2019-09-24 18:05:43,726 - [WARN] o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=KMOffsetCache-iZbp11w2fps1wt2ckilxfhZ] Error connecting to node IP(手动替换了):9093 (id: -1 rack: null)
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:343)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:325)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:257)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:474)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:255)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
        at org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureCoordinatorReady(AbstractCoordinator.java:235)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:213)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:334)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:325)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:257)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
        at org.apache.kafka.clients.NetworkClient.ready(NetworkClient.java:287)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.trySend(ConsumerNetworkClient.java:474)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:255)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:215)
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:557)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:164)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:254)
        at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:202)
        at org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:140)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:210)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:334)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:325)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:257)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
2019-09-24 18:05:43,728 - [WARN] o.a.k.c.NetworkClient - [Consumer clientId=consumer-1, groupId=KMOffsetCache-iZbp11w2fps1wt2ckilxfhZ] Error connecting to node IP(手动替换了):9094 (id: -2 rack: null)
java.io.IOException: Channel could not be created for socket java.nio.channels.SocketChannel[closed]
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:343)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:325)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:257)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
        at org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:67)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1090)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:976)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:533)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:236)
Caused by: org.apache.kafka.common.KafkaException: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:213)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:334)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:325)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:257)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)
        at org.apache.kafka.clients.NetworkClient.access$700(NetworkClient.java:67)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:1090)
        at org.apache.kafka.clients.NetworkClient$DefaultMetadataUpdater.maybeUpdate(NetworkClient.java:976)
        at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:533)
        at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:265)
Caused by: org.apache.kafka.common.errors.SaslAuthenticationException: Failed to configure SaslClientAuthenticator
Caused by: org.apache.kafka.common.KafkaException: Principal could not be determined from Subject, this may be a transient failure due to Kerberos re-login
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.firstPrincipal(SaslClientAuthenticator.java:557)
        at org.apache.kafka.common.security.authenticator.SaslClientAuthenticator.<init>(SaslClientAuthenticator.java:164)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildClientAuthenticator(SaslChannelBuilder.java:254)
        at org.apache.kafka.common.network.SaslChannelBuilder.lambda$buildChannel$1(SaslChannelBuilder.java:202)
        at org.apache.kafka.common.network.KafkaChannel.<init>(KafkaChannel.java:140)
        at org.apache.kafka.common.network.SaslChannelBuilder.buildChannel(SaslChannelBuilder.java:210)
        at org.apache.kafka.common.network.Selector.buildAndAttachKafkaChannel(Selector.java:334)
        at org.apache.kafka.common.network.Selector.registerChannel(Selector.java:325)
        at org.apache.kafka.common.network.Selector.connect(Selector.java:257)
        at org.apache.kafka.clients.NetworkClient.initiateConnect(NetworkClient.java:920)

启动命令

/usr/local/install/kafka-manager-2.0.0.2/bin/kafka-manager -Dconfig.file=/usr/local/install/kafka-manager-2.0.0.2/conf/application.conf -Dhttp.port=9095 -Djava.security.auth.login.config=/usr/local/install/kafka23/config/kafka_manager_jaas.conf

kafka_manager_jaas.conf为以下:

KafkaClient {
        org.apache.kafka.common.security.plain.PlainLoginModule required
        serviceName="kafka"
        username="xxx"
        password="xxx";
};
发表于 2019-09-24
.
添加评论
Login module not specified in JAAS config

在 Kafka 配置 SASL 后,需要对 kafka-manager 增加相应配置,监控 offset 等指标。
可以搜索一下 kafka manager sasl

. -> 半兽人 4年前

感谢回答,第一次用kafka-manager,感觉资料太少了,解决问题后再来反馈,谢谢!

Consumers consuming from this topic这里看不到消费者是什么情况 kafkamanager.我也没有用权限啊

你的答案

查看kafka相关的其他问题或提一个您自己的问题