Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

Kafka consumers on Sentinel are not shown on kafka-manager

Description

I was trying latest development snapshot of Sentinel, from its RPMs, and its Docker image through Kubernetes:

https://github.com/OpenNMS/opennms-drift-aws/tree/release/horizon-23

https://github.com/OpenNMS/opennms-drift-kubernetes/tree/release/horizon-23

In both cases, I saw that the solution appears to be working as expected, as I can see flow data on Elasticsearch, and Telemetryd is not handling flows on the OpenNMS Core instance; only Sentinel instances are processing flow data.

Typically, when debugging Kafka clients, I use Kafka Manager (a tool from Yahoo https://github.com/yahoo/kafka-manager), to inspect the Kafka cluster, see the amount of messages per topic, see the number of consumers currently connected and details about them, message lag, etc.

This tools shows the consumers from OpenNMS for Sink, RPC and Flows when Telemetryd is handling flows within OpenNMS. Now, when using Sentinel, the consumers for the flow topics are not shown.

Interestingly, if you manually check Zookeeper, or use the kafka-consumer-groups.sh command on any Kafka node, you can see the consumers, but after specifying additional flags; as they are not registered on the usual way.

For example:

The following command returns nothing:

/opt/kafka/bin/kafka-consumer-groups.sh --describe --group Sentinel --bootstrap-server kafka1:9092

But, the following returns some information:

[ec2-user@kafka1 bin]$ /opt/kafka/bin/kafka-consumer-groups.sh --describe --group Sentinel --bootstrap-server kafka1:9092 --members --verbose CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT consumer-3-770960d3-fc8f-4843-b7f6-a564fb48871f /18.191.19.2 consumer-3 4 OpenNMS.Sink.Telemetry-Netflow-9(8,9,10,11) consumer-2-79f33919-4971-4574-8e76-5e30cc5e7264 /18.191.19.2 consumer-2 4 OpenNMS.Sink.Telemetry-Netflow-5(8,9,10,11) consumer-13-9f5ea5a7-44c0-4e3c-af4f-bcc33954b307 /18.191.19.2 consumer-13 4 OpenNMS.Sink.Telemetry-Netflow-9(4,5,6,7) consumer-7-e3f51ab9-622b-4a5e-8b2a-a9c360db09ab /18.191.19.2 consumer-7 4 OpenNMS.Sink.Telemetry-Netflow-9(12,13,14,15) consumer-5-478e9b87-0665-4c40-b69b-2426c2ae81f1 /18.191.19.2 consumer-5 4 OpenNMS.Sink.Telemetry-Netflow-5(12,13,14,15) consumer-10-22ac5f9b-5a9b-4aa9-8d4c-90cd75ec7345 /18.191.19.2 consumer-10 4 OpenNMS.Sink.Telemetry-Netflow-9(0,1,2,3) consumer-11-d18cd114-a3a1-4dd2-a950-a53adf17ff7f /18.191.19.2 consumer-11 4 OpenNMS.Sink.Telemetry-Netflow-5(0,1,2,3) consumer-1-f9928728-48e9-4ef1-8713-e8e4574b71ba /18.191.19.2 consumer-1 50 __consumer_offsets(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49) consumer-16-f0da778a-ff46-4558-a768-04198b833b35 /18.191.19.2 consumer-16 4 OpenNMS.Sink.Telemetry-Netflow-5(4,5,6,7)

Note that --members has been passed in order to see the data.

If you just list the consumer groups, the Sentinel group is shown (also in kafka-manager), but no client information is displayed.

I've been using Kafka Manager a lot, not only with OpenNMS, but also for personal projects, and I've never seen this behavior before with consumer groups.

I can provide more details if required, and I can even provide access to any of my lab. Please let me know which environment you want to check.

Environment

Running latest RPMs from the develop branch on the Core OpenNMS, and latest RPMs from the features/sentinel branch on 2 Sentinel VMs

Acceptance / Success Criteria

None

Attachments

1
  • 09 Aug 2018, 12:59 PM

Lucidchart Diagrams

Activity

Show:

Alejandro Galue August 10, 2018 at 3:44 PM

The solution for https://opennms.atlassian.net/browse/HZN-1343#icft=HZN-1343 has fixed the problem reported on this issue. I was able to verify that with latest RPMs from the features/sentinel branch, the consumer groups statistics are properly shown on kafka-manager.

Alejandro Galue August 9, 2018 at 3:42 PM

I've updated my lab to use the RPMs from features/branch everywhere and now the consumer groups clients are NOT displayed regardless which consumer group you're looking at. You can only see members, meaning that you won't be able to use kafka-manager to see consumer statistics.

Something has changed on the Kafka handling within OpenNMS and Sentinel preventing Kafka to see consumer statistics.

Alejandro Galue August 9, 2018 at 1:00 PM

The attached file kafka.txt contains the whole session showing that Sentinel doesn't show the consumers, but only the members of the consumer groups, which is not the case of the other consumer groups maintained by OpenNMS.

Fixed

Details

Assignee

Reporter

Labels

Components

Fix versions

Affects versions

Priority

PagerDuty

Created August 6, 2018 at 10:12 PM
Updated August 10, 2018 at 3:44 PM
Resolved August 10, 2018 at 3:44 PM

Flag notifications