Tutorial For Kafka Monitoring Using JMX, Prometheus And Grafana

If you are new to Kafka, please read the first three posts of the series given below. Else dive in. 

Introduction to Kafka

Kafka Internals

Reliable Data Delivery in Kafka

How do your monitor your Kafka setup?

There are a number of measurement collected while Kafka is operational. We are going to collect these measurements in Prometheus and later make a dashboard in Grafana. All of the metrics exposed by Kafka can be accessed via the Java Management Extensions (JMX) interface. One way to use them in an external monitoring system is to use a collection agent provided by Prometheus and attach it to the Kafka process.

What is JMX?

JMX provides API to monitor and manage your resources at runtime.  It provides Java developers with the means to instrument Java code, create smart Java agents, implement distributed management middleware and managers, and easily integrate these solutions into existing management and monitoring systems.

Application, in our case, Kafka has to implement an interface called MBean of Java. You can see the implementation here. Kafka created the MBean and registered it. We need to now access the data by exposing it in some manner. We can get access through JConsole or through the communication adaptor or connectors available. In this tutorial, we are going to use JMX Prometheus exporter. There are other exporters too like Nagios XI check_jmx plugin, jmxtrans, Jolokia and MX4J.

Quickstart Kafka

If your kafka broker is not set up, please follow the step by step guide here.

JMX Prometheus Java Agent

Java agents are part of the Java Instrumentation API. The Instrumentation APIs provide a mechanism to modify bytecodes of methods. This can be done both statically and dynamically. This means that we can change a program by adding code to it without having to touch upon the actual source code of the program. The result can have a significant impact on the overall behavior of the application. JMX Prometheus exporter is a collector that can bed configured to scrape and expose mBeans of a JMX target. The collected data in Promethues can be later shown in Grafana. This exporter is intended to be run as a Java Agent, exposing a HTTP server and serving metrics of the local JVM.

Download the JMX Prometheus Java agent by the following command:

sudo wget -P /opt/kafka/prometheus/ https://repo1.maven.org/maven2/io/prometheus/jmx/jmx_prometheus_javaagent/0.12.0/jmx_prometheus_javaagent-0.12.0.jar

Download the config for java agent:

cd /opt/kafka/prometheus/
wget https://raw.githubusercontent.com/prometheus/jmx_exporter/master/example_configs/kafka-2_0_0.yml

The description of yml file can be found here. Basically, the conifguration is used to transform the metrics in the way prometheus understands.

Setting Kafka Options

Kafka_OPTS is an environment variable that you can set to pass custom settings to the Java Virtual Machine (JVM) that runs Kafka

export KAFKA_OPTS="-javaagent:/opt/kafka/prometheus/jmx_prometheus_javaagent-0.12.0.jar=7071:/opt/kafka/prometheus/kafka-2_0_0.yml"

Setting Heap Options

KAFKA_HEAP_OPTS is an environment variable that you can set to pass custom heap settings to the Java Virtual Machine (JVM) that runs Kafka

export KAFA_HEAP_OPTS="-Xmx1000M -Xms1000M"

Start your kafka

bin/kafka-server-start.sh config/server.properties

Metrics exposed over http

You can check metrics at localhost:7071

Scrape Metrics data in Prometheus

Follow the instruction here to download Prometheus. Add localhost:7071 as the scrape target as shown below.

# my global config
global:
  scrape_interval:     5s # Set the scrape interval to every 15 seconds. Default is every 1 minute.
  evaluation_interval: 5s # Evaluate rules every 15 seconds. The default is every 1 minute.
  # scrape_timeout is set to the global default (10s).

# Alertmanager configuration
alerting:
  alertmanagers:
  - static_configs:
    - targets:
      # - alertmanager:9093

# Load rules once and periodically evaluate them according to the global 'evaluation_interval'.
rule_files:
  # - "first_rules.yml"
  # - "second_rules.yml"

# A scrape configuration containing exactly one endpoint to scrape:
# Here it's Prometheus itself.

scrape_configs:
  - job_name: 'kafka-monitoring'
    #metrics_path: '/actuator/prometheus'
    static_configs:
      - targets: ['localhost:7071']

Run the prometheus using following command.

 ./prometheus --config.file=prometheus.yml

You should be able to see following screen on prometheus for any kafka metrics of your choice. I have randomly picked

kafka_server_kafkaserver_yammer_metrics_count as the metric.

Display the metrics in Grafana

Download the grafana from here.

Connect the prometheus as the data source as shown below.

Download the dashboard json file from here.

Import the downloaded json in prometheus and you will see a functional dashboard.

---

Like the post? Please subscribe to the blog to get regular updates.