Skip to content

Lab: Monitoring with Prometheus & Grafana

Description Link
Grafana https://grafana.sebastian.spielwiese.k8s.workshop.thinkport.cloud/
Kubernetes Dashboard https://dashboard.sebastian.spielwiese.k8s.workshop.thinkport.cloud/

Lab Exercise: Accessing Grafana

You should have been assigned a username to log in to Grafana. Your password is stored in a Kubernetes secret and can be retrieved through the Kubernetes Dashboard.

Get your password by viewing the secret grafana-users in the namespace monitoring. Try to log in to Grafana and verify to have access to the dashboard folder Strimzi Examples.

Lab Exercise: Activating the Prometheus jmxExporter

Step 1: Deploying the jmxExporter Config

Deploy the metrics configuration as a ConfigMap in your personal namespace. Use the file workspace/exercises/monitoring/kafka-metrics-config.yaml.

Solution

Replace MY_NAMESPACE with the name of your personal namespace. 

$ kubectl apply -n MY_NAMESPACE workspace/exercises/monitoring/kafka-metrics-config.yaml

Step 2: Activating the jmxExporter

Modify your cluster to activate the jxmExporter.

Hint: Documentation

Strimzi > Deploying > Metrics

Solution

Add the following to your Kafka cluster definition: 

spec:
  kafka: #...
    metricsConfig:
      type: jmxPrometheusExporter
      valueFrom:
        configMapKeyRef:
          name: kafka-metrics
          key: kafka-metrics-config.yml

Step 3: Look Up Your Cluster in Grafana

Now verify that your cluster is successfully scraped by Prometheus by opening the Grafana dashboard Strimzi Kafka, which can be found in Home > Dashboards > Strimzi Examples > Strimzi Kafka. In the header, select your namespace and cluster.

Hint: Dashboard Location

Link to Strimzi Kafka

Hint: Dashboard Filters

Lab Exercise: Watch Metrics Change

We would like to observe metrics change with traffic going into our cluster. To create our own statistics, we will be using the Explore feature of Grafana to display the number of messages in a specific topic as a graph. We will then publish messages into the same topic to observe the graph change.

Step 1: Build a Query

Using the Explore feature of Grafana, build a simple query by selecting the metric kafka_server_brokertopicmetrics_messagesin_total{topic="test-0"}. Now apply a label filter for topic and filter for your topic test-0. You now should see the total number of messages in your topic displayed as a graph.

Solution

Step 2: Publish Messages

Using the topic and commands from the Create Topic Exercises, publish some messages to the topic. Have a look at the graph, refreshing by running the query or enabling a 5s auto-refresh.

Solution

You can see the number of messages have increased after publishing some messages.

Step 3: Applying a Transformation

In a production Kafka setup, we can expect topics to grow to vast sizes exceeding hundred of gigabytes or even terabytes. In such cases, the total number of messages will become less relevant for monitoring. Instead, we could monitor the rate in which the total number increases or decreases to easily visualize peaks in load.

After enabling the auto-refresh to 5s, apply a simple rate to your query.

Hint: Rate Function

Add an “Operation” and select Range functions > Rate, leaving the rate parameter unchanged.

Solution

This graph shows that the total number of messages in the topic changed two times within the last 15 minutes. The first time was caused by producing 15 messages at once, the second time 30.