If you are new to Kafka, please read the first three posts of the series given below. Else dive in.
Reliable Data Delivery in Kafka
Troubleshooting Under Replicated Kafka Partitions
If you are preparing for an interview, this post contains most of the things that you should know about Kafka.
Active Controller Count
Active controller is one of the brokers of Kafka cluster which is designated to do administrative tasks like reassigning partitions. The active controller count metric tells us id the broker is the controller for cluster or not. The value of this metric could be 0 or 1. This metric is emitted per broker.
What if two brokers say that they are the controller?
The active controller count metric indicates whether the broker is currently the controller for the cluster. The metric will either be 0 or 1, with 1 showing that the broker is currently the controller. Kafka cluster require one broker to be the controller and only one broker can be a controller at any given time.
What should you do when more than one broker claims to become controller?
This situation will affect the administrative tasks of cluster. The first step could be restart of the brokers claiming to be controller.
Metric Name
kafka.controller:type=KafkaController,name=ActiveControllerCountRequest Handler Idle Ratio
Following are the two thread pools used by Kafka to handle requests:
Network Handlers
These are responsible for reading and writing data to the clients across the network. This does not require significant processing, so network handler don’t get exhausted easily.
Request Handlers
The request handler threads, however, are responsible for servicing the client request itself, which includes reading or writing the messages to disk. The request handler idle ratio metric indicates the percentage of time the request handlers are not in use. The lower this number, the more loaded the broker is. It is advisable to check the cluster for size or any other potential problem if the idle ratios goes lower than 20%.
Kafka uses purgatory to efficiently handle requests.
Read about purgatory here.
Metric Name
kafka.server:type=KafkaRequestHandlerPool,name=RequestHandlerAvgIdlePercentAll Topics Bytes In
The all topics bytes in rate, expressed in bytes per second, is useful as a measurement of how much message traffic your brokers are receiving from producing clients. This is a good metric to trend over time to help you determine when you need to expand
the cluster or do other growth-related work. It is also useful for evaluating if one broker in a cluster is receiving more traffic than the others, which would indicate that it is necessary to rebalance the partitions in the cluster.
Metric Name
kafka.server:type=BrokerTopicMetrics,name=BytesInPerSecAll Topics Bytes Out
The all topics bytes out rate, similar to the bytes in rate, is another overall growth metric. In this case, the bytes out rate shows the rate at which consumers are reading messages out. The outbound bytes rate may scale differently than the inbound bytes rate.
The outbound bytes rate also includes the replica traffic. This means that if all of the topics are configured with a replication factor of 2, we will see a bytes out rate equal to the bytes in rate when there are no consumer clients.
Metric Name
kafka.server:type=BrokerTopicMetrics,name=BytesOutPerSecOther Important Kafka Broker Metrics
| Name | Description | Metrics Name |
| All topics messages in | The messages in rate shows the number of individual messages, regardless of their size, produced per second. This is useful as a growth metric as a different measure of producer traffic. | kafka.server:type=BrokerTopicMetrics,name=MessagesInPerSec |
| Partition count | The partition count for a broker generally doesn’t change that much, as it is the total number of partitions assigned to that broker. This includes every replica the broker has, regardless of whether it is a leader or follower for that partition. | kafka.server:type=ReplicaManager,name=PartitionCount |
| Leader count | The leader count metric shows the number of partitions that the broker is currently the leader for. As with most other measurements in the brokers, this one should be generally even across the brokers in the cluster. | kafka.server:type=ReplicaManager,name=LeaderCount |
| Offline partitions | This measurement is only provided by the broker that is the controller for the cluster (all other brokers will report 0), and shows the number of partitions in the cluster that currently have no leader. | kafka.controller:type=KafkaController,name=OfflinePartitionsCount |
Reference:
Kafka – The Definitive Guide by Neha Narkhede, Gwen Shapira & Todd Palino