Install and configure the Kafka plugin to monitor the fault-tolerant, high capacity messaging middleware Kafka. Take informed troubleshooting decisions by keeping track of critical metrics like connection count, incoming and outgoing bytes rate and lot more.
This document details how to configure the Apache Kafka plugin and the monitoring metrics for providing in-depth visibility into the performance, availability, and usage stats of Kafka servers.
Apache Kafka performance monitoring metrics:
Use our wide array of metrics and get notified of hazardous errors that require your attention. Keep track of unexpected trends through our metric graphs and troubleshoot as quickly as possible. Various out-of-the-box metrics we support are:
Connection count
The connection_count gives the current number of active connections with the kafka cluster
Incoming/outgoing byte rate
The incoming_byte_rate and outgoing_byte_rate gives the average number of incoming bytes received per second and outgoing bytes sent per second respectively
Average request latency
A measure of time between a request sent by the producer till a response received from the broker is taken as the avg_request_latency
Number of network operations
The average number of network operations (read or write) on all connections per second is given as the network_io_rate
Request rate
The rate at which producers send data to brokers is calculated by the metric request_rate
Response rate
The response_rate shows the rate of response received from the brokers
Time spent by the input/output thread
io_time_ns_avg gives the average length of time the input/output thread spends on waiting for a socket (in ns)
How it works?
- Log-in to your Site24x7 account. Sign up here if you don't have one.
- Download and install the latest version of Site24x7 Linux agent
- Install the Kafka plugin
- The agent will execute the Kafka plugin and push the data to the Site24x7 server
Prerequisites:
- Download and extract the 'kafka-python-1.3.1.tar.gz' file from https://pypi.python.org/pypi/kafka-python
- Download the file from https://pypi.python.org/pypi/kafka-python
sudo wget https://pypi.python.org/packages/81/94/f9151888b40339f2624759df8241080020acafed49d3a4369ceedd72c096/kafka-python-1.3.1.tar.gz#md5=2494d82f096d21af7a72b3c5bd26bdfa - Extract the file
sudo tar -xvzf kafka-python-1.3.1.tar.gz - Go to the directory where the file is extracted
cd kafka-python-1.3.1/ - Install the module
sudo python setup.py build
sudo python setup.py install
Kafka plugin installation:
- Create a directory with the name "kafka_producer", under Site24x7 Linux agent plugin directory - /opt/site24x7/monagent/plugins/
sudo mkdir kafka_producer
- Download the "kafka_producer.py" from our GitHub repository and place it under the "kafka_producer" directory
- The default python path given in the plugin script is #!/usr/bin/python. If you wish to provide an alternate python path, replace the existing one preceded by the shebang character "#!".
Eg : #!/usr/local/bin/python3
- Change the values of BROKER_NAME and PORT to match your configuration
- The server agent will report stats on the performance of Kafka under the Plugins tab in the Site24x7 web client. In case the plugin is not listed in the Site24x7 web client, restart the agent.
sudo /etc/init.d/site24x7monagent restart