Prerequisite

A basic understanding about Apache Kafka. Please go through the introduction once, should you need it.

Installation

Download the Kafka from here.

Untar the tar file that is downloaded.

tar -xzf kafka_2.11-2.1.0.tgz

Run The Server

cd kafka_2.11-2.1.0

Start the zookeeper first

bin/zookeeper-server-start.sh config/zookeeper.properties

Zookeeper would be running at 2181 port. You can check that in the following log:

[2018-12-25 14:32:23,634] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)

Start the Kafka Server in another tab.

bin/kafka-server-start.sh config/server.properties 

Kakfa server would be waiting for connection at 9092.

[2018-12-25 14:35:40,373] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)    

You can see the kafka config values printed in the logs, a portion of that is shown here:

[2018-12-25 14:35:39,962] INFO KafkaConfig values:

advertised.host.name = null

advertised.listeners = null

advertised.port = null

…………………………………………….

Creating a Topic

In another command tab, run the following command:

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic myFirstTopic

You should see following message after creating the topic:

Created topic “myFirstTopic”.

Creating Console Producer

Run the following command:

bin/kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic

You should be able to see a kafka command prompt there.

Write some messages there.

Creating Kafka Console Consumer

bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --from-beginning

Run the above command in a new tab and you should be able to see all the messages that were produced before:

Kakfa Java Producer

We have seen how to produce  and consume records using Kafka console producer and consumer. We will now produce and consume using kafka java client.

To demonstrate the Kafka in Java, we will create a dropwizard project

We will follow the following steps:

1. Create a dropwizard project
2. Add dependency of kafka client in pom.xml
3. Create an API to post data, let’s name it /bulk-light
4. bulk-light will put data in Kafka Queue which will then be consumed by a kafka consumer in same project

Set Up

Set up a maven project. Use the following dependency to add kafka java client:

<dependency>
    <groupId>org.apache.kafka</groupId>
    <artifactId>kafka_2.12</artifactId>
    <version>2.1.0</version>
</dependency>

Kafka Producer

Kafka Consumer

Initialising Kafka Consumer

Download the demo project from github

Please download the demo project using the link below. Run it on your local to understand it better. I will write another blog post to explain the demo project.

https://github.com/rohitsingh20122992/kafka-demo

Further Reading:

Kafka Official Documentation

Dropwizard Introduction

https://www.cloudkarafka.com/blog/2016-11-30-part1-kafka-for-beginners-what-is-apache-kafka.html

https://github.com/rohitsingh20122992/kafka-demo