A basic understanding about Apache Kafka. Please go through the introduction once, should you need it.
Untar the tar file that is downloaded.
tar -xzf kafka_2.11-2.1.0.tgz
Run The Server
Start the zookeeper first
Zookeeper would be running at 2181 port. You can check that in the following log:
[2018-12-25 14:32:23,634] INFO binding to port 0.0.0.0/0.0.0.0:2181 (org.apache.zookeeper.server.NIOServerCnxnFactory)
Start the Kafka Server in another tab.
Kakfa server would be waiting for connection at 9092.
[2018-12-25 14:35:40,373] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
You can see the kafka config values printed in the logs, a portion of that is shown here:
[2018-12-25 14:35:39,962] INFO KafkaConfig values:
advertised.host.name = null
advertised.listeners = null
advertised.port = null
Creating a Topic
In another command tab, run the following command:
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic myFirstTopic
You should see following message after creating the topic:
Created topic “myFirstTopic”.
Creating Console Producer
Run the following command:
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic myFirstTopic
You should be able to see a kafka command prompt there.
Write some messages there.
Creating Kafka Console Consumer
bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic myFirstTopic --from-beginning
Run the above command in a new tab and you should be able to see all the messages that were produced before:
Kakfa Java Producer
We have seen how to produce and consume records using Kafka console producer and consumer. We will now produce and consume using kafka java client.
To demonstrate the Kafka in Java, we will create a dropwizard project
We will follow the following steps:
- Create a dropwizard project
- Add dependency of kafka client in pom.xml
- Create an API to post data, let’s name it /bulk-light
- bulk-light will put data in Kafka Queue which will then be consumed by a kafka consumer in same project
Set up a maven project. Use the following dependency to add kafka java client:
Initialising Kafka Consumer
Download the demo project from github
Please download the demo project using the link below. Run it on your local to understand it better. I will write another blog post to explain the demo project.