This post describes how to quickly setup development environment for Apache Kafka in Docker on Windows. Apache Kafka is distributed streaming platform, which allows you to build reliable messaging channel between various systems. Its functionality supports publish/subscribe model, reliable message persistence, real-time stream processing and many more.
When I start learning any new technology, I want to get my hands drity as fast as possible, without spending too much time on environment setup. I wrote this post, to help you get started with Apache Kafka as fast as possible. In this post I cover full setup on Windows, but since it uses Docker, the process should be easily applicable for Linux/Mac machines. Besides Apache Kafka itself, the setup involves lauching Kafka-Manager UI (developed by Yahoo), so you will be able to manage your Kafka instance with ease. Let's get started!
How to setup development environment for Apache Kafka + Kafka-Manager on Windows (using Docker).
- Docker installed (I use version 17.03.1-ce on Windows)
- docker-compose installed (I use version 1.11.2)
- knowledge of Docker
- knowledge of Apache Kafka
Setup Apache Kafka with docker-compose
The setup will contain two containers:
- Apache Kafka container (with Zookeeper built-in)
- Kafka-Manger container
In order to join those two containers, we will use docker-compose. Create
docker-compose.yml file and put the following content:
version: "2" services: kafkaserver: image: "spotify/kafka:latest" container_name: kafka hostname: kafkaserver networks: - kafkanet ports: - 2181:2181 - 9092:9092 environment: ADVERTISED_HOST: kafkaserver ADVERTISED_PORT: 9092 kafka_manager: image: "mzagar/kafka-manager-docker:22.214.171.124" container_name: kafkamanager networks: - kafkanet ports: - 9000:9000 links: - kafkaserver environment: ZK_HOSTS: "kafkaserver:2181" networks: kafkanet: driver: bridge
This docker-compose file decalres two mentioned containers. First one, kafkaserver runs Kafka image developed by Spotify team (it includes Zookeeper). It exposes itself under kafkaserver hostname in kafkanet network. Two environment variables need to be set:
- ADVERTISED_HOST: kafkaserver
- ADVERTISED_PORT: 9092
Those two describe to which hostname the Kafka server should respond from within the container.
Second container, kafka_manager runs newest (at the time of writing this post) kafka-manager tool. It needs to connect to the running Kafka server, that's why there is the link entry in docker-compose:
links: - kafkaserver
ZK_HOST also needs to be set in order to connect it to our Kafka server. We use hostname from kafkaserver container in order to achieve that.
Last thing in docker-compose file is network declaration, we use simple bridge networking there, so both containers can see one another.
Starting the environment
docker-compose.yml file is ready, open your favorite terminal in the folder which contains it and run:
After running this command, you can check status of the containers by invoking:
You should see output similar to this one:
Creating Kafka topic through Kafka-Manger UI
Now when the Kafka server is up and running, we can create a Kafka Topic. Topic in the Apache Kafka terminology is an isolated fragment to which multiple applications/systems can send messages and from which multiple consumers can receive (subscribe) data. One Kafka server can have any number of topics. I would say that Kafka Topic is equivalent of Table in SQL databases. It's a separate category, isolated from the rest of the system.
Let's launch Kafka-Manager, by opening
in the browser:
Creating Kafka Cluster
As you can see, there are no Clusters yet (since it's still an empty environment). Cluster is where Kafka stores all the data pushed to the topics. Let's create one by using the Cluster menu:
For the quick setup, you can enter only:
- Cluster Name - the name of your choice
- Cluster Zookeeper Hosts - hostname of zookeeper (which is kafkaserver as configured in docker-compose.yml file)
- Kafka Version 0.10.1.0 - which is the latest version at the time of writing this post
After entering those, click Save. You should be redirected to the homepage and see the cluster there:
Creating Kafka Topic
Now, when the cluster is present, navigate to it's properties and open Create Topic from the menu:
Entering only Topic name is enough. Just hit Create from there. Done! Topic is created!
Testing Apache Kafka on Windows
It is done, environment is up and running. Let's test it. The quickest way to do this is to use Apache-provided scripts and just launching two of them - one for the producer and one for the subscriber. In order to get those scripts, download the folowing file (it's the actual Apache Kafka distribution, but we only need scripts from it):
and extract it. Then follow the path to the following folder:
This folder contains bunch of script files:
Name ---- connect-distributed.bat connect-standalone.bat kafka-acls.bat kafka-configs.bat kafka-console-consumer.bat kafka-console-producer.bat (...) zookeeper-server-start.bat zookeeper-server-stop.bat zookeeper-shell.bat
We are interested in
kafka-console-producer.bat, but before launching them, we need one more thing. In order to connect to Kafka in Docker from the Windows host, we need to map Kafka container hostname to the localhost, by editing hosts file:
C:\Windows\System32\drivers\etc\hosts Append this line: 127.0.0.1 kafkaserver
Ok, let's launch Kafka Consumer first:
K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic kafka_test _topic
After launching this script, PowerShell window will hang, that's a good thing. Open second terminal and launch Kafka Producer:
PS K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-producer.bat --broker-list localhost:9092 --topic kafka_test_topi c
Type any message and hit ENTER, you should see the messages flowing, like this:
Stopping Apache Kafka docker
When the work is done, you can easily turn off running containers by invoking:
in the folder when you have your
Now you have Apache Kafka with Kafka-Manger development environment up and running on your Windows machine. I hope that this short post helped you with the setup. Please share or leave a comment if you liked it!
Additional links & resources
- full docker-compose.yml file used in this post: docker-compose.yml
- Spotify Apache Kafka docker image: https://github.com/spotify/docker-kafka
- Kafka-Manager docker image: https://hub.docker.com/r/mzagar/kafka-manager-docker/