Setup Apache Kafka in Docker on Windows

Marcin Zabłocki — Wed, 26 Apr 2017 22:00:00 +0000

This post describes how to quickly setup development environment for Apache Kafka in Docker on Windows. Apache Kafka is distributed streaming platform, which allows you to build reliable messaging channel between various systems. Its functionality supports publish/subscribe model, reliable message persistence, real-time stream processing and many more.

When I start learning any new technology, I want to get my hands drity as fast as possible, without spending too much time on environment setup. I wrote this post, to help you get started with Apache Kafka as fast as possible. In this post I cover full setup on Windows, but since it uses Docker, the process should be easily applicable for Linux/Mac machines. Besides Apache Kafka itself, the setup involves lauching Kafka-Manager UI (developed by Yahoo), so you will be able to manage your Kafka instance with ease. Let’s get started!

TL;DR

How to setup development environment for Apache Kafka + Kafka-Manager on Windows (using Docker).

Prerequisites

Docker installed (I use version 17.03.1-ce on Windows)
docker-compose installed (I use version 1.11.2)

Not required

knowledge of Docker
knowledge of Apache Kafka

Setup Apache Kafka with docker-compose

The setup will contain two containers:

Apache Kafka container (with Zookeeper built-in)
Kafka-Manger container

In order to join those two containers, we will use docker-compose. Create docker-compose.yml file and put the following content:

html

version: "2"

services:
  kafkaserver:
    image: "spotify/kafka:latest"
    container_name: kafka
    hostname: kafkaserver
    networks:
      - kafkanet
    ports:
      - 2181:2181
      - 9092:9092
    environment:
      ADVERTISED_HOST: kafkaserver
      ADVERTISED_PORT: 9092
  kafka_manager:
    image: "mzagar/kafka-manager-docker:1.3.3.4"
    container_name: kafkamanager
    networks:
      - kafkanet
    ports:
      - 9000:9000
    links:
      - kafkaserver
    environment:
      ZK_HOSTS: "kafkaserver:2181"

networks:
  kafkanet:
    driver: bridge

This docker-compose file decalres two mentioned containers. First one, kafkaserver runs Kafka image developed by Spotify team (it includes Zookeeper). It exposes itself under kafkaserver hostname in kafkanet network. Two environment variables need to be set:

ADVERTISED_HOST: kafkaserver
ADVERTISED_PORT: 9092

Those two describe to which hostname the Kafka server should respond from within the container.

Second container, kafka_manager runs newest (at the time of writing this post) kafka-manager tool. It needs to connect to the running Kafka server, that’s why there is the link entry in docker-compose:

html

links:
    - kafkaserver

Environment variable ZK_HOST also needs to be set in order to connect it to our Kafka server. We use hostname from kafkaserver container in order to achieve that.

Last thing in docker-compose file is network declaration, we use simple bridge networking there, so both containers can see one another.

Starting the environment

Once the docker-compose.yml file is ready, open your favorite terminal in the folder which contains it and run:

powershell

docker-compose up

After running this command, you can check status of the containers by invoking:

powershell

docker-compose ps

You should see output similar to this one:

Creating Kafka topic through Kafka-Manger UI

Now when the Kafka server is up and running, we can create a Kafka Topic. Topic in the Apache Kafka terminology is an isolated fragment to which multiple applications/systems can send messages and from which multiple consumers can receive (subscribe) data. One Kafka server can have any number of topics. I would say that Kafka Topic is equivalent of Table in SQL databases. It’s a separate category, isolated from the rest of the system.

Let’s launch Kafka-Manager, by opening

html

http://localhost:9000

in the browser:

Creating Kafka Cluster

As you can see, there are no Clusters yet (since it’s still an empty environment). Cluster is where Kafka stores all the data pushed to the topics. Let’s create one by using the Cluster menu:

For the quick setup, you can enter only:

Cluster Name - the name of your choice
Cluster Zookeeper Hosts - hostname of zookeeper (which is kafkaserver as configured in docker-compose.yml file)
Kafka Version 0.10.1.0 - which is the latest version at the time of writing this post

After entering those, click Save. You should be redirected to the homepage and see the cluster there:

Creating Kafka Topic

Now, when the cluster is present, navigate to it’s properties and open Create Topic from the menu:

Entering only Topic name is enough. Just hit Create from there. Done! Topic is created!

Testing Apache Kafka on Windows

It is done, environment is up and running. Let’s test it. The quickest way to do this is to use Apache-provided scripts and just launching two of them - one for the producer and one for the subscriber. In order to get those scripts, download the folowing file (it’s the actual Apache Kafka distribution, but we only need scripts from it):

html

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz

and extract it. Then follow the path to the following folder:

powershell

kafka_2.11-0.10.2.0.tgz\kafka_2.11-0.10.2.0.tar\kafka_2.11-0.10.2.0\bin\windows\

This folder contains bunch of script files:

powershell

Name
----
connect-distributed.bat
connect-standalone.bat
kafka-acls.bat
kafka-configs.bat
kafka-console-consumer.bat
kafka-console-producer.bat
(...)
zookeeper-server-start.bat
zookeeper-server-stop.bat
zookeeper-shell.bat

We are interested in kafka-console-consumer.bat and kafka-console-producer.bat, but before launching them, we need one more thing. In order to connect to Kafka in Docker from the Windows host, we need to map Kafka container hostname to the localhost, by editing hosts file:

powershell

C:\Windows\System32\drivers\etc\hosts

Append this line:
127.0.0.1 kafkaserver

Ok, let’s launch Kafka Consumer first:

powershell

K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic kafka_test
_topic

After launching this script, PowerShell window will hang, that’s a good thing. Open second terminal and launch Kafka Producer:

powershell

PS K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-producer.bat --broker-list localhost:9092 --topic kafka_test_topi
c

Type any message and hit ENTER, you should see the messages flowing, like this:

Stopping Apache Kafka docker

When the work is done, you can easily turn off running containers by invoking:

powershell

docker-compose down

in the folder when you have your docker-compose.yml file:

Summary

Now you have Apache Kafka with Kafka-Manger development environment up and running on your Windows machine. I hope that this short post helped you with the setup. Please share or leave a comment if you liked it!

Additional links & resources

full docker-compose.yml file used in this post: docker-compose.yml
Spotify Apache Kafka docker image: https://github.com/spotify/docker-kafka
Kafka-Manager docker image: https://hub.docker.com/r/mzagar/kafka-manager-docker/

apache-kafka on Marcin Zabłocki blog

Setup Apache Kafka in Docker on Windows

TL;DR

Prerequisites

Not required

Setup Apache Kafka with docker-compose

Starting the environment

Creating Kafka topic through Kafka-Manger UI

Creating Kafka Cluster

Creating Kafka Topic

Testing Apache Kafka on Windows

Stopping Apache Kafka docker

Summary

Additional links & resources