This post describes how to quickly setup development environment for Apache Kafka in Docker on Windows. Apache Kafka is distributed streaming platform, which allows you to build reliable messaging channel between various systems. Its functionality supports publish/subscribe model, reliable message persistence, real-time stream processing and many more.

When I start learning any new technology, I want to get my hands drity as fast as possible, without spending too much time on environment setup. I wrote this post, to help you get started with Apache Kafka as fast as possible. In this post I cover full setup on Windows, but since it uses Docker, the process should be easily applicable for Linux/Mac machines. Besides Apache Kafka itself, the setup involves lauching Kafka-Manager UI (developed by Yahoo), so you will be able to manage your Kafka instance with ease. Let’s get started!

TL;DR

How to setup development environment for Apache Kafka + Kafka-Manager on Windows (using Docker).

Prerequisites

  • Docker installed (I use version 17.03.1-ce on Windows)
  • docker-compose installed (I use version 1.11.2)

Not required

  • knowledge of Docker
  • knowledge of Apache Kafka

Setup Apache Kafka with docker-compose

The setup will contain two containers:

  1. Apache Kafka container (with Zookeeper built-in)
  2. Kafka-Manger container

In order to join those two containers, we will use docker-compose. Create docker-compose.yml file and put the following content:

version: "2"

services:
  kafkaserver:
    image: "spotify/kafka:latest"
    container_name: kafka
    hostname: kafkaserver
    networks:
      - kafkanet
    ports:
      - 2181:2181
      - 9092:9092
    environment:
      ADVERTISED_HOST: kafkaserver
      ADVERTISED_PORT: 9092
  kafka_manager:
    image: "mzagar/kafka-manager-docker:1.3.3.4"
    container_name: kafkamanager
    networks:
      - kafkanet
    ports:
      - 9000:9000
    links:
      - kafkaserver
    environment:
      ZK_HOSTS: "kafkaserver:2181"

networks:
  kafkanet:
    driver: bridge

This docker-compose file decalres two mentioned containers. First one, kafkaserver runs Kafka image developed by Spotify team (it includes Zookeeper). It exposes itself under kafkaserver hostname in kafkanet network. Two environment variables need to be set:

  • ADVERTISED_HOST: kafkaserver
  • ADVERTISED_PORT: 9092

Those two describe to which hostname the Kafka server should respond from within the container.

Second container, kafka_manager runs newest (at the time of writing this post) kafka-manager tool. It needs to connect to the running Kafka server, that’s why there is the link entry in docker-compose:

links:
    - kafkaserver

Environment variable ZK_HOST also needs to be set in order to connect it to our Kafka server. We use hostname from kafkaserver container in order to achieve that.

Last thing in docker-compose file is network declaration, we use simple bridge networking there, so both containers can see one another.

Starting the environment

Once the docker-compose.yml file is ready, open your favorite terminal in the folder which contains it and run:

docker-compose up

After running this command, you can check status of the containers by invoking:

docker-compose ps

You should see output similar to this one:

Starting Apache Kafka in Docker on Windows

Creating Kafka topic through Kafka-Manger UI

Now when the Kafka server is up and running, we can create a Kafka Topic. Topic in the Apache Kafka terminology is an isolated fragment to which multiple applications/systems can send messages and from which multiple consumers can receive (subscribe) data. One Kafka server can have any number of topics. I would say that Kafka Topic is equivalent of Table in SQL databases. It’s a separate category, isolated from the rest of the system.

Let’s launch Kafka-Manager, by opening

http://localhost:9000

in the browser:

Kafka Manager on Windows in Docker

Creating Kafka Cluster

As you can see, there are no Clusters yet (since it’s still an empty environment). Cluster is where Kafka stores all the data pushed to the topics. Let’s create one by using the Cluster menu:

Create Kafka Cluster on Windows in Docker

For the quick setup, you can enter only:

  • Cluster Name - the name of your choice
  • Cluster Zookeeper Hosts - hostname of zookeeper (which is kafkaserver as configured in docker-compose.yml file)
  • Kafka Version 0.10.1.0 - which is the latest version at the time of writing this post

After entering those, click Save. You should be redirected to the homepage and see the cluster there:

Kafka Cluster on Windows in Docker

Creating Kafka Topic

Now, when the cluster is present, navigate to it’s properties and open Create Topic from the menu:

Create Kafka Topic on Windows in Docker

Entering only Topic name is enough. Just hit Create from there. Done! Topic is created!

Testing Apache Kafka on Windows

It is done, environment is up and running. Let’s test it. The quickest way to do this is to use Apache-provided scripts and just launching two of them - one for the producer and one for the subscriber. In order to get those scripts, download the folowing file (it’s the actual Apache Kafka distribution, but we only need scripts from it):

https://www.apache.org/dyn/closer.cgi?path=/kafka/0.10.2.0/kafka_2.11-0.10.2.0.tgz

and extract it. Then follow the path to the following folder:

kafka_2.11-0.10.2.0.tgz\kafka_2.11-0.10.2.0.tar\kafka_2.11-0.10.2.0\bin\windows\

This folder contains bunch of script files:

Name
----
connect-distributed.bat
connect-standalone.bat
kafka-acls.bat
kafka-configs.bat
kafka-console-consumer.bat
kafka-console-producer.bat
(...)
zookeeper-server-start.bat
zookeeper-server-stop.bat
zookeeper-shell.bat

We are interested in kafka-console-consumer.bat and kafka-console-producer.bat, but before launching them, we need one more thing. In order to connect to Kafka in Docker from the Windows host, we need to map Kafka container hostname to the localhost, by editing hosts file:

C:\Windows\System32\drivers\etc\hosts

Append this line:
127.0.0.1 kafkaserver

Ok, let’s launch Kafka Consumer first:

K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-consumer.bat --bootstrap-server localhost:9092 --topic kafka_test
_topic

After launching this script, PowerShell window will hang, that’s a good thing. Open second terminal and launch Kafka Producer:

PS K:\kafka_2.11-0.10.2.0\bin\windows> .\kafka-console-producer.bat --broker-list localhost:9092 --topic kafka_test_topi
c

Type any message and hit ENTER, you should see the messages flowing, like this:

Testing Apache Kafka on Windows in PowerShell

Stopping Apache Kafka docker

When the work is done, you can easily turn off running containers by invoking:

docker-compose down

in the folder when you have your docker-compose.yml file:

Stop Apache Kafka docker image

Summary

Now you have Apache Kafka with Kafka-Manger development environment up and running on your Windows machine. I hope that this short post helped you with the setup. Please share or leave a comment if you liked it!

Additional links & resources

Tags

apache-kafka kafka docker windows

Comments