How to create a cluster in Kafka Manager using cURL

Kafka Manager is a great tool to check the status of Kafka topics. It’s created by Yahoo! and written in Scala. I found it quite handy and straightforward to utilize. Mainly, when I want to know about the lags, consumer groups, and message processing speed. In this post, I focus on how to create a cluster in Kafka Manager using cURL.

As I stated above, Kafka Manager is one of the best. However, it has one huge drawback. It does not come with any REST APIs. This means any configuration should be done through UI which is a pain in the neck, especially when dealing with Docker.

But don’t worry, we are geeks and suppose to have a solution to every problem!

To me personally, I can deal with the UI in every case except cluster configuration. To understand better, allow me to present my case. Let’s assume for development purposes I want to set up Kafka, ZooKeeper, and Kafka Manager in Docker. So that I can destroy the infrastructure easily and bring it up fast. The basic approach is to put all Kafka, ZooKeeper, and Kafka Manager in a single docker-compose file and use docker-compose up to bring the infrastructure. And hence expecting when hitting the Kafka Manager endpoint in the browser, be able to see the status of topics.

But wait, I can’t do that. Because Kafka Manager requires manual cluster creation and that’s a pain in the neck to do. And I don’t want every time that I do docker-compose up go and add the cluster to Kafka Manager. I want to have the Kafka Manager cluster be configured in docker-compose. And this is exactly where REST APIs become useful in which Kafka Manager doesn’t have one.

So what we have to do when REST APIs are not available is to mimic the POST HTML form with cURL. I went through many GitHub issues and discussions and found nothing useful. In the end, I had to track the POST request from my browser to find a solid and workable solution. All we have to do is to use cURL as follows to create a cluster in Kafka Manager,

Just need to replace the KAFKA MANAGER URL, CLUSTER NAME, ZOOKEEPER URL, and KAFKA VERSION based on existing configuration and then set the variables.

Then make a call and ignore the outcome as it will be HTML.

Finally, open the browser, should be able to see the new cluster created in Kafka Manager.