KCPilot Under the Hood: Building an AI-Powered Kafka Diagnostics Tool in Rust

Fri, 17 Oct 2025 12:58:00 +0000
Kafka, Apache Kafka, Rust, KCPilot, Diagnostics Tool

If you have ever gone beyond running Kafka on your local machine and moved your ideas to real-world production projects utilizing Kafka, you probably know the drill - Kafka clusters are complex, and the problems you can encounter within your Kafka infrastructure have endless possibilities.

Introduction: The problem space

There are multiple aspects to this, starting from your setup (which is complex by itself and you need to understand many things about Kafka to make it right and secure) and ending in day-to-day operations, where a new set of problems arise and you need to get to the PhD level of expertise to understand what is going on and how that can affect your business.

When investigating your Kafka cluster, you often need to gather data from multiple sources, check out the configuration, look into the logs, and check the JMX or Prometheus metrics to get started.

Many of the problems you can usually find on existing Kafka clusters are repeated and are common errors made by DevOps or other people on your development team who set the cluster up at the beginning of its life. It’s been running fine for ages until the time you need to investigate because something is off with your data.

At Softwaremill, we are trying to gather that tribal knowledge into a standard set of problems that can be automatically discovered on any Kafka cluster. For this exact purpose, we have created a new open-source project called KCPilot.

KCPilot is a CLI tool that helps you gather all the data and analyze it to identify common problems that can occur in your Kafka cluster. It utilizes LLM to execute each analysis task and is easily extensible, allowing us to add new tasks and solve more problems in the future.

KCPilot is part of SoftwareMill’s Innovation Hub and is currently in the MVP stage. While functional, it may contain bugs and has significant room for improvement. We welcome feedback and contributions.Treat this project as a base, invent your own tasks, and share them with the world by opening a pull request.

The tool is written in Rust, and is a simple CLI tool that you can run locally or on the bastion server. All you need to provide (at least for now) is the ssh access to your workers (or also to your bastion server if you run it locally and want to inspect the production cluster).

KCPilot scans your workers for configuration files, logs, and metrics, and stores them for analysis. Once the data is gathered, you can analyze it and produce a report showing you all the good and bad aspects of your current cluster setup. Currently, the tool has 17 built-in checks, but can be easily extended with new checks (tasks) which are created in yaml and are translated into LLM prompts with provided context from data gathered earlier.

This article is for Rust developers interested in building infrastructure tools (especially CLI-based) as well as for Kafka administrators and all...