Benchmarking Made Simple with Benchi

Benchmarking is one of those things every developer faces one day. It all starts with a question. What kind of load can my app handle? What's the maximum throughput of my pipeline? Is that shiny new data tool really faster than the one I'm using?

We're excited to introduce Benchi, a minimal yet powerful benchmarking framework that makes it easy to answer these questions. It was originally built to benchmark Conduit, but it quickly became clear that Benchi could work for any tool.

Why Benchi?

Benchmarking is supposed to give you confidence. But if you've tried benchmarking tools before, you've probably run into this:

• The need to manually set up the environment.
• Metrics scattered across logs, dashboards, or worse — missing.
• No easy way to track progress or compare results across runs and tools.

Benchi fixes this. It's a framework for running and collecting benchmarks without reinventing the wheel for each project.

Key features of Benchi:

• ⚙️ Docker-Driven: Use Docker Compose to spin up isolated, reproducible benchmarking environments.
• 📊 Terminal UI: Real-time feedback while your benchmarks run. See metrics as they are collected.
• 📁 Metrics Collection: Performance data is automatically collected and saved as CSV for further analysis. If your app exposes a Prometheus endpoint, Benchi can collect metrics from it!
• 🪝 Custom Hooks: Run any script or command at any stage - before, during, or after the benchmark.
• 🔄 Multi-Tool Comparison: Benchmark multiple tools under identical conditions to see which one holds up.

Example: Benchmarking Conduit with Benchi

Let's say you want to benchmark Conduit streaming data between two Kafka topics. We prepared an example configuration, so this can be achieved using a one-liner:

benchi -config ./example/bench-kafka-kafka/bench.yml

Running this command will do the following:

• Run the infrastructure (Kafka) and tool (Conduit) in a clean, isolated Docker environment.
• Produce the test data using custom hooks.
• Display progress and collected metrics as the benchmark is running.
• Save all logs and metrics in the results folder.

Start Benchmarking Smarter

🚀 Try it now:

go install github.com/conduitio/benchi@latest

📚 Full docs + examples: github.com/conduitio/benchi

Let us know what you're benchmarking and how Benchi helps you optimize it!