kafka-wire

A Kafka-compatible message broker in a single Go binary.

Your disk plus any S3 bucket. No ZooKeeper, no KRaft, no JVM, no cluster to operate.

Sixty seconds

docker run -p 9092:9092 -v kw:/data \
  -e KAFKA_WIRE_AUTH_ALLOWANON=true \
  ghcr.io/csnyder256/kafka-wire:latest

Then point any Kafka client at localhost:9092. Nothing about your code changes.

What it is

One binary

The broker, the producer, the consumer, the topic admin and the diagnostic tool are the same executable.

Kafka on the wire

19 protocol APIs, consumer groups, SASL/SCRAM, TLS, ACLs and Prometheus metrics. Real clients, unmodified.

Opaque to your data

JSON, Avro, Protobuf, images, invalid UTF-8. Keys, values and headers are bytes, and the tests assert byte equality.

Cold storage anywhere

A directory, or any S3-compatible store: MinIO, Cloudflare R2, Backblaze B2, Ceph, Wasabi, Garage, GCS.

What it is not

Read this part before adopting it.

Not replicated. One node, one copy. If the disk dies, everything not archived dies with it.
Not a Kafka feature replica. No transactions, no exactly-once, no Streams, no Connect runtime, no log compaction.
Not a distributed system. If you need multi-node durability, run Apache Kafka or Redpanda. That is the correct answer, not a hedge.

If your workload fits on one machine and you would rather own it than operate a cluster, this is built for you. If it does not, it is not.

Does my Kafka client work?

Yes. The one accommodation: any client that turns idempotent producing on by default needs it switched off, because idempotence requires an API a broker with no transaction coordinator does not have. That is the Apache Kafka Java client (default since 3.0) and kafka-python 3.x. Set enable.idempotence=false, or enable_idempotence=False in Python. librdkafka-based clients and kafka-python 2.x already default it off.

Client	Works
kcat, kafka-python 2.x	yes
kafka-python 3.x	yes, with the setting above
confluent-kafka (Python, Go, .NET), librdkafka	yes
KafkaJS, franz-go, Sarama, segmentio/kafka-go	yes
Apache Kafka Java client 3.x / 4.x	yes, with the setting above
Kafka Connect, Debezium	partial, no transactions
Kafka Streams, ksqlDB	no, needs transactions

Where can I run it?

Anywhere with a persistent disk and a raw TCP port: Docker, Kubernetes, systemd on a VM, Nomad, Fly.io, Railway, Render, Koyeb, Hetzner, EC2, a Raspberry Pi.

Platforms with an ephemeral filesystem or HTTP-only routing, such as Cloud Run, Heroku and Vercel, cannot run it, and the deployment guide says so plainly rather than letting you find out in production.

Configuration

It starts with no configuration at all. When you want some, every setting has one dotted path and one mechanically derived environment variable:

storage.datadir    ->  KAFKA_WIRE_STORAGE_DATADIR
archive.s3.bucket  ->  KAFKA_WIRE_ARCHIVE_S3_BUCKET

kafka-wire config print      # what is in effect, and where each value came from
kafka-wire config validate   # fail before you deploy, not after
kafka-wire doctor            # ports, disk, advertised address, object store

Where this came from

Built as ClarusStream, the message broker for the Clarus contract platform, to replace a managed Kafka service that cost more than the rest of the infrastructure combined. It has carried that platform's production traffic since May 2026. This repository is a rebuilt, vendor-neutral version: the proven protocol and storage core, with every assumption about one particular stack removed and replaced with a choice.