Concurrent Message Broker.

Why build a message broker?

Concurrent Message Broker is Sumit Kushwaha’s framework-free Java project exploring what happens between a producer sending an event and a consumer reading it later. The engineering problem is to preserve a clear ordering boundary, recover stored records after a restart, and keep concurrent work bounded when callers send faster than the broker can process.

The core separates producers and consumers from topic management, storage, and TCP networking. Java clients can use the broker API directly; remote clients enter through a binary TCP request/response protocol. Both paths converge on the same broker logic.

Architecture and request flow

1. Produce: a producer submits a record through an asynchronous CompletableFuture API. A bounded executor queues work, with caller-runs backpressure when the queue fills.
2. Route: round-robin or key-based partitioning chooses a topic partition. A partition write lock protects offset allocation and record insertion; independent partitions can progress concurrently.
3. Persist: durable mode appends framed records through FileChannel before exposing them in memory. Log segments roll at a configured size, and a sparse offset index supports seeking within stored records.
4. Consume: consumers fetch using a topic, partition, and offset. Per-group next-offset commits, with an optional offset journal, let separate groups progress independently and replay earlier records.

Correctness under concurrency and failure

Ordering is partition-local. A broker-wide message ID identifies a generated record, while the partition offset identifies its position in a particular stream. Keeping these concepts separate avoids implying a total order across every topic and partition.

The binary codec checks lengths, protocol fields, and CRC32 values. Startup scans log segments, rebuilds sparse indexes, and truncates an incomplete or corrupt tail to the last valid record boundary. TCP framing adds request correlation IDs and validation of malformed inputs.

The implementation uses immutable record data and defensive byte-array copies to prevent callers from mutating a payload while another thread stores or consumes it. Bounded producer and network worker pools make overload visible rather than allowing unlimited queued work.

Verification and trade-offs

The repository documents JUnit 5 coverage for concurrent publishing, persistence and restart recovery, consumer offsets, malformed frames, and TCP integration. These tests examine correctness boundaries rather than establishing a performance benchmark.

This is an educational systems project. Coordination utilities explore heartbeats and leader epochs, but there is no integrated replicated log or production consensus protocol. Consumer coordination is in-process, and durable offsets use a local journal. It is not a production-ready replacement for Apache Kafka.