Event-driven CQRS with Kafka — 1.8M records in <10min

TL;DR: Built a CQRS-style event pipeline using Kafka for ingestion, Rust/node consumers for processing, and a split write/query model to achieve 1.8M records processed in under 10 minutes.

Problem & goals

• Ingest massive volumes without slowing writes.
• Maintain near-real-time projections for query consumers.
• Keep the system testable with E2E scenarios.

Architecture overview

• Ingestion: Producers publish to Kafka topics (partitioning by entity id)
• Processing: Consumer groups (Rust & Node.js) commit offsets carefully with batching
• Command side: Postgres for canonical writes (ACID)
• Query side: MongoDB projections for read-optimized queries
• Event store: Kafka acts as the append-only log

Performance tuning

• Carefully size partitions to match consumer parallelism.
• Use batched commits and idempotent consumers.
• Optimize serialization format (Avro / Protobuf recommended).
• Use consumer prefetch & flow-control to avoid GC spikes.

Results & metrics

• Records processed: ~1.8M
• Time: < 10 minutes in optimized runs
• Lessons: Partition key design and consumer concurrency make or break high-throughput pipelines.

Stack

Kafka · Rust · Node.js · PostgreSQL · MongoDB · Docker

Repo / demo

• Public repo and demo links available on my GitHub. See: https://github.com/Adityaadpandey. :contentReference[oaicite:2]{index=2}