Streaming Data Pipelines

Streaming Data Pipelines is a practical engineering guide to building real-time data systems that are correct, scalable, and observable. It starts from a single observation: streaming is not batch with a shorter interval — it is a different discipline with its own semantics, failure modes, and engineering practices. The book walks through the full stack — why streaming matters and what it changes, streaming fundamentals including topics, partitions, consumer groups, and offsets, Apache Kafka in depth with its log-structured storage model, Apache Pulsar and the alternatives with their trade-offs, stream processing with Flink and its event-time model, Kafka Streams and ksqlDB for embedded stream processing, exactly-once semantics and how it is achieved end to end, event time vs. processing time and the watermark mechanism that tracks progress, state management and checkpointing, backpressure and flow control, streaming SQL and analytical patterns, production pipelines in practice, observability for streaming systems, and the trends reshaping the field. It covers the failure modes that quietly wreck streaming pipelines: a consumer that falls behind and never catches up because the partition count is fixed, a state store that grows without bound because the retention policy never triggers compaction, a watermark that advances too quickly and drops late events from the window, a transactional producer that is aborted mid-commit and leaves partial state in the output topic, a rebalance that pauses consumption for thirty seconds during peak traffic, an exactly-once guarantee that is broken because the sink is not transactional, a checkpoint that fails silently and the application runs on stale state for hours, a backpressure policy that drops events silently. Each is presented with the failure, the countermeasure, and the operational tradeoff.