Batch vs Stream Processing
Compare batch and stream processing on the same events. A small simulation shows event-time windows, a watermark closing a window, and a late event that the stream drops but the batch counts.
Batch processing computes over a complete, stored dataset and gives exact results later; stream processing computes continuously over events as they arrive and gives fast but sometimes approximate results. The detail interviewers expect: a stream uses a watermark to decide when an event-time window is final, and events arriving after that are dropped or handled separately.
Batch processing computes over a complete, stored dataset and gives exact results later. Stream processing computes continuously over events as they arrive and gives fast but sometimes approximate results. The difference shows up with late data: a stream must decide when a window is final, using a watermark, and events arriving after that point are dropped or handled separately.
- Batch processing runs over a bounded dataset and gives exact results hours later, while stream processing runs continuously over unbounded events and produces results within seconds.
- Event time is the timestamp on the event itself and processing time is when the system receives it; counting by event time gives correct results despite network delay and reordering.
- A watermark is a stream processor's estimate of how far event time has progressed; once it passes the end of a window, the window's result is emitted and the window is closed.
- In a simulation of one-minute windows, a late event from second 59 arrived after the watermark passed 60, so the stream counted 2 events in window 0 where the batch job counted 3.
- Late events in stream processing can be kept with an allowed-lateness grace period, sent to a side output, or corrected by a batch job that recomputes exact results.
- The lambda architecture runs batch and stream layers side by side, while the kappa architecture uses only streaming and replays the event log to recompute.
01Two ways to process the same data
Batch processing runs a job over a bounded dataset, such as yesterday's logs. It is simple, exact, easy to re-run, and results arrive hours later. Stream processing runs continuously over an unbounded flow of events, such as clicks or sensor readings, and produces results within seconds.
Choose by how fresh the answer needs to be. A monthly report is a batch problem. Fraud detection on a card swipe is a streaming problem. Many systems run both: a stream for fast approximate numbers and a batch job that later recomputes the exact figures.
02Event time versus processing time
Event time is the timestamp on the event itself, the moment the user clicked. Processing time is when your system receives it. Networks delay and reorder messages, so an event from second 59 can arrive after one from second 61. If you count by arrival time, results depend on network luck. Counting by event time gives the right answer, but forces the question: how long do you wait for stragglers?
03A tiny simulation: batch
Each event is (seconds since start, user). The last one, at second 59, arrives late, after events from later minutes.
from collections import defaultdict
events = [(1, 'a'), (2, 'b'), (61, 'a'), (62, 'a'), (125, 'b'), (59, 'a')]
def window(t):
return t // 60
batch = defaultdict(int)
for t, u in events:
batch[window(t)] += 1
print(dict(sorted(batch.items()))) # {0: 3, 1: 2, 2: 1}The batch job sees all the data, so window 0 correctly has 3 events (seconds 1, 2 and 59), window 1 has 2 and window 2 has 1.
04The same events as a stream
A watermark is the system's estimate of how far event time has progressed. Here we use the simplest rule: the watermark is the largest event time seen so far. Once the watermark passes the end of a window, we emit that window's result and close it.
stream = defaultdict(int)
watermark = -1
emitted = {}
for t, u in events:
w = window(t)
if w in emitted:
print('late dropped', t)
continue
stream[w] += 1
watermark = max(watermark, t)
for ww in sorted(stream):
if (ww + 1) * 60 <= watermark and ww not in emitted:
emitted[ww] = stream[ww]
print('emit window', ww, stream[ww], 'at event', t)
print(emitted)The run prints:
emit window 0 2 at event 61
emit window 1 2 at event 125
late dropped 59
{0: 2, 1: 2}Window 0 was emitted with 2 events when the event at second 61 pushed the watermark past 60. When the event from second 59 finally arrived, window 0 was already closed, so it was dropped. The stream says 2 where the batch says 3. Window 2 has not been emitted, because the watermark has not passed it yet.
This is the core trade-off in one example: speed against completeness.
05Handling late data
There are three common responses. Allowed lateness: keep windows open for a grace period, say 30 seconds past the watermark, which delays results but catches more stragglers. Late-event side outputs: send the late events to a separate stream so they can be counted or audited. Reconciliation: let a nightly batch job recompute the exact numbers and overwrite the streaming approximations, which is the idea behind the lambda architecture. The kappa architecture instead uses only streaming and replays the event log when it needs to recompute.
A longer wait gives more accurate windows and slower answers. There is no setting that is best for everything.
06Window types
Tumbling windows are fixed and do not overlap, like the one-minute windows above. Sliding (hopping) windows overlap, such as a 5-minute window that moves every minute, which gives smooth rolling averages. Session windows group events by activity, closing after a gap of inactivity, so they fit user visits of varying length. Each event can fall into one window for tumbling, several for sliding, and a window whose length depends on the data for sessions.
07Common interview questions
Q: Batch or stream?
A: Batch when results can wait and exactness matters. Stream when decisions are time-sensitive. Often both.
Q: What is a watermark?
A: A marker of how far event time is believed to have progressed. It tells the system when a window can be closed.
Q: What happens to late events?
A: They can be dropped, allowed within a lateness window, sent to a side output, or fixed by a later batch recomputation.
Q: Event time or processing time?
A: Event time for correct results despite delays and reordering. Processing time is simpler but varies with system load.
Q: Lambda versus kappa architecture?
A: Lambda runs batch and stream layers side by side. Kappa uses only a stream and replays the log to recompute.
Q: Name some stream tools.
A: Apache Kafka for the event log, and Flink, Spark Structured Streaming or Kafka Streams for processing.