Data Pipelines
Batch and streaming, distributed processing with Spark, and the idempotency that makes a rerun safe.
Moving Data Without Losing It
The old shape was ETL — extract, transform, then load the finished result — because storage and compute were expensive and the warehouse could not do the work. Cheap columnar warehouses inverted it into ELT: load the raw data first, transform it in place, and keep the raw copy so a mistake in the transform is recoverable without going back to the source.
Batch Processing
Batch runs on a schedule over a bounded window of data. It is simpler, cheaper and easier to reason about than streaming, and it is the right default for anything that does not genuinely need sub-minute freshness.
| Batch | Streaming | |
|---|---|---|
| Latency | Minutes to hours | Seconds |
| Reruns | Straightforward | Hard — state and ordering |
| Cost | Lower, bursty | Always-on infrastructure |
| Debugging | Inspect the window | Reproduce a moving target |
Process by partition, usually a date. Partitioned inputs and outputs are what make a rerun mean "recompute this day" rather than "recompute everything", and they are what let a backfill run in parallel.
Spark & Distributed Processing
Once the data does not fit on one machine, the work is split across a cluster: a driver plans, executors compute over partitions, and results are combined. Spark is the standard engine for that, and its performance story is almost entirely about how much data has to move between machines.
orders = spark.read.parquet("s3://lake/raw/orders/dt=2026-08-08")
daily = (orders
.filter(orders.status == "settled") # filter early: less to shuffle
.groupBy("store_id") # a shuffle happens here
.agg(F.sum("net_amount").alias("revenue")))
daily.write.mode("overwrite").parquet("s3://lake/marts/daily/dt=2026-08-08")| Concept | Why it decides the runtime |
|---|---|
| Shuffle | Moving data across the network — the dominant cost |
| Partitions | Too few starves the cluster, too many drowns it in overhead |
| Skew | One enormous key makes one task run for an hour alone |
| Lazy evaluation | Nothing runs until an action, so the plan can be optimised |
| Broadcast join | Sending a small table everywhere beats shuffling a huge one |
Idempotency & Backfills
A pipeline will be rerun: after a failure, after a bug fix, after a schema change, and every time someone backfills a year of history. Idempotent means running it twice leaves the same result as running it once — and it is the property that decides whether reruns are routine or terrifying.
| Pattern | Effect |
|---|---|
| Overwrite a partition | The rerun replaces exactly what it owns |
| Merge on a natural key | Updates rather than duplicates |
| Blind append | Doubles the data on every rerun — the classic bug |
| Deterministic transforms | No now(), no random ids inside the logic |
Late-arriving data is the other half. Events turn up hours or days after the timestamp they carry, so a partition that was correct yesterday is incomplete today. Decide a lateness window, reprocess recent partitions on a rolling basis, and make sure that reprocessing is idempotent — which is why the two topics belong together.
Interview Questions
ETL or ELT?
ELT is the modern default: load raw first, transform in the warehouse, and keep the raw copy so a bad transform can be fixed without re-extracting. ETL made sense when compute was scarce and the warehouse could not do the work.
When is streaming actually justified?
When a decision changes with sub-minute data — fraud checks, live operations. Otherwise batch delivers the same product with far less operational cost and much easier reruns.
What makes a pipeline idempotent?
Rerunning produces the same result: overwrite the partition you own or merge on a key rather than appending blindly, and keep transforms deterministic with no wall-clock reads.
Why is a shuffle the thing to watch in Spark?
It moves data across the network between executors. Filtering and aggregating before the shuffle, or broadcasting a small table instead, is where most of the runtime is won.
What is data skew?
One key holding a disproportionate share of rows, so a single task runs long after the rest finish. It is usually addressed by salting the key or handling the hot key separately.
How do you handle late-arriving data?
Choose a lateness window and reprocess recent partitions on a rolling schedule. That only works if the pipeline is idempotent, which is why the two are designed together.