Warehouses & Lakes
Columnar storage, Parquet, partitioning, and the lakehouse pattern the field has converged on.
Why Analytics Needs Different Storage
Transactional systems answer questions about one row: fetch this order, update that balance. Analytical systems answer questions about one column across millions of rows: sum revenue by month.
That difference is why the two diverged. A row store must read every column of every row to sum one of them. A column store reads only the column you asked for, which is often a hundredth of the bytes.
| OLTP — the application database | OLAP — the warehouse | |
|---|---|---|
| Question | One row, right now | One column, across everything |
| Layout | Row-oriented | Column-oriented |
| Writes | Constant, small | Batched, large |
| Normalised? | Heavily | Deliberately not |
| Optimised for | Latency | Throughput |
Parquet & Columnar Formats
Columnar files are the reason modern analytics is affordable. Storing a column together means its values are similar, and similar values compress extremely well.
| Format | Layout | Use it for |
|---|---|---|
| CSV | Row, untyped | Interchange only — no schema, no compression |
| JSON | Row, nested | APIs and raw landing zones |
| Parquet | Columnar | The default for analytics at rest |
| Avro | Row, schema-carrying | Streaming and message payloads |
| ORC | Columnar | Similar to Parquet, common in Hive estates |
Parquet also stores min/max statistics per chunk, so a query filtering on a date can skip entire files without opening them — predicate pushdown. Combined with partitioning the data by that same date, a query over one day of a five-year table touches a fraction of a percent of it.
Lakehouse Architecture
Warehouses gave structure and transactions but were expensive and rigid. Data lakes gave cheap object storage and any format, and frequently became swamps with no schema, no transactions and no idea what was in them.
The lakehouse keeps files in cheap object storage and adds a metadata layer on top that provides schema, ACID transactions and time travel. Delta Lake, Apache Iceberg and Hudi are the three implementations you will meet.
| Warehouse | Lake | Lakehouse | |
|---|---|---|---|
| Storage cost | High | Low | Low |
| Schema | On write | On read | Enforced, evolvable |
| Transactions | Yes | No | Yes |
| Any file type | No | Yes | Yes |
Layers are conventionally named bronze, silver and gold: raw as it arrived, cleaned and conformed, then aggregated for consumption. The names vary; the discipline of never letting consumers read the raw layer does not.
Interview Questions
Why is columnar storage faster for analytics?
An analytical query reads a few columns across many rows. A column store reads only those columns, so it moves far fewer bytes, and values within a column are similar enough to compress heavily.
OLTP versus OLAP?
OLTP serves the application: many small reads and writes of individual rows, normalised, latency-sensitive. OLAP serves analysis: large scans over few columns, denormalised, throughput-sensitive.
What does a lakehouse add to a data lake?
A metadata layer giving schema enforcement, ACID transactions and time travel over files that still sit in cheap object storage. It is what stops a lake becoming a swamp.
What is the small files problem?
Thousands of tiny files make per-file overhead dominate scan time, so a query is slower than the same data in a few large files. Compaction jobs are routine maintenance.
Why is a warehouse deliberately denormalised?
Joins are expensive at scale and storage is cheap. Duplicating dimension attributes into wide tables trades disk for far fewer joins on every query.