Real project mental model: Think of data engineering as plumbing. Sources are faucets. Pipelines are pipes. Warehouses & lakes are tanks. Dashboards & ML models are the taps people use. Your job: keep the water flowing clean, fast and without leaks.
// tools & technology stack
// end-to-end etl pipeline flow
← Click a stage to explore
Each stage in a real data pipeline has specific tools, responsibilities, and failure modes. Click any stage above to see what happens inside it, what tools are commonly used, and a code snippet you'd actually write.
// architecture patterns
When to use Lambda: When you need both historical accuracy (reprocessing) AND real-time updates. The downside: you maintain two separate codebases. Used heavily by LinkedIn, Netflix in early years. Modern teams often move to Kappa or Lakehouse instead.
When to use Kappa: Single codebase — everything is a stream. Historical reprocessing = replay Kafka from offset. Simpler to operate than Lambda. Used by Uber, Twitter. Best when your latency requirement is always near-real-time and you want one mental model.
When to use Lakehouse: Merges Data Lake (cheap raw storage) + Data Warehouse (SQL + ACID). One copy of data, many access patterns. Databricks pioneered Delta Lake, Snowflake supports Iceberg. Used by most modern orgs — Airbnb, Stripe, Grab.
When to use Data Mesh: Large org, many domain teams, central data team becomes a bottleneck. Domains treat data as a product they own and SLA. Hard to implement — needs organisational change. Zalando, JPMorgan have public case studies. Requires a strong self-serve platform underneath.