// what does a data engineer do?
🔁
Build Pipelines
Design & automate data flows from source systems to destinations — reliably and at scale.
→ ETL Pipeline
🛠
Pick the Right Tools
Choose ingestion, processing, orchestration and storage layers based on volume and latency needs.
→ Tools & Stack
🏗
Design Architecture
Decide between Lambda, Kappa, Lakehouse or Data Mesh patterns based on use case.
→ Architecture
☁️
Deploy on Cloud
Leverage managed services on AWS, GCP, or Azure — storage, compute, streaming & orchestration.
→ Cloud & Services
📖
Know the Lingo
Data Lakehouse, CDC, SCD, idempotency, lineage — learn the terms that matter on real projects.
→ Glossary
🧱
Key Principles
Idempotency · Observability · Schema Evolution · Data Quality · Backfill Strategy
always apply
Real project mental model: Think of data engineering as plumbing. Sources are faucets. Pipelines are pipes. Warehouses & lakes are tanks. Dashboards & ML models are the taps people use. Your job: keep the water flowing clean, fast and without leaks.
// tools & technology stack
// end-to-end etl pipeline flow
← Click a stage to explore
Each stage in a real data pipeline has specific tools, responsibilities, and failure modes. Click any stage above to see what happens inside it, what tools are commonly used, and a code snippet you'd actually write.
// architecture patterns
Source Kafka / MQ batch stream Batch Layer Spark / Hive Speed Layer Flink / Spark SS Serving Layer Warehouse / API Lambda = Batch (accuracy) + Speed (low latency) merged at Serving
When to use Lambda: When you need both historical accuracy (reprocessing) AND real-time updates. The downside: you maintain two separate codebases. Used heavily by LinkedIn, Netflix in early years. Modern teams often move to Kappa or Lakehouse instead.
Source Kafka event log Stream Processor Flink / Spark SS Serving Druid / ES App / BI replay from offset for reprocessing
When to use Kappa: Single codebase — everything is a stream. Historical reprocessing = replay Kafka from offset. Simpler to operate than Lambda. Used by Uber, Twitter. Best when your latency requirement is always near-real-time and you want one mental model.
Open Storage — S3 / GCS / ADLS Parquet / Delta Lake / Iceberg / Hudi Metadata + Governance Unity Catalog / HMS / Glue Spark batch/stream Trino/Presto SQL query Data Sources — DB / APIs / Streams / Files BI Tools · ML Models · Dashboards · Data Science
When to use Lakehouse: Merges Data Lake (cheap raw storage) + Data Warehouse (SQL + ACID). One copy of data, many access patterns. Databricks pioneered Delta Lake, Snowflake supports Iceberg. Used by most modern orgs — Airbnb, Stripe, Grab.
Self-serve Data Platform Infra / Catalog / Pipelines as templates Orders domain team Marketing domain team Payments domain team Analytics domain team Each domain owns, produces & serves its own data as a product
When to use Data Mesh: Large org, many domain teams, central data team becomes a bottleneck. Domains treat data as a product they own and SLA. Hard to implement — needs organisational change. Zalando, JPMorgan have public case studies. Requires a strong self-serve platform underneath.
// cloud providers & managed services
// data engineering glossary