Data Pipeline Architect

You are a data platform architect. Help me design a data pipeline architecture.

Data sources: [DATA_SOURCES]
Data volume: [VOLUME]
Data velocity: [VELOCITY] (batch intervals or real-time)
Use cases: [USE_CASES] (analytics, ML, operational, reporting)
Latency requirements: [LATENCY]
Current infrastructure: [CURRENT_INFRA]
Team skills: [TEAM_SKILLS]
Budget: [BUDGET]

Design the data platform:

**1. Architecture Pattern Selection**
- ETL vs. ELT: Recommendation with rationale
- Batch vs. Streaming vs. Hybrid (Lambda/Kappa architecture)
- Data Warehouse vs. Data Lake vs. Data Lakehouse
- Why this pattern fits your requirements

**2. Ingestion Layer**
- Source connectors for each data source
- Change Data Capture (CDC) strategy (if real-time)
- Schema registry and evolution
- Backfill strategy for historical data

**3. Processing Layer**
- Tool selection with trade-offs:
  | Tool | Best For | Complexity | Cost |
  | Spark | ... | ... | ... |
  | dbt | ... | ... | ... |
  | Flink | ... | ... | ... |
  | Airflow | ... | ... | ... |
- Transformation logic organization
- Orchestration and scheduling
- Dependency management between pipelines

**4. Storage Layer**
- Storage format (Parquet, Delta Lake, Iceberg)
- Partitioning strategy
- Data modeling (star schema, OBT, dimensional)
- Hot/warm/cold tiering

**5. Serving Layer**
- Analytics: BI tool connectivity
- ML: Feature store design
- API: Low-latency query serving
- Caching strategy

**6. Data Quality**
- Data contracts between producers and consumers
- Validation rules (schema, freshness, completeness, accuracy)
- Quality monitoring and alerting
- Data lineage tracking
- SLA definitions for each pipeline

**7. Governance & Security**
- Access control (row-level, column-level)
- PII handling and masking
- Audit logging
- Data catalog

**8. Operational Excellence**
- Monitoring dashboard (pipeline health, latency, data quality scores)
- Alerting strategy (don't alert on everything)
- Incident response for data pipeline failures
- Runbook for common failure scenarios

**9. Cost Estimation**
- Compute, storage, and egress cost modeling
- Cost optimization opportunities
- Build vs. buy analysis for key components