Skills to Develop

• Write complex SQL queries using window functions, CTEs, and subqueries to answer analytical questions on real datasets.
• Build a Python-based data pipeline with proper error handling, logging, retry logic, and idempotent design.
• Implement basic data quality checks in a pipeline - null rates, row counts, value range validation - and fail the pipeline clearly when checks do not pass.
• Use a workflow orchestrator such as Airflow or Prefect to schedule and monitor a pipeline, understanding DAG structure and dependency management.
• Load data into a cloud data warehouse, apply appropriate data types and partitioning, and query it efficiently.
• Write tests for pipeline logic using Python unit tests or a framework such as dbt test.
• Version control pipeline code in Git and participate in code review for data engineering work.
• Document a pipeline with enough context that a colleague could understand its purpose, inputs, outputs, and failure modes without asking you.

Behaviours to Demonstrate

• Asks clarifying questions about data sources and business requirements before writing code, rather than assuming and discovering problems later.
• Validates data at every stage of a pipeline and makes failures visible rather than silently passing bad data downstream.
• Flags data quality anomalies immediately to stakeholders rather than waiting until they surface in a dashboard.
• Maintains pipeline documentation and updates it when behaviour changes.
• Runs pipelines in a development environment before deploying to production and verifies outputs against expectations.
• Seeks feedback on pipeline design before investing significant effort in implementation.
• Tracks down the source of a data issue systematically, documenting the investigation process.