Senior Data Engineer | Role Archetype

Leading data platform architecture decisions, shaping data mesh and lakehouse patterns, driving technical direction across teams, and providing technical leadership without requiring a management title.

📚

Growing into this role

Intermediate Data Engineer to Senior Data Engineer

→

🚀

Next step pathway

Senior Data Engineer to Data Architect

→

☍

Performance Calibration

Senior Data Engineer - Performance Levels

↗

☑

Behaviour Tracker

Senior Data Engineer - Performance Tracker

↗

Overview

As a Senior Data Engineer, you are a technical authority for data engineering within the organisation. You make and shape significant architecture decisions, drive the evolution of the data platform, and set technical direction that influences engineers across multiple teams. Your impact extends well beyond your own delivery.

You are expected to lead the resolution of the hardest data engineering problems, mentor and grow the team's capability, represent data engineering credibly in cross-functional conversations, and ensure the platform evolves in a coherent direction. You operate with high autonomy and are accountable for outcomes, not just outputs.

Key Responsibilities

Data Platform Architecture

Lead the design and evolution of the data platform architecture - lakehouse patterns, data mesh domains, ingestion frameworks, and serving layers.
Make and document significant architectural decisions, presenting trade-offs clearly and building consensus across stakeholders.
Establish and maintain platform standards - data modelling conventions, pipeline patterns, observability requirements - that other engineers apply consistently.
Evaluate and recommend tools and technologies, making informed recommendations based on the organisation's scale, capability, and strategy.
Identify structural problems in the existing platform and lead multi-sprint improvement programmes to address them.

Technical Leadership

Set the technical direction for data engineering within the team and influence it across adjacent teams.
Lead complex technical deliveries - coordinating work across multiple engineers, managing dependencies, and ensuring coherent architecture.
Drive data engineering best practices into the organisation - testing standards, data quality approaches, observability patterns.
Represent data engineering in architecture forums, technical strategy discussions, and cross-team decision-making bodies.
Lead post-incident reviews for significant data outages - driving root cause analysis, remediation, and systemic improvement.

Mentoring and Team Development

Provide senior mentoring to intermediate and junior engineers - shaping their technical development, not just reviewing their code.
Identify and grow the next generation of technical leaders within the data engineering team.
Create learning environments - documentation, talks, pairing sessions - that raise the floor of the entire team.
Contribute to hiring - reviewing technical assessments, conducting interviews, and helping define what good looks like for the discipline.

Cross-Team Influence and Stakeholder Engagement

Build credibility and influence with data analysts, platform engineers, product managers, and senior business stakeholders.
Translate complex data engineering concepts into clear business value narratives for non-technical audiences.
Drive data governance conversations - lineage, cataloguing, ownership, and quality - across organisational boundaries.
Collaborate with the Data Architect on long-term technical strategy and ensure the team's roadmap aligns with it.

Role Specific

Data Mesh and Lakehouse Architecture

Lead the design and adoption of modern data platform patterns - data mesh domain ownership, lakehouse architectures, or equivalent - ensuring the organisation's data platform can scale with the business.

Cross-Team Technical Influence

Shape data engineering standards, tooling decisions, and architectural patterns across multiple teams - driving coherence and quality at scale without relying on authority.

Platform Reliability and Observability

Establish the observability, monitoring, and reliability practices that ensure the data platform operates with predictable quality - including data freshness, completeness, and correctness at scale.

Behaviours

Learning & Growth

Actively monitors developments in data engineering - emerging architecture patterns, new tooling, industry case studies - and evaluates their relevance to the organisation's context.
Invests in depth across the data stack - not just pipeline development but storage optimisation, compute patterns, and data governance frameworks.
Seeks out peer review and challenge from other senior technical practitioners, not just validation.
Identifies areas where the team's collective capability needs to grow and creates structured opportunities to develop it.
Brings external perspectives into the team - from conference talks, industry publications, and peer network conversations.
Reflects critically on past architectural decisions - capturing what was learned and ensuring those lessons shape future choices.

Delivery

Leads the delivery of large, complex data platform work - coordinating across engineers, managing dependencies, and ensuring coherent technical execution.
Maintains personal delivery velocity on significant technical work alongside leadership and mentoring responsibilities.
Drives delivery rhythm on platform improvement programmes - breaking ambiguous work into deliverable increments and maintaining momentum.
Identifies and removes delivery blockers for the broader team - dependency management, decision facilitation, technical unblocking.
Manages the tension between long-term platform investment and short-term delivery pressure - making reasoned trade-offs transparently.
Ensures large deliveries land fully - monitoring, documentation, knowledge transfer, and team readiness - not just code in production.

Quality & Craft

Sets the quality standard for the discipline - their own work is a reference implementation of what good data engineering looks like.
Leads the design of data quality frameworks that provide genuine confidence in data reliability across the platform.
Identifies systemic quality problems - fragile ingestion patterns, untested transformation logic, missing data contracts - and drives structured remediation.
Reviews architectural proposals critically - identifying failure modes, scalability limits, and operational risks before they are built in.
Champions data modelling rigour - ensuring the organisation's data models are designed for long-term usefulness, not just immediate delivery.
Establishes engineering standards that others apply consistently - pipeline patterns, testing requirements, documentation standards.

Communication

Communicates architectural decisions with clarity and rigour - presenting options, trade-offs, and recommendations in formats accessible to both technical and non-technical audiences.
Writes architecture decision records (ADRs) and technical documentation that stand as durable records of intent and reasoning.
Influences without authority - building persuasive cases for technical direction through evidence, credibility, and relationship.
Facilitates technical discussions productively - drawing out perspectives, resolving disagreement, and reaching clear decisions.
Communicates data engineering risk clearly to senior leadership - translating technical concerns into business impact language.
Represents the data engineering discipline in cross-functional forums with confidence and depth.

Collaboration

Builds strong, trust-based relationships with senior stakeholders - data product owners, heads of analytics, platform leads - enabling effective cross-boundary collaboration.
Works with the Data Architect to align team delivery with long-term technical strategy.
Partners with platform engineers to ensure the data platform is built on reliable, maintainable infrastructure.
Drives collaboration across data domain teams - establishing shared standards, shared tooling, and mutual accountability for data quality.
Creates a collaborative culture within the data engineering team - psychological safety, knowledge sharing, and open technical debate.
Represents data engineering interests in product and delivery planning, ensuring data work is appropriately sequenced and resourced.

Ownership

Takes accountability for the health and direction of the data platform - not just their own work but the coherence and reliability of the whole.
Leads incident response for significant data platform failures - investigating, communicating, resolving, and preventing recurrence.
Owns the quality of the team's output collectively - raising concerns about standards, practices, and technical direction when needed.
Makes bold technical recommendations when the evidence supports them - not defaulting to safe, familiar choices when better options exist.
Takes responsibility for the technical environment junior and intermediate engineers work in - setting them up for success through standards, tooling, and support.
Holds themselves and others accountable for the long-term health of the data platform, not just short-term delivery.

Technical Foundation

Demonstrates mastery-level SQL and Python capability and applies it in the design of high-quality, production-grade platform components.
Designs and validates data architectures at scale - understanding the failure modes, operational costs, and evolution paths of significant architectural choices.
Operates at the intersection of data engineering and data platform engineering - designing systems that are observable, reliable, and maintainable at scale.
Deep understanding of cloud cost optimisation, query performance at scale, and storage economics in cloud data platforms.
Maintains current knowledge of the data tooling landscape - understanding the capabilities and trade-offs of leading tools in orchestration, transformation, cataloguing, and quality.
Understands data governance - lineage, cataloguing, access controls, and regulatory considerations - and incorporates governance requirements into architectural decisions.
Contributes to the data engineering discipline beyond the team - writing, speaking, or open source contribution that builds the organisation's external reputation.

Skills

Deep expertise in cloud data platform architecture - BigQuery, Snowflake, Databricks, or equivalent - at production scale.

Advanced data modelling - dimensional, data vault, one big table, and lakehouse patterns - with ability to reason about trade-offs.

Strong understanding of data mesh principles - domain ownership, data products, self-serve infrastructure, and federated governance.

Experience with streaming data architectures - Kafka, Pub/Sub, Kinesis, or equivalent - and real-time pipeline patterns.

Proficiency in infrastructure and platform tooling relevant to data - dbt at scale, orchestration at enterprise scale, data cataloguing.

Ability to evaluate and select tools, making recommendations grounded in evidence and organisational context.

Experience designing and operating data observability - data contracts, SLAs, anomaly detection, and alerting.

Strong communication and influence skills - able to build consensus across technical and non-technical stakeholders.

AI AI & Automation Expectations Updated for the AI-augmented era

AI Augmented Delivery

Defines the team's approach to AI-augmented data engineering - establishing guidelines for when and how to use AI tools in pipeline development, code review, and incident response.
Uses AI to accelerate architectural exploration - generating comparison matrices for technology options, producing draft ADRs, and stress-testing architectural assumptions - while applying expert judgement to conclusions.
Applies AI to data observability and quality - using AI-powered anomaly detection tools and automated data profiling as part of the data platform's quality layer.
Uses AI for pipeline generation at scale - generating dbt model stubs from source schemas, producing ingestion configurations from API specs - then applies engineering rigour to validate and refine outputs.
Coaches the team on responsible AI use in data engineering - distinguishing high-value use cases (boilerplate generation, query optimisation exploration) from high-risk ones (business logic generation without domain expert review).
Evaluates and recommends AI-powered data tooling - data catalogues with AI-generated documentation, intelligent data quality tools - as part of the platform tooling strategy.