Senior Data Engineer

Technical Leadership:

• Lead data engineering workstreams from discovery through to design, build, test, deployment, and support transition.
• Provide technical guidance, mentoring, and code reviews for junior and mid-level data engineers.
• Translate high-level architecture into practical engineering designs and delivery tasks.
• Contribute to technical decision-making, estimation, planning, and risk management.
• Identify engineering risks, dependencies, blockers, and improvement opportunities early.
• Promote engineering standards, reusable patterns, documentation, and good development practices.

Stakeholder and Delivery Management:

• Work closely with product owners, business analysts, architects, testers, data analysts, and client stakeholders.
• Participate in agile ceremonies including sprint planning, daily stand-ups, backlog refinement, reviews, and retrospectives.
• Support discovery workshops, requirements analysis, technical design sessions, and show-and-tell demonstrations.
• Produce clear technical documentation, data flow diagrams, mapping specifications, deployment guides, and support documentation.
• Support transition into live service, including knowledge transfer, runbooks, monitoring, incident response, and handover to support teams.

Required Skills and Experience

Essential Technical Skills

• Strong experience as a Data Engineer or Senior Data Engineer in enterprise or cloud environments.
• Strong SQL skills, including query optimisation, stored procedures, data modelling, and performance tuning.
• Strong Python or PySpark experience for data processing, automation, and transformation logic.
• Experience building ETL/ELT pipelines using tools such as AWS Glue, Azure Data Factory, Databricks, Synapse, Fabric, dbt, Airflow, Informatica, Talend, or similar.
• Experience working with cloud data platforms, preferably Microsoft Azure.
• Experience with data lake, lakehouse, data warehouse, and analytical platform architectures.
• Good understanding of batch processing, incremental loads, CDC, API ingestion, and file-based ingestion patterns.
• Experience with data validation, reconciliation, error handling, and data quality controls.
• Experience using Git-based source control and CI/CD practices.
• Understanding of security, access control, encryption, data privacy, and environment management.

Essential Delivery Experience:

• Experience delivering production-grade data platforms or pipelines in complex organisations.
• Ability to work across the full delivery lifecycle from requirements and design through to build, test, release, and support.
• Experience working in agile delivery teams.
• Ability to produce clear technical documentation and explain technical concepts to non-technical stakeholders.
• Experience leading technical workstreams or mentoring other engineers.
• Strong analytical, problem-solving, and troubleshooting skills.
• Ability to work independently, manage priorities, and take ownership of outcomes.

Desirable Skills and Experience:

• Microsoft Azure certifications, such as Azure Data Engineer Associate or equivalent.
• Experience with AWS Glue, Microsoft Fabric, Azure Synapse Analytics, Azure Data Lake, Azure SQL, Azure Functions, Logic Apps, Event Hubs, or Azure Purview.
• Experience with Databricks, Delta Lake, Spark, Unity Catalog, MLflow, or lakehouse patterns.
• Experience with Snowflake, Redshift, BigQuery, or other cloud data warehouse platforms.
• Experience with dbt, data transformation frameworks, or analytics engineering practices.
• Experience with streaming technologies such as Kafka, Event Hubs, Kinesis, or Pub/Sub.
• Experience with Power BI semantic models, reporting datasets, or analytical consumption layers.
• Experience with data governance, data lineage, metadata management, master data management, or data cataloguing.
• Experience with Terraform, Bicep, ARM templates, Docker, Kubernetes, or other infrastructure and deployment tooling.

Behavioural Competencies:

• Strong ownership mindset with the ability to take accountability for technical delivery.
• Clear and confident communicator, able to engage with technical and business stakeholders.
• Pragmatic problem solver who balances engineering quality with delivery timelines.
• Collaborative team player who supports others and contributes to shared outcomes.
• Detail-oriented, with strong focus on data accuracy, quality, and operational reliability.
• Comfortable working in fast-paced, multi-disciplinary, and multi-supplier environments.
• Able to challenge constructively and recommend practical improvements.
• Committed to continuous learning and keeping up to date with modern data engineering practices.

Typical Deliverables:

• Data pipeline designs and implemented ETL/ELT workflows.
• Data ingestion, transformation, validation, and publishing components.
• Data models, schemas, mapping documents, and transformation specifications.
• Automated deployment pipelines and environment configuration.
• Data quality checks, reconciliation reports, and exception handling processes.
• Technical design documentation and data flow diagrams.
• Runbooks, operational guides, and support handover documentation.
• Performance optimisation recommendations and implemented improvements.
• Knowledge transfer sessions and mentoring for internal teams.

Qualifications:

• Degree in Computer Science, Data Engineering, Software Engineering, Information Systems, Mathematics, Statistics, or a related discipline, or equivalent professional experience.
• Relevant cloud or data engineering certifications are desirable but not mandatory.

Experience Level:

• 5+ years of experience in data engineering, software engineering, or data platform delivery.
• At least 2+ years of hands-on experience delivering cloud-based data engineering solutions.
• Prior experience in a senior, lead, or workstream ownership role is preferred.