Data Engineer

Whitehall Resources are looking for a Data Engineer. This role is hybrid working with 2 days per week onsite in Cambridge, and the remainder remote working on an initial 6 month contract.

Inside IR35

Job Overview:

We are seeking an experienced Data Engineer to design and deliver scalable, high-quality data solutions in Databricks. You will play a key role in evolving our enterprise data platform, creating trusted, AI-ready data products that power reporting, analytics, automation and AI across Enterprise IT. You will design and build modern data pipelines integrating enterprise platforms such as AI platforms including Microsoft Copilot, OpenAI and other enterprise AI services. The role will also contribute to our growing AI capabilities, engineering solutions for AI usage and adoption data, Retrieval-Augmented Generation (RAG), LLM integration and AI orchestration.

Responsibilities:

• Design, build and maintain scalable ETL/ELT pipelines using Databricks, PySpark, SQL and Python.
• Design scalable data models using Data Vault principles.
• Build trusted data products using Lakehouse and Medallion Architecture principles.
• Integrate data from enterprise systems including ServiceNow, Jira, Azure DevOps and AI platforms such as Copilot and OpenAI.
• Build data pipelines for AI usage, adoption, telemetry and performance analytics.
• Design and implement RAG pipelines, including ingestion, embeddings, vector/semantic search and retrieval.
• Integrate LLMs, APIs and enterprise data to support AI applications and agents.
• Build and maintain AI orchestration workflows connecting models, data, tools and business processes.
• Implement automated data quality, testing, monitoring and validation.
• Optimise solutions for performance, scalability, reliability and cost.
• Contribute to data governance, metadata, lineage, security and documentation.
• Use Git and CI/CD to support automated and reliable deployment.