Deloitte
Manager, Senior Databricks Data & AI Engineer/Technical Lead/Databricks Architect (Solution Engineering | AI/ML), Engineering, AI & Data, Technology & Transformation
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Connect to your Industry
Do you want to lead the engineering delivery and shape the solution architect of some of the UK's largest and most ambitious Data & AI programmes?
Within Technology & Transformation organisations are increasingly relying on modern data platforms and AI to accelerate innovation, improve decision-making, and unlock new business value. As a Senior Databricks Data & AI Engineer / Technical Lead, you'll provide the technical leadership that enables these transformations—leading engineering teams, shaping solution architecture, and delivering enterprise-scale data, machine learning, and AI solutions on the Databricks Data Intelligence Platform.
Working alongside some of the industry's leading architects, engineers, and AI specialists, you'll help clients build secure, scalable, and high-performing data and AI platforms that power the next generation of intelligent applications.
Connect to your career at Deloitte
Deloitte drives progress. Using our vast range of expertise, we help our clients' become leaders wherever they choose to compete. To do this, we invest in outstanding people. We build teams of future thinkers, with diverse talents and backgrounds, and empower them all to reach for and achieve more.
What brings us all together at Deloitte? It’s how we approach the thousands of decisions we make every day. How we behave, our beliefs and our attitudes. In other words: our values. Whatever we do, wherever we are in the world, we lead the way, serve with integrity, take care of each other, foster inclusion, and collaborate for measurable impact. These five shared values lead every decision we make and action we take, guiding us to deliver impact how and where it matters most.
Connect to your opportunity
We're expanding our Databricks Engineering practice and are looking for experienced technical leaders who combine deep engineering expertise, solution architecture capability with strong delivery leadership. You'll work alongside exceptional data, AI, and cloud specialists to design, build, and deliver complex enterprise solutions while leading distributed engineering teams and shaping technical direction across multiple workstreams.
Depending on your strengths and client needs, you may focus more on hands-on engineering leadership and delivery, end-to-end solution and platform architecture, or a combination of both across enterprise Databricks transformation programmes. The ideal candidate will possess extensive hands-on expertise across the Databricks Lakehouse Platform, modern data engineering, machine learning, and Generative AI, while providing technical leadership, design authority, and engineering governance throughout the delivery lifecycle.
As a key member of our team, you will:
- Lead the technical delivery and/or solution architecture of enterprise-scale Databricks Data & AI implementations from solution design through to production deployment.
- Act as the technical lead and design authority across multiple workstreams, ensuring solution quality, scalability, security, and alignment with engineering best practices.
- Lead and mentor onshore and offshore engineering teams, fostering technical excellence, collaboration, and continuous improvement.
- Drive engineering delivery by establishing technical standards, reviewing solution designs, managing technical risks, and ensuring successful execution against project objectives.
- Conduct code reviews and engineering quality assurance, promoting clean, maintainable, high-quality code and adherence to software engineering best practices.
- Optimise platform performance across Spark workloads, Delta Lake, data pipelines, storage, and compute to maximise scalability, reliability, and cost efficiency.
- Design and deliver enterprise-grade Machine Learning, Generative AI and Agentic AI solutions leveraging the Databricks Data Intelligence Platform, MLflow, Mosaic AI, Vector Search, and modern LLM capabilities where appropriate.
- Partner with solution architects, product owners, client stakeholders, and engineering teams to translate business requirements into scalable technical solutions.
- Coach and develop engineers through technical mentoring, knowledge sharing, and establishing reusable engineering frameworks and best practices.
- Own technical governance by maintaining engineering standards, architectural integrity, documentation, and design decisions throughout project delivery.
- Define enterprise solution and platform architectures on Databricks — producing reference architectures, blueprints, and target-state designs that span data engineering, ML, and Generative AI.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Connect to your skills and professional experience
Overall, you're an inspiring technical leader, an experienced engineer, and a delivery-focused problem solver. You understand that successful programmes are built on strong technical leadership, engineering excellence, and collaborative teams.
You're passionate about mentoring others, making sound architectural decisions, and delivering innovative Data & AI solutions that create measurable business value.
You'll have strong technical skills including some or all of the following:
- Extensive experience delivering enterprise-scale Data Engineering and AI solutions using the Databricks Data Intelligence Platform.
- Proven experience as a solution or platform architect, defining reference architectures, integration patterns, and target-state designs across enterprise Databricks environments.
- Deep expertise across Databricks including Delta Lake, Unity Catalog, Spark, PySpark, Databricks SQL, Databricks Workflows, Lakeflow, Lakehouse architecture, MLflow, AI/BI Genie, Mosaic AI, and Agent Bricks.
- Proven experience leading engineering teams across both onshore and offshore delivery models.
- Strong experience establishing engineering standards, technical governance, and design authority across large programmes.
- Expertise in software engineering best practices including Git, CI/CD, automated testing, Infrastructure as Code, and peer code reviews.
- Strong experience designing, implementing and optimising large-scale batch and streaming data pipelines.
- Advanced Spark performance tuning and optimisation experience across storage, compute, and workload execution.
- Experience delivering production Machine Learning solutions using MLflow, feature engineering, model lifecycle management, and MLOps principles.
- Experience designing and delivering enterprise AI solutions, with deep expertise in Agentic AI architectures, AI Agents, Retrieval-Augmented Generation (RAG), Vector Search, LLM orchestration, prompt engineering, AI governance and responsible AI practices.
- Excellent communication and stakeholder management skills, with the ability to influence technical and non-technical audiences alike.
- Experience working across Azure, AWS, or GCP cloud ecosystems.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Essential:
- Proven experience leading enterprise-scale Databricks implementation programmes.
- Extensive experience acting as a Technical Lead, Engineering Lead, Solution/Platform Architect, or Technical Design Authority, with responsibility for architecture, solution governance, and technical decision making.
- Demonstrable experience leading offshore engineering teams.
- Strong hands-on expertise in Python, PySpark, and SQL.
- Experience designing & delivering enterprise Generative AI & Agentic AI solutions including AI Agents, RAG, Vector Search, LLM Orchestration and AI governance.
- Experience designing & delivering production-grade Machine Learning solutions.
- Proven experience conducting solution design reviews, architecture governance, and code reviews.
- Strong understanding of modern software engineering and DevOps practices.
- Experience working within Agile delivery methodologies.
- Deep expertise in the Databricks Data Intelligence Platform, including Apache Spark, PySpark, Delta Lake, Unity Catalog, Databricks SQL, Lakeflow, Mosaic AI,AI/BI Genie and Agent Bricks.
- Strong experience designing and delivering production-grade batch and streaming data pipelines using modern Databricks engineering patterns.
- Advanced knowledge of Spark performance tuning, workload optimisation and Databricks platform cost optimisation.
- Strong understanding of Databricks governance, security, enterprise platform architecture, and multi-workspace operating models including Unity Catalog.
- Experience implementing CI/CD, automated testing and deployment practices for Databricks, including Infrastructure as Code and Databricks Asset Bundles.
- Experience enabling ML and Generative AI workloads using MLflow, Mosaic AI and associated Databricks capabilities.
Preferred:
- Databricks Machine Learning Engineer or Generative AI certifications. Databricks Certified Data Engineer Professional (or equivalent advanced Databricks certification).
- Experience with Azure DevOps, GitHub Actions, or equivalent CI/CD tooling.
- Experience designing and delivering scalable, production-grade batch and streaming data pipelines on Databricks, including orchestration and integration with enterprise data sources.
- Experience optimising and operationalising data pipelines for performance, reliability and cost, including CI/CD, automated testing and monitoring.
- Experience with orchestration technologies such as Azure Data Factory, Airflow, or Databricks Workflows.
- Experience with Docker, Kubernetes, and containerised deployments.
- Consulting experience delivering complex Data & AI engagements across multiple industries.
For a full job description, please visit our online Deloitte Careers portal.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location