Stott and May
Lead Data Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Lead Data Engineer
Location: UK – Fully Remote
Contract: 6-month initial contract
Rate: Competitive day rate (Inside IR35)
We are looking for an experienced Lead Data Engineer to join a leading insurance client, providing technical leadership across data engineering initiatives and supporting the development of a high-performing engineering team.
The role is both hands-on and leadership-focused, with Databricks at the centre of the technology stack alongside Python, PySpark, and SQL. You will take ownership of the Databricks platform and architecture while working closely with senior stakeholders across Risk, Quantitative Finance, Compliance, and Data Science.
Key Responsibilities
- Own the Databricks architecture, strategy, and roadmap, ensuring performance, availability, and cost efficiency.
- Design, build, and operationalize scalable ETL/ELT data pipelines using Databricks, Python, and SQL.
- Develop and optimize solutions using Databricks Lakehouse and Delta Lake.
- Implement data governance, lineage, controls, and audit frameworks within a regulated environment.
- Lead, mentor, and upskill a team of Data Engineers, establishing technical standards and conducting code reviews.
- Provide technical direction across data engineering projects and drive engineering best practices.
- Work closely with Risk, Quantitative Finance, Compliance, and Data Science teams to translate business requirements into technical solutions.
- Communicate complex technical concepts and architecture clearly to senior and executive stakeholders.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Required Skills & Experience
Get help with your application
Your very own career expert that helps elevate your application to the next level.
- 7+ years' experience in Data Engineering.
- 3+ years' hands-on experience delivering large-scale Databricks implementations within financial services.
- Strong experience within Insurance, Banking, Asset Management, or FinTech.
- Advanced Python, PySpark, and SQL skills.
- Deep knowledge of Databricks Lakehouse architecture and Delta Lake.
- Experience with at least one major cloud platform: AWS, Azure, or GCP.
- Proven experience leading or mentoring Data Engineering teams.
- Strong stakeholder management and communication skills.
- Experience working within regulated environments, with an understanding of data governance and compliance requirements.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Location