Pioneer Search
Lead Data Engineer
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Lead Data Engineer
Employment Type: Permanent
Location: London - Hybrid (3 days in the office)
Salary: £95,000 - £110,000 + benefits & bonus
About the Client
Our client is a leading specialist insurance provider, recognised for its innovative approach to technology and data. Operating globally, they are investing heavily in their modern data platform, leveraging cloud technologies to deliver scalable, high-quality data solutions that support underwriting, analytics and business decision-making.
Role Overview
As a Lead Data Engineer, you'll play a key role in designing and evolving a modern cloud-based data platform built on Databricks and Azure. You'll lead the development of scalable data pipelines using Python and PySpark, enabling high-quality data to power underwriting, analytics and business-critical decision-making across the organisation.
Working closely with Product Owners, Architects and business stakeholders, you'll provide technical leadership throughout the full data lifecycle. Alongside remaining hands-on, you'll mentor a team of Data Engineers, champion engineering best practices and help shape the future of the organisation's cloud data platform.
Key Responsibilities:
- Lead and mentor a team of Data Engineers, supporting the successful delivery of data engineering projects from design through to production.
- Design, develop and optimise scalable data pipelines using Python, PySpark and Databricks.
- Build and maintain cloud-native data solutions across the Azure ecosystem, including Azure Functions and related services.
- Collaborate with Product Owners, Architects and business stakeholders to define technical solutions and deliver against business priorities.
- Drive engineering best practices through code reviews, mentoring and continuous improvement initiatives.
- Identify opportunities to improve platform performance, data quality, scalability and operational efficiency.
- Champion automation, testing, CI/CD and modern engineering practices across the data engineering function.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Key Requirements:
- Strong commercial experience building data engineering solutions using Python, PySpark and Databricks.
- Proven experience designing and optimising complex SQL queries and working with large-scale datasets.
- Strong experience across the Azure technology stack, including Azure Functions and cloud-native data services.
- Previous experience leading or mentoring Data Engineers within an Agile environment.
- Strong understanding of distributed data processing, modern data engineering practices and CI/CD.
- Experience building scalable, reliable and high-performing data platforms.
- Experience within Financial Services or Insurance is highly advantageous.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Why Join:
- Opportunity to shape the future of a modern, cloud-native data platform built on Databricks and Azure.
- Hands-on leadership role combining technical delivery with engineering strategy.
- Work with modern technologies including Python, PySpark, Databricks and the wider Azure ecosystem.
- Collaborative engineering culture focused on innovation, quality and continuous improvement.
- Competitive salary, bonus and comprehensive benefits package.
- Hybrid working from a modern London office.
Data & Application Notice
We process personal data submitted as part of your application for recruitment purposes in line with UK GDPR. All applications are reviewed by a member of our team, we do not use automated decision-making in our selection process. Your application will be assessed against the requirements of this role.
By applying to this advert, you consent to receiving future relevant roles and industry news and insights from us. To opt out of this, please contact: compliance@pioneer-search.com.
For more information on how we use your data, including how long we retain it & your rights, please refer to our Privacy Policy: www.pioneer-search.com/privacy-policy.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills