CipherTek Recruitment
Senior Data Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
🔥 Senior / Lead Data Engineer – Databricks / Spark (High-Performance Platform)
£850 p/d OUTSIDE IR35 (higher for Lead)
12-month rolling (multi-year programme)
1 day/week – St Paul’s, London
We’re hiring a Senior & Lead Data Engineer to build a Databricks lakehouse platform in a high-performance, business-critical Front office Risk trading environment.
This is a hands-on engineering role focused on building and optimising large-scale distributed data systems.
This is a highly technical team operating at scale, we’re looking for engineers with deep data engineering expertise, strong low-level Spark knowledge, and experience building high-performance systems using modern Databricks and AI-driven platforms.
This is working for a Front office Risk Data team, so you MUST have experience working closely with Front/middle office users and have good domain knowledge covering Risk and derivatives.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
What you’ll do
- Build and optimise Spark pipelines on Databricks
- Develop a lakehouse platform (Medallion architecture)
- Own data modelling, architecture, and pipeline design
- Work with large-scale data (TB–PB)
- Drive performance, scalability, and reliability in production
What we’re looking for
- Strong experience running Spark workloads in production
- Proven ability to optimise Spark at scale (Tb/PB datasets)
- Solid Python (Scala beneficial, not essential)
- Experience with data modelling and lakehouse architecture
- Ability to debug and improve performance in distributed systems
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Important
- Must have recent, hands-on Spark experience
- Databricks strongly preferred (not essential if Spark depth is very strong)
- Experience supporting AI/ML or advanced analytics platforms is a big plus
- Financial services / trading exposure
- Experience in performance-critical environments
Not a fit if
- Primarily BI / reporting focused
- Spark used only at small scale or outside production
- No experience with performance optimisation in distributed systems
Stack
- Databricks (Azure)
- Spark
- Delta Lake
- Python (+ Scala optional)
Bottom line
We’re looking for engineers who can design, build, and optimise Spark-based systems at scale and operate effectively in a performance-critical environment from day one.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills