IntelStack
Senior Data Engineer
Get notified of more jobs like this · No spam, ever
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Data Engineer | Azure Databricks | SC Cleared | London (Hybrid, 3 days on site)
Build the data platform behind national economic analysis. We are hiring a Senior Data Engineer to develop and scale an Azure Databricks platform that powers monetary analysis, economic forecasting and modelling. The data you engineer feeds the models that inform decisions at a national level, so quality, lineage and reliability genuinely matter here.
This is hands on engineering, not oversight. You will own pipelines end to end, from ingestion through transformation to governed, production ready datasets used daily by economists and data scientists.
What you will be doing
- Designing and building scalable ingestion pipelines from APIs, relational databases and financial data providers into Azure Databricks
- Writing complex transformations in PySpark and Spark SQL, with Delta Lake as the storage foundation
- Tuning Databricks workloads for performance and cost, and embedding quality checks and validation into every pipeline
- Implementing governance in practice: Unity Catalog, Azure Purview, data lineage, metadata and monitoring
- Automating deployments through CI/CD in Azure DevOps and promoting DevOps practice across the team
- Working directly with economists and data scientists to turn analytical requirements into engineered solutions
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
What we need from you
- 8+ years in data engineering, including 3+ years hands on with Azure Databricks
- Strong Python and PySpark, backed by advanced SQL
- Deep grounding in data warehousing, data modelling and integration patterns
- Broad Azure data stack experience: Data Factory, Blob Storage or ADLS, Azure SQL
- Unity Catalog and Delta Lake in a production environment
- Data quality and governance experience, ideally including Azure Purview
- Git, Agile delivery and a collaborative approach to code review
- Active SC clearance. This is a firm requirement and we are unable to sponsor or process new clearance for this role
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Nice to have
- Streaming experience with Kafka or Azure Event Hubs
- Financial services, economic or public sector data experience
- Azure Data Engineer Associate or Databricks certification
- Power BI or Tableau exposure
- Docker, Kubernetes, Terraform or infrastructure as code
- R, YAML or JavaScript alongside the core stack
Why this role
You get a modern, well funded Azure estate, a mandate to build it properly rather than patch it, and work with real public significance. Hybrid working, three days on site in central London.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location