Empresaria Group plc
Data Engineer
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Data Engineer (Azure Databricks) – SC Cleared
Location
London, UK (3 days per week onsite)
Salary
Up to £75,000 per annum
Job Type
Permanent, Full-Time
Security Clearance
Active SC Clearance Required
The Opportunity
We are seeking an experienced Senior Data Engineer to join a high-performing team responsible for developing and enhancing a modern Azure-based data platform supporting economic analysis, forecasting, and modelling activities.
This role offers the opportunity to work with large-scale datasets, cloud-native technologies, and modern data engineering practices. You will play a key role in designing and optimising data pipelines, implementing data governance processes, and delivering reliable data solutions that enable critical business insights.
Key Responsibilities
Data Pipeline Development & Optimisation
- Design, develop, and maintain scalable and reliable data pipelines.
- Build ETL/ELT solutions to ingest, transform, and load data from multiple sources, including APIs, databases, and external data providers.
- Optimise pipeline performance, scalability, and cost efficiency.
- Implement data quality controls, validation processes, and monitoring solutions.
Data Transformation & Processing
- Develop complex data transformation logic using Spark (PySpark or Scala).
- Build data cleansing, enrichment, aggregation, and processing solutions.
- Ensure data accuracy, consistency, and reliability throughout the data lifecycle.
Azure Databricks Development
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
- Develop and support solutions within Azure Databricks.
- Work extensively with Unity Catalog, Delta Lake, Spark SQL, and related Databricks services.
- Apply Databricks best practices for development, deployment, and workload optimisation.
- Develop solutions using SQL, Python, R, YAML, and JavaScript.
Data Integration
- Integrate data from relational databases, APIs, and streaming platforms.
- Design and implement robust data integration patterns.
- Collaborate with development teams to support seamless data exchange across systems.
Data Quality & Governance
- Utilise Azure Purview for data governance and data quality initiatives.
- Implement data lineage, metadata management, monitoring, and alerting capabilities.
- Ensure compliance with organisational data governance standards and best practices.
Collaboration & Delivery
- Work closely with data scientists, analysts, economists, and technical stakeholders.
- Translate business requirements into scalable technical solutions.
- Participate in code reviews, technical discussions, and knowledge-sharing sessions.
- Contribute to Agile delivery processes and continuous improvement initiatives.
DevOps & Automation
- Automate data engineering workflows and deployments.
- Develop and maintain CI/CD pipelines.
- Promote DevOps best practices across the wider engineering team.
Required Skills & Experience
- 8+ years of Data Engineering experience (10+ years preferred).
- Minimum 3 years of hands-on Azure Databricks experience.
- Strong expertise in Python, PySpark, and/or Scala.
- Extensive experience with:
- Azure Databricks
- Azure Data Factory
- Azure Blob Storage
- Azure SQL Database
- Delta Lake
- Spark SQL
- Strong understanding of:
- Data architecture
- Data modelling
- Data warehousing
- Data integration patterns
- Experience working with large-scale datasets and complex data pipelines.
- Knowledge of SQL and NoSQL databases.
- Experience with data governance, data quality frameworks, and metadata management.
- Experience using Git and working within Agile delivery environments.
- Excellent communication, stakeholder management, and problem-solving skills.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Desirable Skills
- Azure Purview
- Kafka
- Azure Event Hubs
- Power BI
- Tableau
- Azure DevOps
- Jenkins
- Docker
- Kubernetes
- Experience within Financial Services, Banking, or Economic Data environments
- Microsoft Certified: Azure Data Engineer Associate
- Databricks Certifications
What’s On Offer
- Salary up to £75,000.
- Hybrid working model with 3 days per week in a London office.
- Opportunity to work on a large-scale Azure and Databricks data platform.
- Exposure to enterprise-level data engineering challenges and modern cloud technologies.
- Collaborative and technically focused environment.
- Career development and certification support opportunities.
Please note: Active SC Clearance is required for this position. Applicants without current SC Clearance will not be considered.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills