Montash
Python Data Engineer (SC Cleared)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Job Title: Python Data Engineer
Location: Remote (ad hoc travel to London office)
Contract Length: 3 Months (with scope to extend)
Start Date: ASAP
IR35: Inside
Interview Process: 1 Stage, MS Teams
Clearance: SC
We are supporting a public sector client in hiring a Python Data Engineer with strong PySpark, Delta Lake, and Azure cloud experience to join a data engineering team delivering scalable, test-driven data processing solutions in a containerised environment.
This is a hands-on engineering role requiring deep Python and PySpark proficiency, with a strong emphasis on behaviour-driven development, modular code design, and Delta Lake optimisation. The successful candidate will work closely with cloud architects, data scientists, and DevOps teams to deliver robust, well-tested pipelines across Azure infrastructure.
This role suits an experienced Python Data Engineer comfortable with test-driven and configuration-driven development, confident working independently in a fast-paced Agile environment, and familiar with Azure cloud services across Functions, Key Vault, Blob Storage, and Data Lake Storage Gen2.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Key Responsibilities
- Develop and maintain data ingestion, transformation, and validation pipelines using Python and PySpark
- Implement unit and behaviour-driven tests using Behave, including robust mocking and patching of dependencies
- Design and maintain Delta Lake tables for optimised query performance, ACID compliance, and incremental data loads
- Build and manage containerised environments using Docker for consistent development, testing, and deployment workflows
- Develop configurable, parameter-driven codebases to support modular and reusable data solutions across environments
- Integrate Azure services including Azure Functions, Azure Key Vault, Azure Blob Storage, and Data Lake Storage Gen2
- Collaborate with cloud architects, data scientists, and DevOps teams to ensure seamless CI/CD workflows, version control, and environment consistency
- Troubleshoot and optimise Spark jobs for performance and scalability in production environments
- Maintain technical documentation and adhere to best practices in cloud security and data governance
Essential Skills
Get help with your application
Your very own career expert that helps elevate your application to the next level.
- Strong Python proficiency with emphasis on modular, test-driven, and configuration-driven design
- Demonstrated experience writing unit tests and BDD scenarios using Behave or similar frameworks
- In-depth understanding of mocking, patching, and dependency injection in Python testing
- Proficiency in PySpark with hands-on experience in distributed data processing and performance tuning
- Solid understanding of Delta Lake concepts, transactional guarantees, and schema evolution
- Experience with Docker for development, testing, and deployment workflows
- Familiarity with Azure services including Azure Functions, Key Vault, Blob Storage, and Data Lake Storage Gen2
- Ability to implement configuration-driven applications for flexible deployment across environments
Desirable Skills
- Experience developing in Databricks or Synapse with Delta Lake integration
- CI/CD pipeline experience using Azure DevOps or similar tooling
- Knowledge of data security and governance best practices within Azure ecosystems
- Experience collaborating with distributed teams in Agile delivery environments
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location