UST
Lead Data engineer (Lead II - Data Engineering)
Leeds
Posted 1 day ago
How your CV stacks up
1Upload CV
2Analyse CV
3Improve CV
Upload your CV to see how well it fits this job role
?%
Role Description
Role: Lead Data Engineer
Fixed term contract
Contract Length: Initial 3-6 months with possible extensions
Start Date: ASAP
Experience Range: 10-12 Years
Location Requirement: Onsite (3 days per week in the office and 2 days remote)
Applicants must be legally authorized to work in the United Kingdom without the need for current or future visa sponsorship.
Responsibilities
- Design, build and optimize robust data ingestion pipelines to acquire, transform and load data vendor platforms into client's data ecosystem.
- Develop scalable data engineering solutions using Databricks, PySpark, SparkSQL and associated cloud technologies.
- Build and maintain secure, reliable and automated data ingestion processes from external vendor systems, including SFTP-based file transfers and other integration methods.
- Ensure data is landed, structured, governed and accessible to support reporting, analytics and business use cases.
- Work with Business Analysts, Product Owners, Architects and delivery squads to translate business requirements into technical data solutions.
- Support data discovery, profiling and validation activities to understand source data structures, data quality issues and data gaps.
- Develop and maintain data transformations, curated datasets and data models required to support reporting and analytical use cases.
- Monitor, troubleshoot and resolve data ingestion issues, defects and enhancements identified during development, testing, UAT and production support.
- Ensure solutions comply with data architecture standards, engineering best practices, security requirements and governance frameworks.
- Produce clear technical documentation for data ingestion processes, data flows and operational support requirements.
- Provide comprehensive handover documentation and knowledge transfer to the Data Support team following delivery of data ingestion pipelines.
- Collaborate within Agile delivery teams, actively contributing to sprint planning, stand-ups, retrospectives and continuous improvement activities.
- Identify opportunities to improve pipeline performance, automation, scalability and maintainability through process and technology enhancements.
- Support knowledge sharing and contribute to Engineering and Data Communities of Practice.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Essential Skills & Experience
- Proven experience designing, building and supporting enterprise-scale data ingestion pipelines and ETL/ELT solutions.
- Strong hands-on experience with Databricks, PySpark and SparkSQL.
- Experience developing and supporting secure data integrations using SFTP and other file-based or API-driven ingestion mechanisms.
- Experience ingesting and processing structured, semi-structured and unstructured data from internal and third-party source systems.
- Strong understanding of data modelling, transformation techniques and data warehousing principles.
- Experience working with cloud-based data lake and analytics platforms.
- Strong understanding of batch and near real-time data processing patterns.
- Experience conducting data profiling, discovery and validation activities to assess data quality, completeness and suitability for business requirements.
- Experience implementing data quality checks, reconciliations and monitoring processes.
- Ability to investigate and resolve ingestion, transformation and data quality issues identified during testing, UAT or production support.
- Understanding of data governance, security, data lineage and documentation standards.
- Experience producing technical documentation and operational handover materials.
- Strong stakeholder engagement skills with the ability to work effectively across business, architecture, engineering and analytics teams.
- Experience working within Agile delivery environments.
- Knowledge of source control, CI/CD practices and release management processes.
- Ability to work independently while collaborating effectively within cross-functional squads.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Desirable
- Experience integrating data from retail technology platforms, IoT devices or third-party vendor systems.
- Experience working with AI Camera, Computer Vision or Electronic Shelf Edge Label (eSEL) technologies.
- Knowledge of Azure Data Lake, Azure Data Factory and related Azure data services.
- Experience supporting reporting, analytics or BI solutions through the creation of trusted and governed data assets.
- Experience working within large-scale retail or data transformation programmes.
If interested, please apply with your updated CV for an immediate discussion.
#UST
Skills
PySpark, Azure Data Factory, Agile, CI/CD
Trusted by 25,000+ job seekers
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Databricks
PySpark
SparkSQL
Azure Data Factory
ETL/ELT
Data Ingestion
Data Modelling
CI/CD
Agile
SFTP
Cloud Data Lakes
Data Governance
Location