Cognitive Group | Part of the Focus Cloud Group
Data Ingestion Engineer
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Data Engineer
We are looking for an experienced Senior Data Engineer to join a high-performing data engineering team responsible for large-scale data ingestion and processing.
You'll play a key role in ensuring critical data pipelines remain reliable, scalable and efficient, working with high-volume datasets that support analytics, data science and machine learning initiatives.
This is a hands-on engineering role focused on solving complex production challenges, improving data pipeline resilience and optimising distributed data processing systems.
Key Responsibilities:
- Build, maintain and optimise large-scale data ingestion and ETL pipelines using Apache Spark and Python.
- Debug and resolve failing or blocked data ingestion pipelines in production.
- Investigate issues caused by corrupt, malformed or unexpected data and implement robust solutions to prevent pipeline failures.
- Improve pipeline resilience so individual data quality issues do not interrupt wider workflows.
- Process and transform large datasets from multiple internal and third-party data sources.
- Design and improve handling of varied data formats from partners, suppliers and external providers.
- Support orchestration across multi-stage ingestion workflows, including dependency management, retries and queue handling.
- Optimise Spark jobs for throughput, reliability and compute efficiency.
- Reduce operational overhead by minimising failed jobs, stalled pipelines and manual intervention.
- Work closely with engineering teams to ensure critical datasets are delivered to downstream analytics, data science and machine learning teams.
- Contribute practical improvements that enhance the scalability, reliability and maintainability of the data platform.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Required Skills & Experience:
- Strong commercial experience with Apache Spark (PySpark) in production environments.
- Excellent Python development skills.
- Proven experience designing, building and supporting large-scale data ingestion, ETL or data processing pipelines.
- Experience working with distributed data processing technologies.
- Strong understanding of data ingestion, transformation and loading processes.
- Experience diagnosing and resolving production pipeline failures.
- Comfortable working with incomplete, inconsistent or corrupt datasets.
- Experience optimising data processing jobs for performance, scalability and cost efficiency.
- Understanding of workflow orchestration, multi-stage pipelines and downstream dependencies.
- Ability to work independently in a fast-paced, highly technical environment.
- A pragmatic, delivery-focused approach with excellent problem-solving skills.
- Experience working with very large datasets, ideally in petabyte-scale or other high-throughput environments.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Desirable Experience:
- Experience with workflow orchestration tools such as Airflow, Dagster or similar.
- Cloud platform experience (AWS, Azure or GCP).
- Knowledge of modern data lake or data warehouse architectures.
- Experience supporting machine learning or data science platforms.
This is an Inside IR35 contract role working via Umbrella Company so you must be willing and able to work on this basis to be considered for this role.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location