Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer (AI Pilot)
About the Role
Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join the Tendem project, where an AI Pilot (as Mindrift refers to this role) collaborates with Tendem Agents that handle repetitive tasks. You’ll contribute critical thinking, domain expertise, and quality control to ensure accurate, structured, and actionable results.
This part-time remote freelance opportunity is perfect for technical professionals specialising in web scraping, data extraction, and processing. Tapped into the Mindrift platform, you’ll work with major tech innovators, combining real-world expertise with Generative AI to unlock its potential by extracting and refining data.
Project Overview
The Tendem project involves sophisticated data scraping workflows, utilising tools such as Apify and OpenRouter, alongside your custom approaches for efficient data extraction and processing.
Key Responsibilities
As a Senior Python Data Scraping Engineer, you will:
- Own end-to-end data extraction workflows across complex websites, ensuring full coverage, accuracy, and reliable delivery of structured datasets (CSV, JSON, Google Sheets).
- Leverage internal tools (like Apify and OpenRouter) alongside custom workflow automation, ensuring fast, validated, and high-quality data collection.
- Extract data from dynamic and interactive web sources (JS/AJAX, infinite scroll, mobile-equivalent rendering) while adapting to site structure changes.
- Enforce data quality standards through validation checks, format consistency, and systematic verification before delivery.
- Scale scraping operations for large datasets using batch processing, parallelization, and failure monitoring, maintaining stability against minor web changes.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Requirements
Technical Skills & Experience
- 5+ years of experience in data engineering, web scraping, automation, or software development.
- Degree preferred (Bachelor’s/Master’s) in fields such as Applied Mathematics, Computer Science, or Engineering.
- Strong fundamental knowledge in scripting, automation, and AI-assisted workflows, with the ability to solve complex, non-trivial problems.
- Proficiency in Python for web scraping using BeautifulSoup, Selenium, or similar tools.
- Experience handling dynamic content (JavaScript, AJAX, infinite scroll) and API integration (with proxies).
- Ability to extract data from complex structures (archived pages, inconsistent HTML).
- Data cleaning, normalization, and validation expertise to produce polished datasets.
- Skills in avoiding anti-bot mechanisms and dynamically-resolved structures at scale.
- Hands-on experience with cloud computing (AWS or equivalent) and containerization (Docker).
- Experience with LLM frameworks (LangChain, OpenRouter, etc.) applied to automation.
- Familiarity with GitHub is a plus.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Soft Skills & Workflow
- Analytical and detail-oriented approach to ensure data accuracy.
- Highly self-directed to troubleshoot and solve issues independently.
- Upper-intermediate English (B2) for clear communication and documentation.
Project Time Expectations
Messy Material - Our Work Efficiently**
- Estimated workload: ~10–20 hours per week (active phases only).
- Not guaranteed, contingent on project requirements.
Compensation
- Pay starts at $45/hour for this project, depending on contribution pace and expertise.
- Varies per project based on scope, complexity, and required skills.
Note: Other projects may offer different compensation structures.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills