Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer (AI Pilot)
Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join the Tendem project and drive specialised data scraping workflows within our hybrid AI + human system.
In this role—as an AI Pilot at Mindrift—you’ll collaborate with Tendem Agents that handle repetitive tasks, while providing critical thinking, domain expertise, and quality control to deliver accurate and actionable results.
This part-time, remote freelance opportunity is ideal for technical professionals with hands-on experience in web scraping, data extraction, and processing.
About Mindrift
The Mindrift platform connects specialists with AI projects from major tech innovators. Our mission is to unlock the potential of Generative AI by tapping into real-world expertise globally.
As a Senior Python Data Scraping Engineer within this Tendem project, you’ll manage data scraping tasks demanding technical precision for robust web extraction and processing, utilising tools like Apify and OpenRouter, along with custom methodologies.
Key Responsibilities
- Own end-to-end data extraction workflows across complex websites, ensuring complete coverage, accuracy, and reliable delivery of structured datasets
- Leverage internal tools (Apify, OpenRouter) alongside custom workflows to accelerate data collection, validation, and execution while meeting project requirements
- Extract data reliably from dynamic and interactive web sources, adapting strategies for JavaScript-rendered content and changing site structures
- Implement data quality standards through validation checks, cross-source consistency controls, formatting adherence, and systematic verification before delivery
- Scale scraping operations for large datasets using:
- Efficient batching or parallelisation
- Failure monitoring
- Stability against structural changes
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Requirements
Core Qualifications
- 5+ years of experience in data engineering, web scraping, automation, or software development (required)
- Bachelor’s or Master’s Degree in:
- Engineering
- Applied Mathematics
- Computer Science
- Related technical fields (a plus, but not mandatory)
Technical Proficiencies
- Strong background in Python for web scraping (e.g., BeautifulSoup, Selenium, or similar), including:
- Dynamic content (JavaScript, AJAX, infinite scroll)
- APIs via proxies
- Proven ability to extract data from complex structures such as:
- Hierarchies
- Archived pages
- Inconsistent HTML
- Experience in data cleaning, normalisation, and validation, delivering structured outputs in:
- CSV
- JSON
- Google Sheets
- Hands-on experience with anti-bot mechanisms and dynamic sites at scale
- Familiarity with cloud infrastructure (e.g., AWS) and containerisation (Docker) in workflows
- Experience with LLM frameworks (e.g., LangChain, OpenRouter) for automation tasks
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Soft Skills & Mind-set
- Detail-oriented with a commitment to data accuracy
- Self-directed, with the ability to troubleshoot independently
- A personal GitHub link is a plus
- English proficiency at Upper-intermediate (B2) or above (required)
Project & Compensation
Time Commitment
- Active phases require 10–20 hours per week (depending on project scope).
- This is an estimate; exact workload varies by activity level.
Compensation
- $45/hour equivalent (varies by project difficulty/expertise).
- Other platforms projects may offer different rates based on scope, complexity, and required skills.
Mindrift: Helping AI Advance Through Human Expertise.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location