Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer (AI Pilot) – Tendem Project
Mindrift is seeking hands-on Senior Python Data Scraping Engineers to join the Tendem project, where you’ll expertly manage specialized data scraping workflows within our hybrid AI + human system.
In this part-time, remote freelance role, you’ll act as an AI Pilot, collaborating with Tendem Agents to tackle repetitive tasks while providing critical thinking, domain expertise, and quality control for accuracy and actionable results.
This opportunity is ideal for professionals with practical experience in web scraping, data extraction, and processing.
What We Do
Mindrift connects specialist human experts with AI projects from leading technology innovators. Our mission: unlocking the potential of Generative AI by leveraging global real-world expertise.
As a Senior Python Data Scraping Engineer, you’ll:
- Handle data scraping tasks requiring technical precision for web extraction and processing using our platform’s tools (Apify, OpenRouter) and custom approaches.
Key Responsibilities
- Own end-to-end data extraction workflows across complex websites, ensuring:
- Full coverage
- Data accuracy
- Reliable delivery of structured datasets
- Leverage tools (Apify, OpenRouter) and custom workflows to:
- Speed up data collection, validation, and execution
- Meet project requirements efficiently
- Extract data reliably from dynamic, interactive sources, adapting to:
- JavaScript-rendered content
- Evolving website behaviors
- Enforce data quality standards through:
- Validation checks
- Cross-source consistency controls
- Strict formatting adherence
- Systematic verification before delivery
- Optimize systems for large-scale scraping:
- Implement batching/parallelization
- Monitor failures
- Maintain stability against minor structural changes
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Requirements
Technical Proficiency
- 5+ years of experience in data engineering, web scraping, automation, or software development (required)
- Bachelor’s/Master’s degree in Engineering, Applied Mathematics, Computer Science, or related field (preferred)
- Strong understanding of scripting, automation, and AI-assisted workflows
- Ability to:
- Solve complex technical problems
- Work confidently with large language models (LLMs)
- Systematically collect, structure, and validate data
Python & Web Scraping Expertise
- Hands-on experience with:
- Python web scraping (BeautifulSoup, Selenium)
- Dynamic content (JavaScript, AJAX, infinite scroll)
- APIs via proxies
- Proven ability to extract data from:
- Complex structures (nested hierarchies, archived pages, inconsistent HTML)
- Data cleaning/validation skills to produce structured datasets (CSV, JSON, Google Sheets)
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Scalability & Automation
- Experience with:
- Anti-bot mechanisms
- Large-scale dynamic site structures
- Familiarity with:
- Cloud infrastructure (AWS or equivalent)
- Containerization (Docker)
LLM & AI Awareness
- Demonstrated experience with:
- LLM frameworks (LangChain, OpenRouter)
- AI-assisted automation
Soft Skills
-
Attention to detail and commitment to data accuracy
-
Self-directed & independent troubleshooting
-
GitHub portfolio link (a plus)
-
English proficiency: Upper-intermediate (B2) or higher (required)
Time Commitment
This is a freelance project with estimated workloads of 10–20 hours/week during active phases (adjusts per project needs).
Compensation
Contributors can earn up to $45/hour (equivalent), based on expertise and contribution pace. Note: Earnings vary per project scope, complexity, and requirements.
Other Mindrift projects may offer different compensation rates.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills