Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer – AI Pilot (Freelance)
About the Role
Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join the Tendem project and specialise in advanced data scraping workflows for our hybrid AI + human system.
In this role, as an AI Pilot (our terminology for this position), you’ll collaborate with Tendem Agents handling repetitive tasks while driving critical thinking, domain expertise, and quality control. This part-time, remote freelance opportunity is perfect for professionals with hands-on experience in web scraping, data extraction, and automation.
About Mindrift
The Mindrift platform connects specialist experts with AI-driven projects from leading tech innovators. Our mission is to unlock the potential of Generative AI by integrating real-world expertise globally.
Key Responsibilities
- Owner and architect of end-to-end data extraction workflows, ensuring comprehensive coverage, accuracy, and reliable delivery of structured datasets for complex websites.
- Leverage proprietary tools (Apify, OpenRouter) alongside custom workflows to accelerate data collection, validation, and task execution while meeting project requirements.
- Extract data from dynamic, JS-rendered content and interactive sources, adapting techniques as sites evolve.
- Enforce data quality standards through validation checks, cross-source consistency verification, and adherence to formatting specs.
- Optimize scraping for large datasets using batching/parallelization while monitoring failures and maintaining stability amidst minor structural changes.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Requirements
Essential
- 5+ years of experience in data engineering, web scraping, automation, or software development (required).
- Bachelor’s or Master’s degree in Engineering, Applied Mathematics, Computer Science, or related fields (preferred).
- Core skills in Python web scraping (BeautifulSoup, Selenium, etc.), dynamic content (AJAX, infinite scroll), and API integration with proxies.
- Strong expertise in extracting data from complex structures (archived pages, inconsistent HTML).
- Hands-on experience with data cleaning, normalization, and validation, delivering structured outputs (CSV, JSON, Google Sheets).
- Proven ability to handle anti-bot mechanisms, dynamic site structures, and scaling tasks at enterprise levels.
- Prior experience with cloud infrastructure (AWS/GCP), Docker, and real-world containerized workflows.
- LLM frameworks expertise: LangChain, OpenRouter, or equivalent for automation tasks.
- Fluent in English (Upper-intermediate B2 or higher).
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Highly Preferred
- Personal GitHub demonstrating unique scraping solutions.
- Enhanced problem-solving and independent troubleshooting mindset.
Note: This contract-based project expects 10–20 hours/week during active phases (numbers contingent on workload and progress delivery).
Compensation & Perks
Compensation starts at up to $45/hour (pro-rated), depending on expertise and engagement scope. Projects outside this initiative may offer varying rates based on complexity.
Apply with a tailored summary if you bring domain-specific scraping experience to deliver consistent quality results!
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills