Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer – Remote (Freelance)
Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join their Tendem project, specialising in building robust data scraping workflows as part of a hybrid AI + human system.
In this role (referred to as an AI Pilot at Mindrift), you’ll work alongside Tendem Agents handling repetitive tasks while providing critical thinking, domain expertise, and quality assurance to ensure accurate, structured, and actionable datasets.
This is a part-time remote freelance opportunity ideal for technical professionals with hands-on experience in:
- Web scraping
- Data extraction and processing
- Automation workflows
Why Mindrift?
The Mindrift platform connects specialists with AI-driven projects from leading tech innovators. Our mission is to unlock Generative AI’s potential by integrating real-world expertise globally.
As a Senior Python Data Scraping Engineer on this Tendem project, you’ll:
- Use provided tools (like Apify and OpenRouter) alongside custom solutions.
- Execute scalable data extraction pipelines with precision.
- Handle dynamic and interactive websites, including JavaScript-rendered content.
- Ensure data accuracy through validation, consistency checks, and cross-source verification.
- Optimise workflows for performance, stability, and efficiency.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Key Responsibilities
- Own end-to-end data extraction workflows across complex websites, ensuring full coverage, accuracy, and reliable structured datasets.
- Leverage technical tools (Apify, OpenRouter) and customised scripts to accelerate data collection, validation, and execution efficiency.
- Extract data from dynamic sites (AJAX, infinite scroll, evolving structures) with adaptable scraping techniques.
- Enforce data quality via strict validation, consistency checks, formatting adherence, and pre-delivery verification.
- Scale operations for large datasets (batching, parallelisation) while maintaining stability against structural changes.
Requirements
Must-have qualifications include:
- 5+ years of experience in data engineering, web scraping, automation, or software development.
- Degree (Bachelor’s or Master’s) in Engineering, Applied Mathematics, Computer Science, or a related field (preferred—not mandatory but advantageous).
- Proven expertise in:
- Python web scraping (BeautifulSoup, Selenium, aliases, proxies).
- Dynamic content scraping (JS/AJAX, infinite scroll).
- Pooling data from complex structures (archived pages, inconsistent HTML).
- Data cleaning, normalisation, and validation (output formats: CSV, JSON, Google Sheets).
- Hands-on experience with anti-bot detection and scalable scraping at a technical level.
- Cloud/GitOps familiarity (AWS, Docker deployments).
- Working knowledge of LLM frameworks (LangChain, OpenRouter) for workflow automation.
- Strong attention to detail and commitment to data integrity.
- Self-directed problem solver, able to independently troubleshoot and maintain performance.
- GitHub portfolio (a plus—link appreciated).
- English proficiency: B2 (Upper-intermediate) or higher (essential).
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Time Commitment & Compensation
- Estimated workload: 10–20 hours per week (during active project phases). This is an estimate—not guaranteed—and only applies while the project is live.
- Compensation: $45/hour (equitable variation based on individual pace and project scope). (Note: Rates may differ across Mindrift’s other projects depending on complexity/expertise.)
Apply today! Contribute at your own pace as a freelancer, shaping data-driven solutions in collaboration with AI.
(This is a direct referral to freelance contributors on Mindrift’s platform.)
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location