Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer (AI Pilot) – Tendem Project
Mindrift
About the Role: Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join the Tendem project and drive specialized data scraping workflows within our hybrid AI + human system. As an AI Pilot—this is our role designation at Mindrift—you will collaborate with Tendem Agents (which handle repetitive tasks) while applying critical thinking, domain expertise, and quality control to deliver precise and actionable results. This part-time, remote freelance opportunity is ideal for professionals with hands-on experience in:
- Data extraction
- Web scraping
- Data processing automation
- Working within AI-assisted workflows.
What We Do
The Mindrift platform connects specialists with AI projects from leading tech innovators. Our mission is to maximize the potential of Generative AI by integrating real-world expertise globally.
This is a freelance role focused on the Tendem project. As a Senior Python Data Scraping Engineer, you will:
- Execute data scraping tasks requiring technical precision for web extraction and processing using Apify and OpenRouter, alongside your own techniques.
- Implement custom workflows to streamline data collection, validation, and execution.
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Key Responsibilities
Manage end-to-end data extraction workflows across websites with complex structures, ensuring:
- Full coverage and accuracy
- Consistent delivery of structured datasets Leverage internal tools (Apify, OpenRouter) and custom solutions to:
- Speed up data collection and validation
- Meet project requirements efficiently Ensure reliable extraction from dynamic and interactive web sources, including:
- JavaScript-rendered content
- Changing site behavior Enforce data quality standards through:
- Validation checks
- Cross-source consistency controls
- Adherence to formatting specifications
- Systematic verification prior to delivery Optimize scraping operations for large datasets by:
- Implementing efficient batching
- Parallel processing
- Monitoring failures
- Stabilizing workflows against minor site changes
Requirements
- 5+ years of relevant experience in:
- Data engineering
- Web scraping
- Automation
- Software development (required)
- Bachelor’s or Master’s Degree in:
- Engineering
- Applied Mathematics
- Computer Science
- Related technical fields (a plus but not mandatory)
- Technical Proficiency
- Python (required)
- Web scraping libraries: BeautifulSoup, Selenium, or similar
- Handling dynamic content: JavaScript, AJAX, infinite scroll, APIs
- Working with proxies for API/data extraction
- Data cleaning, normalization, and validation (e.g., CSV, JSON, Google Sheets)
- Anti-bot mechanism detection and bypasses
- Scaling efforts using cloud infrastructure (AWS) and containerization (Docker)
- LLM/Automation Integration
- Experience with frameworks like LangChain, OpenRouter
- Problem-solving at scale through AI-assisted tools
- Soft Skills:
- Methodical, detail-oriented approach
- Independent troubleshooting
- Strong English proficiency (Upper-intermediate, B2)
- Preferred, but not required:
- Public GitHub portfolio for work examples
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Time Commitment
- Actual active hours: ~10–20 hours per week during the active project phase (estimate only; workload may vary).
- Project-based: Not a guaranteed workload, and phase duration depends on scope.
Compensation
- Accelerated payment rate: Up to $45/hour equivalent, based on expertise, pace, and project phase.
- Rates vary across projects depending on:
- Scope
- Complexity
- Required technical depth
- Please note: Compensation will differ on other Mindrift projects depending on their needs. Earning levels will depend on project specifics.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location