Mindrift
Senior Python Data Scraping Engineer (Freelance)
How your CV stacks up
Upload your CV to see how well it fits this job role
?%
Senior Python Data Scraping Engineer (Freelance)
Senior Python Data Scraping Engineer (AI Pilot) – Tendem Project @ Mindrift
Part-Time Remote, Freelance
Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join the Tendem project, a hybrid AI + human system where you’ll drive specialized data scraping workflows while collaborating with autonomous Tendem Agents. As an AI Pilot—how we refer to this role—you’ll provide critical thinking, domain expertise, and quality control to extract accurate, actionable datasets from complex digital sources.
The Opportunity
This remote, part-time freelance role is tailored for technical professionals with hands-on experience in web scraping, data extraction, and automation. You’ll work with Tendem Agents handling repetitive tasks while ensuring precision, scalability, and integrity in your outputs. Mindrift’s platform connects expert contributors with AI projects from leading tech innovators, empowering generative AI through real-world expertise.
In this specialist-centric Tendem project, you’ll leverage provided tools (Apify, OpenRouter) alongside custom workflows to extract, process, and validate data from dynamic websites. Your role is critical in ensuring end-to-end data integrity, from initial extraction to final delivery.
Key Responsibilities
- Own end-to-end data extraction workflows:
- Design, execute, and optimize scraping pipelines for complex websites, ensuring complete coverage, accuracy, and structured datasets.
- Adapt methodologies for dynamic/interactive content (JavaScript-rendered pages, AJAX, infinite scroll).
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
-
Leverage technical tools & custom solutions:
- Use Apify and OpenRouter alongside Python-based scrapers (BeautifulSoup, Selenium, or equivalents).
- Develop custom automation scripts to enhance efficiency and reliability.
-
Ensure data quality & validation:
- Enforce validation checks, cross-source consistency, and formatting standards.
- Conduct systematic verification before final delivery to meet project requirements.
-
Scale operations for large datasets:
- Implement batching, parallelization, and error handling to maintain performance under scale.
- Monitor failures, adapt to minor structural changes, and mitigate risks of scraping disruptions.
Requirements
Core Experience (Mandatory)
- At least 5+ years in data engineering, web scraping, automation, or software development.
- Hands-on expertise in Python web scraping, including:
- Parsing dynamic content (Beautm fenóutifulSoup, Selenium, or equivalent).
- Handling JS-rendered pages, APIs, and proxied requests.
- Proven ability to extract data from:
- Complex hierarchical structures (e.g., archived pages, inconsistent HTML).
- APIs, archived datasets, or large-scale datasets with anti-bot defenses.
- Data cleaning & normalization skills:
- Transform raw data into structured formats (CSV, JSON, Google Sheets).
- Validate and ensure cross-source consistency.
- Cloud & DevOps familiarity:
- Experience with AWS (or similar) and containerization (Docker) in real-world workflows.
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Advanced Skills (Preferred)
- LLM/automation frameworks:
- Hands-on experience with LangChain, Hugging Face, or OpenRouter for AI-assisted workflows.
- Anti-bot evasion techniques:
- Scaling scrapers to bypass CAPTCHAs, rate limits, and dynamic obscuration.
- Team collaboration:
- Ability to document workflows, troubleshoot independently, and integrate with AI agents.
- Technical degree or portfolio:
- Bachelor’s/Master’s in Computer Science, Applied Math, or Engineering.
- GitHub/Lab portal link demonstrating past projects (a plus).
Soft Skills
- Detail-oriented mindset with a strong emphasis on data accuracy.
- Self-directed problem-solving and independent troubleshooting.
- Upper-intermediate (B2) English proficiency (written/spoken) for remote collaboration.
Time Commitment & Compensation
- Estimated workload: 10–20 hours/week (active phases only; workload varies by project scope).
- Compensation: Up to $45/hr (equivalent), adjusted for expertise, speed, and contribution. Rates may differ per project based on complexity and requirements.
Note: This is a Tendem project; remuneration is tied to delivered output rather than fixed hours.
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Location