Senior Python Data Scraping Engineer (AI Pilot)

Mindrift is seeking highly skilled Senior Python Data Scraping Engineers to join the Tendem project and drive specialized data scraping workflows within our hybrid AI + human system.

In this part-time remote freelance role (referred to as an AI Pilot at Mindrift), you’ll collaborate with Tendem Agents—handling repetitive tasks—while providing critical thinking, domain expertise, and quality control to deliver accurate, actionable results.

About the Role

Mindrift’s platform connects specialists with AI projects from major tech innovators, unlocking the potential of Generative AI through real-world technical expertise globally.

As a Senior Python Data Scraping Engineer, your responsibilities include:

• Manually or programmatically extracting, validating, and processing data from web sources for Tendem’s automated workflows.
• Ensuring accurate and actionable results that integrate seamlessly with AI agents.

Key Responsibilities

• Own end-to-end data extraction workflows across complex websites, ensuring complete coverage, accuracy, and delivery of structured datasets.
• Leverage provided tools (Apify, OpenRouter) alongside custom Python workflows to accelerate data collection, validation, and execution while meeting requirements.
• Extract data from dynamic/dynamic sources (JavaScript-rendered, AJAX, infinite scroll) and adapt approaches as needed.
• Enforce data quality standards through validation checks, cross-source consistency, formatting adherence, and systematic verification.
• Scale scraping operations for large datasets using batching/parallelization, monitor failures, and maintain stability against structural changes.
• Proactively troubleshoot and iterate on workflows to minimize bottlenecks and ensure reliability.