ANTTECHS
Research Crawling Engineer
United Kingdom
Posted about 14 hours ago
How your CV stacks up
1Upload CV
2Analyse CV
3Improve CV
Upload your CV to see how well it fits this job role
?%
Position: Research Crawling Engineer
📍 Location: Remote (Global)
🕒 Requirement: Must have at least 6 hours of overlap with US Eastern Time (EST)
We're looking for a Research Crawling Engineer to help build and operate web-scale crawling infrastructure that powers AI training datasets.
What You'll Do
- Build and maintain large-scale web crawlers across diverse domains
- Design high-throughput, fault-tolerant distributed data collection systems
- Handle anti-bot protections, rate limits, proxy orchestration, and JS-heavy websites
- Develop pipelines for data cleaning, deduplication, normalization, and dataset creation
- Monitor crawl performance, data quality, and infrastructure reliability
- Collaborate with research teams to support AI and machine learning initiatives
Reasons to use Rodeo
Start with a chat, not a search bar
Grad scheme, placement, apprenticeship? Not sure what you want yet — that's fine. Your agent talks it through with you and turns "I have no idea" into a shortlist.
It searches the market for you
Every day your agent scans the market matching roles against what actually matters to you, not just keywords on a CV.
Only hits
No noise. No "maybe this fits." Just roles with a clear explanation of why they're right — and where to focus when applying.
Requirements
- 3+ years of experience building large-scale web crawlers or distributed data pipelines
- Strong programming skills in Go, Rust, Python, Java, or C++
- Solid understanding of HTTP, networking, browser behavior, and distributed systems
- Experience handling anti-bot systems, rate limiting, proxies, and dynamic websites
- Experience building and maintaining large datasets for research or machine learning
Get help with your application
Your very own career expert that helps elevate your application to the next level.
Nice to Have
- Experience with NLP pipelines or LLM training datasets
- Experience with Playwright, Puppeteer, or Chrome DevTools Protocol
- Knowledge of cloud or bare-metal infrastructure
- Experience with data quality evaluation or benchmarking
Trusted by 25,000+ job seekers
“It took my CV and asked me questions relevant to understanding what kind of jobs to suggest for me. Suggestions were almost perfect. Jobs were exactly what I’ve been looking for.”
Jessica, London
Skills
Web Crawling
Distributed Systems
Go
Rust
Python
Java
C++
HTTP
Networking
Anti-bot Protection
Proxy Orchestration
Data Cleaning
Deduplication
Playwright
Puppeteer
Chrome DevTools Protocol
Location