Please mention DailyRemote when applying
See how much of this job your resume covers, and what’s missing.
Want a recruiter to go through it line by line?
Get professional reviewOne click makes a copy of your resume for this role, scores it against the posting, and proposes edits line by line. Your default resume is never changed.
Upload your resume and we draft a letter for this exact role, tailored to what it asks for.
Design and maintain high-throughput, fault-tolerant web crawling systems to collect large-scale datasets for AI research. Develop data pipelines to clean, normalize, and manage TB-to-PB scale data while optimizing infrastructure for performance and cost.
Join a data infrastructure team building large-scale web crawling systems and datasets for AI research and model development. This role focuses on reliable data acquisition across diverse, changing web environments.
Build and maintain web crawlers across social, travel, and multilingual websites.
Design high-throughput, fault-tolerant systems that collect millions to billions of URLs per day.
Adapt crawling approaches to anti-bot systems, rate limits, and dynamic or JavaScript-heavy sites.
Develop pipelines to clean, deduplicate, filter, and normalize web data at TB-to-PB scale.
Build and maintain datasets for research and model training in collaboration with research teams.
Monitor crawl performance, coverage, and data quality, and adjust systems as websites change.
Optimize crawling infrastructure for cost, latency, and reliability across cloud or bare-metal environments.
At least 2 years of experience building web crawlers or large-scale data pipelines; typically 4 or more years, or 2 or more years with a PhD.
Hands-on experience building and operating web crawlers at scale, including data pipelines handling TB-scale workloads.
Strong proficiency in Go, Rust, Python, Java, or C++, with experience in distributed systems or parallel processing.
Deep knowledge of HTTP, networking, and browser behavior, including experience working with anti-bot systems, rate limits, and JavaScript-heavy sites.
Experience with headless browsers, proxy systems, IP rotation, and request orchestration is valuable.
Familiarity with data quality evaluation, large-scale data processing, and cloud or bare-metal infrastructure.
Experience with NLP pipelines, machine learning dataset curation, or AI research data is a plus.
A degree in computer science, engineering, or a related technical field is helpful; a PhD is a plus.
Salary range: $160,000 to $250,000 annually. Visa sponsorship is not available.
Remote.
Stop the endless job search. Our AI finds and applies to the best jobs for you.
Featuring 222,997+ Jobs in Software Development
Answer easy questions
222,997+ jobs across 15+ categories
Get your best job matches
Only hand-screened, legit jobs
Find a remote job faster
No ads, scams, or junk
“I was the first applicant for a remote marketing position that got listed on the company website the same day I applied. Had an interview within 48 hours!”
— Sarah J.
·
Marketing Manager
★★★★★
Verified
