Data Engineer - Web Scraping

This is a remote position.

Requirements

• Collaborate with analysts to understand and anticipate requirements.
  
• Design, implement, and maintain Web scrapers for a wide variety of alternative datasets.
  
• Perform data cleaning, exploration, transformation, etc., on scraped data.
  
• Collaborate with cross-functional teams to understand data requirements and implement efficient data processing workflows.
  
• Author QC checks to validate data availability and integrity.
  
• Maintain alerting systems and investigate time-sensitive data incidents to ensure smooth day-to-day operations.
  
• Design and implement products and tools to enhance the Web scraping Platform.
  
  
  
  Must Have 
  
• Bachelor's/master’s degree in computer science or in any related field. 
  
• 2-4 years of software development experience
  
• Strong Python and SQL/Database skills
  
• Strong expertise in using the Pandas library (Python) is a must
  
• Experience with web technologies (HTML/JS, APIs, etc.)Proven work experience in working with large data sets for Data cleaning, Data transformation, Data manipulation, and Data replacements.
  
• Excellent verbal and written communication skills
  
• Aptitude for designing infrastructure, data products, and tools for Data Scientists
  
  Preferred
• ​Familiarity with scraping and common scraping tools (Selenium, Scrappy, Fiddler, Postman, XPath)
  
• Experience containerizing workloads with Docker (Kubernetes a plus)
  
• Experience with build automation (Jenkins, GitLab CI/CD)
  
• Experience with AWS technologies like S3, RDS, SNS, SQS, Lambda, etc.