Public summary
A freelance senior-level Python engineer is needed for a part-time remote role working on advanced data scraping workflows within a hybrid AI-human system. The engineer will manage end-to-end data extraction from complex and dynamic websites using various tools and custom scripts, ensuring high accuracy and quality of structured datasets for AI projects. The ideal candidate has 5+ years in data engineering or software development, strong Python web scraping expertise including handling dynamic JavaScript content and anti-bot challenges, experience with cloud infrastructure and containerization, and familiarity with large language model (LLM) frameworks for automation. English proficiency at least B2 level is required. The position offers hourly compensation up to $45 depending on experience and contribution pace, with an estimated commitment of 10-20 hours per week during active project phases.
Location and work setup
- Location
- Germany
- Remote status
- Remote
- German requirement signal
- No German Required Detected
- Detected job language
- English
Salary
USD 45.00 - 45.00 hour
Responsibilities
- Manage and optimize end-to-end data extraction workflows across complex, dynamic, and interactive web sources. - Utilize provided internal tools (Apify, OpenRouter) along with custom workflows to accelerate data collection and validation. - Handle JavaScript-rendered content and adapt to changing site structures and behaviors. - Enforce rigorous data quality standards through validation, consistency checks, and verification. - Scale scraping operations efficiently, monitor for failures, and maintain stability. - Collaborate effectively with AI agents to complement automated tasks with domain expertise and quality control.
Qualifications
- 5+ years of experience in data engineering, web scraping, automation, or software development. - Bachelor’s or Master’s degree in Engineering, Applied Mathematics, Computer Science, or related technical fields preferred. - Strong proficiency in Python web scraping tools such as BeautifulSoup and Selenium. - Experience with dynamic web content (JavaScript, AJAX, infinite scrolling) and APIs through proxies. - Proven ability to extract data from complex and inconsistent web structures. - Expertise in data cleaning, normalization, validation, and delivering structured outputs (CSV, JSON, Google Sheets). - Experience managing anti-bot mechanisms and scaling scraping tasks. - Familiar with cloud services (AWS or equivalent) and containerization technologies like Docker. - Hands-on experience with large language model frameworks (LangChain, OpenRouter) for automation. - Detail-oriented, independent worker with strong troubleshooting skills. - Upper-intermediate or higher English proficiency (B2 or above). - GitHub profile is a plus.