Public summary
Seeking a freelance engineer to evaluate AI coding agents by designing and testing realistic developer tasks within simulated environments. Candidates should have 5+ years in software development, proficiency in Python, JavaScript/TypeScript, related technologies, and strong testing experience. English proficiency B2+ is required. This project-based role offers flexible scheduling and payment up to $50 per hour.
Location and work setup
- Location
- Munich
- Remote status
- Remote
- German requirement signal
- No German Required Detected
- Detected job language
- English
Salary
USD 50.00 - 50.00 hour
Responsibilities
Create realistic development environments simulating codebase, infrastructure, and context. Design intermediate-state tasks with clear success criteria suitable for AI agents. Write tests that validate all correct agent solutions and reject incorrect ones with balanced strictness. Iterate on tasks and tests based on qualitative feedback and agent performance analysis to improve evaluation robustness.
Qualifications
Minimum 5 years of software development experience with core technologies including Python (FastAPI), JavaScript/TypeScript (React), Docker, PostgreSQL, Kafka, and Redis. Proven experience in writing functional and integration tests. Strong analytical skills to understand AI model limitations and design challenging evaluation scenarios. English language proficiency at B2 level or higher.