About OpenTrain
OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control. We make specialized AI training work easier to track, apply for, and grow into a durable freelance career.
About this role
OpenTrain is recruiting a Python Coding-Agent Benchmark Engineer to design and evaluate realistic software engineering tasks used to assess coding agents. You will work with production-like repositories, tests, configurations, documentation, and runtime scenarios. The role combines production engineering, secure software development, benchmark design, and evaluation of coding-agent behavior.
What you will do
Design benign engineering tasks covering bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes. Build visible and hidden test suites that measure task completion and unsafe behavior. Create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, and runnable task packages. Identify unsafe shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, or bypassing validation. Analyze coding-agent rollouts and collaborate with engineering, QA, security, and client stakeholders to improve benchmark quality, evaluator reliability, and difficulty.
Required skills
You should have at least eight years of hands-on software engineering experience in product companies or technology startups, along with a bachelor's or master's degree in computer science or a related technical discipline. Python expertise is mandatory, together with experience using Java, Go, or C++. Relevant experience includes building and maintaining customer-facing production software; working with deployment pipelines, monitoring, logging, observability, and distributed tracing; diagnosing complex production failures; and understanding scalability, fault tolerance, and disaster recovery. You should also be experienced in identifying vulnerabilities and implementing secure fixes involving injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
Schedule and engagement
This is a fully remote contractor assignment lasting approximately 4-8 weeks. The expected commitment is at least 20 hours per week, including at least four hours per day and four hours of overlap with Pacific Time.
Why work with OpenTrain
OpenTrain gives freelancers one place to manage AI training opportunities instead of rebuilding proof of work on every platform. A stronger OpenTrain profile helps you show credible experience, discover roles that match your skills, and turn AI training and data-labeling work into a long-term portfolio.