Skip to content
OpenTrain AIFor AI Companies

Python Coding-Agent Benchmark Expert

Apply now

Job details

Engagement

Contract, part-time

Location

Remote

Worldwide

Posted

Nov 18, 2026

OpenTrain on Trustpilot

The work

About OpenTrain
OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control. We make specialized AI training work easier to track, apply for, and grow into a durable freelance career.

About this role
OpenTrain is recruiting for a Python Coding-Agent Benchmark Engineer to create and evaluate realistic software engineering tasks for AI coding agents. You will work with production-like repositories, tests, configurations, documentation, and runtime scenarios to measure useful task completion and adherence to safety and alignment requirements. The role combines production engineering, secure software development, benchmark design, evaluator construction, and analysis of coding-agent rollouts.

What you will do
You will design benchmark tasks involving bug fixes, feature additions, continuous integration repairs, configuration migrations, integration updates, and runtime-state fixes. You will define utility requirements that verify whether an agent completes engineering work correctly, establish safety requirements covering system constraints, developer intent, data integrity, privacy, permissions, and oversight, and build visible and hidden test suites for task completion and unsafe behavior. You will create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, runnable repositories, and reference patches. You will analyze shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, and bypassing validation. You will also review coding-agent rollouts and collaborate with engineering, quality, and security teams to improve benchmark quality and difficulty.

Required skills
You should have a bachelor's or master's degree in computer science or a related technical discipline, at least eight years of hands-on software engineering experience, and professional Python experience alongside Java, Go, or C++ experience. You should have built and maintained production software used by real customers and understand software architecture, debugging, performance optimization, deployment pipelines, monitoring, logging, observability, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery. Practical secure development experience is required, including identifying vulnerabilities and validating fixes involving injection, authentication, authorization, memory safety, race conditions, deserialization, secrets management, and input validation.

Work arrangement
This is a fully remote contractor assignment expected to last 4-8 weeks. The role requires at least 20 hours per week, including at least 4 hours of overlap with Pacific Time, and supports advanced AI projects with technical collaborators.

Why work with OpenTrain
OpenTrain gives freelancers one place to manage AI training opportunities instead of rebuilding proof of work on every platform. A stronger OpenTrain profile helps you show credible experience, discover roles that match your skills, and turn AI training and data-labeling work into a long-term portfolio.

Requirements

  • Languages: English

Open worldwide

How to apply

  1. Apply here on OpenTrain. You create a free account, and we send you to the hiring platform.
  2. Complete your application on the hiring platform.

OpenTrain on Trustpilot

Job ID: cmwmk87bs001i07jp0vncxis6

Keep exploring

Similar open roles

View all AI training jobs

Python Domain Expert - Coding Agent Benchmarking

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

Python
Code
Remote · Worldwide
English
Part-time · Flexible

Posted Nov 18, 2026

Coding Agent Evaluation Researcher

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

LLMsCoding EvaluationAI EvaluationML Systems
Code
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $86.54–$125/hr

Posted Nov 18, 2026

Coding Agent Evaluation Researcher

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

LLMsCoding EvaluationAI EvaluationML Systems
Code
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Flexible hours
Hourly · $96.15–$125/hr

Posted Nov 18, 2026

Applied AI Agent Evaluation Research Engineer

About OpenTrain. OpenTrain is the unified marketplace where AI training freelancers find and consolidate opportunities across platforms, build an AI training portfolio, and develop a durable freelance career. We help ca…

AI Agent Evaluation & Benchmark DesignPython & Research EngineeringAgent Failure Analysis & ExperimentationLLM & Agentic Systems
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Flexible hours
Hourly · $96.15–$125/hr

Posted Nov 17, 2026

Computational Biology AI Benchmarking Expert

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

computational chemistrymolecular simulationspythongaussiangromacslammps
Code
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Mid-Senior level
Hourly · $45–$90/hr

Posted Nov 9, 2026