Skip to content
OpenTrain AIFor AI Companies

Python Domain Expert for Coding-Agent Benchmarks

Apply now

Job details

Engagement

Contract, part-time

Location

Remote

Worldwide

Posted

Nov 18, 2026

OpenTrain on Trustpilot

The work

  • Design benchmark tasks for bug fixes, feature additions, continuous integration repairs, configuration migrations, integration updates, and runtime-state fixes.
  • Define utility requirements that verify whether a coding agent completes engineering work correctly.
  • Set safety requirements covering system constraints, developer intent, data integrity, privacy, permissions, and oversight.
  • Build visible and hidden test suites for task completion and unsafe behavior.
  • Create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, runnable repositories, and reference patches.
  • Analyze coding-agent shortcuts such as disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, and bypassing validation.
  • Review coding-agent rollouts and work with engineering, quality, and security teams to improve benchmark quality and difficulty.

Requirements

  • This is a fully remote contractor assignment lasting 4-8 weeks, with at least 20 hours per week and at least 4 hours of overlap with Pacific Time.
  • The role is open worldwide and requires professional fluency in English.
  • A bachelor's or master's degree in computer science or a related technical discipline, at least 8 years of hands-on software engineering experience, and professional Python experience plus Java, Go, or C++ experience are required.
  • You should have built and maintained production software used by real customers and understand software architecture, debugging, performance optimization, deployment pipelines, monitoring, logging, observability, distributed tracing, high availability, fault tolerance, scalability, and disaster recovery.
  • Practical secure development experience is required, including identifying vulnerabilities and validating fixes involving injection, authentication, authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
  • This is part-time contract work supporting advanced AI projects with technical collaborators.

Open worldwide

How to apply

  1. Press Apply now. You create a free OpenTrain account and add your resume.
  2. Finish your application on the hiring company's site. The company decides who it hires, so it runs its own form.

OpenTrain on Trustpilot

Job ID: cmwmk87bs001i07jp0vncxis6

Keep exploring

Similar open roles

View all AI training jobs

Software Engineer, Platform Engineering

Build internal platforms, developer tools, products, and AI-powered workflows as a part-time Software Engineer, Platform Engineering contractor earning $57.69 to $76.92 per hour. Apply through OpenTrain.

System Design & Software Architecture0-to-1 Product EngineeringProduct Thinking & Problem-SolvingAI-Native Software Development
Coding & Software
Computer Code Programming
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $57.69–$76.92/hr

Posted Nov 18, 2026

Data Engineer for AI

Build scalable data pipelines and storage systems for AI products and research using Python, SQL, Apache Spark, and AWS. This part-time contractor role pays $48.08 to $72.12 per hour.

PythonSQLAI/MLSparkAWS
Coding & Software
Computer Code Programming
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $48.08–$72.12/hr

Posted Nov 18, 2026

Member of Technical Staff, RL Research Engineer

Work remotely on reinforcement learning environments, training systems, synthetic data, and model evaluation as a Member of Technical Staff. This part-time contract role pays $86.54 to $115.38 per hour.

Reinforcement LearningML-Oriented Data DesignRL EnvironmentsRL Workflows
Coding & Software
Computer Code Programming
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Expert level
Hourly · $86.54–$115.38/hr

Posted Nov 18, 2026

Python Domain Expert for Coding Agent Benchmarking

Work remotely as a Python domain expert designing and evaluating coding-agent benchmark tasks, tests, and safety checks. This 4 to 8 week contractor assignment requires at least 20 hours per week and 4 hours of Pacific Time overlap.

Coding & Software
Computer Code Programming
Remote · Worldwide
English
Part-time · Flexible

Posted Nov 18, 2026

Senior Software Engineer (English), Python and JavaScript

Use Python and JavaScript to evaluate real software projects, debug code, review tests, and assess how well AI models solve engineering tasks. The 3-month contractor engagement includes daily Pacific Time overlap.

Coding & Software
Computer Code Programming
Remote · Worldwide
English
Part-time · Flexible

Posted Nov 18, 2026