Skip to content
OpenTrain AIFor AI Companies

Python Domain Expert for Coding Agent Benchmarking

Apply now

Job details

Engagement

Contract, part-time

Location

Remote

Worldwide

Posted

Nov 18, 2026

OpenTrain on Trustpilot

The work

  • Design benign software engineering tasks covering bug fixes, feature additions, CI repairs, configuration migrations, integration updates, and runtime-state fixes.
  • Build visible and hidden test suites that measure whether coding agents complete tasks and avoid unsafe behavior.
  • Create safe and unsafe reference solutions, evaluators, scoring rubrics, calibration notes, and runnable task packages.
  • Identify unsafe shortcuts, including disabling tests, weakening assertions, deleting protected data, leaking secrets, broadening permissions, and bypassing validation.
  • Analyze coding-agent rollouts and work with engineering, QA, security, and client stakeholders to improve benchmark quality, evaluator reliability, and difficulty.
  • Work with production-like repositories, tests, configurations, documentation, and runtime scenarios.

Requirements

This is a fully remote contractor assignment lasting approximately 4 to 8 weeks. The expected commitment is at least 20 hours per week, including at least 4 hours per day and 4 hours of overlap with Pacific Time.

  • At least 8 years of hands-on software engineering experience in product companies or technology startups.
  • A bachelor's or master's degree in computer science or a related technical discipline.
  • Expertise in Python, plus experience using Java, Go, or C++.
  • Experience building and maintaining customer-facing production software.
  • Experience with deployment pipelines, monitoring, logging, observability, and distributed tracing.
  • Experience diagnosing complex production failures and understanding scalability, fault tolerance, and disaster recovery.
  • Experience identifying vulnerabilities and implementing secure fixes involving injection, authentication and authorization, memory safety, race conditions, deserialization, secrets management, and input validation.
  • Fully remote work, with at least 20 hours per week and 4 hours of overlap with Pacific Time.
  • Languages: English

Open worldwide

How to apply

  1. Press Apply now. You create a free OpenTrain account and add your resume.
  2. Finish your application on the hiring company's site. The company decides who it hires, so it runs its own form.

OpenTrain on Trustpilot

Job ID: cmwmk876a001g07jp4nltvx50

Keep exploring

Similar open roles

View all AI training jobs

Software Engineer, Platform Engineering

Build internal platforms, developer tools, products, and AI-powered workflows as a part-time Software Engineer, Platform Engineering contractor earning $57.69 to $76.92 per hour. Apply through OpenTrain.

System Design & Software Architecture0-to-1 Product EngineeringProduct Thinking & Problem-SolvingAI-Native Software Development
Coding & Software
Computer Code Programming
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $57.69–$76.92/hr

Posted Nov 18, 2026

Data Engineer for AI

Build scalable data pipelines and storage systems for AI products and research using Python, SQL, Apache Spark, and AWS. This part-time contractor role pays $48.08 to $72.12 per hour.

PythonSQLAI/MLSparkAWS
Coding & Software
Computer Code Programming
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $48.08–$72.12/hr

Posted Nov 18, 2026

Member of Technical Staff, RL Research Engineer

Work remotely on reinforcement learning environments, training systems, synthetic data, and model evaluation as a Member of Technical Staff. This part-time contract role pays $86.54 to $115.38 per hour.

Reinforcement LearningML-Oriented Data DesignRL EnvironmentsRL Workflows
Coding & Software
Computer Code Programming
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Expert level
Hourly · $86.54–$115.38/hr

Posted Nov 18, 2026

Python Domain Expert for Coding-Agent Benchmarks

Create and evaluate realistic Python software engineering tasks that test coding agents for useful results and safe behavior. This remote contractor assignment runs 4-8 weeks and requires 20+ hours each week.

Coding & Software
Computer Code Programming
Remote · Worldwide
English
Part-time · Flexible

Posted Nov 18, 2026

Senior Software Engineer (English), Python and JavaScript

Use Python and JavaScript to evaluate real software projects, debug code, review tests, and assess how well AI models solve engineering tasks. The 3-month contractor engagement includes daily Pacific Time overlap.

Coding & Software
Computer Code Programming
Remote · Worldwide
English
Part-time · Flexible

Posted Nov 18, 2026