Skip to content
OpenTrain AIFor AI Companies

Evaluation Scenario Writer for AI Agents

Apply now

Job details

Pay

$18–$24 USD/hr

Engagement

Contract, part-time

Location

Remote

Worldwide

Posted

Jan 13, 2026

OpenTrain on Trustpilot

The work

You will create realistic, reusable evaluation scenarios for AI agents that perform real-world tasks. You will define the correct path, acceptable agent behavior, expected outputs, edge cases, and scoring rules.

You will also review agent outputs, improve scenarios for clarity and coverage, and work with developers and other contributors to test and refine evaluation frameworks.

  • Design structured evaluation scenarios for LLM-based agents.
  • Define gold-standard behavior and acceptable alternatives.
  • Annotate task steps, expected outputs, edge cases, and scoring logic.
  • Review agent responses and revise scenarios to improve coverage.
  • Use structured formats such as JSON and YAML.

Requirements

This is an intermediate-level, part-time contractor role for someone who combines careful writing with software testing or quality assurance thinking.

  • Time: 20 or more hours per week.
  • Language: Strong written English.
  • Required: Basic Python and JavaScript experience.
  • Required: Analytical thinking and comfort designing structured evaluations.
  • Experience: Intermediate level

Open worldwide

Preferred

  • Experience in software testing, QA, data analysis, or NLP annotation.

How to apply

  1. Press Apply now. You create a free account and add your resume.
  2. Complete your application for this role on OpenTrain.

OpenTrain on Trustpilot

Job ID: 1768304977234x203523473669357570

Keep exploring

Similar open roles

View all AI training jobs

AI Safety Annotation Reviewer

Review AI safety evaluations for policy alignment, reasoning quality, scoring accuracy, and annotation completeness. This remote role is open worldwide and requires professional English fluency with a 20+ hour weekly commitment.

Generative AI & RLHF
Text
Remote · Worldwide
English
Part-time · Flexible
Intermediate level

Posted Nov 14, 2026

Physics Expert (Gravity and Holography)

Apply advanced physics knowledge to solve, review, and explain challenging problems in gravity, cosmology, and holography. This part-time contract role pays $100 to $170 per hour.

General RelativityCosmologyHolographyQuantum Field TheoryString TheoryLaTeX
Generative AI & RLHF
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $100–$170/hr

Posted Nov 14, 2026

AI Training and Language Specialist (Arabic)

Help train AI in Arabic by rating model responses and writing prompts in Modern Standard and colloquial Arabic. This expert contract role pays $15 to $50 per hour and is open worldwide.

Generative AI & RLHF
Text
Remote · Worldwide
Arabic
Flexible hours
Expert level
Hourly · $15–$50/hr

Posted Nov 14, 2026

Physics Expert (Quantum Optics and Atomic Physics)

Review and improve research-level AI training benchmarks in quantum optics and atomic physics. This remote contractor role offers flexible scheduling and pays $100 to $170 per hour.

Quantum OpticsAtomic PhysicsQuantum SensingOptical LatticesLaser PhysicsLaTeX
Generative AI & RLHF
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $100–$170/hr

Posted Nov 14, 2026

Physics Expert (PhD / Postdoc)

Apply advanced physics expertise to solve, audit, and adjudicate research-level benchmark problems remotely and asynchronously for $100 to $170 per hour. A PhD or postdoctoral background and LaTeX proficiency are required.

PhysicsTheoretical PhysicsMathematical PhysicsQuantum PhysicsLaTeX
Generative AI & RLHF
Document
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $100–$170/hr

Posted Nov 14, 2026