AI Agent Evaluation Scenario Writer

About OpenTrain

OpenTrain AI is the hiring and contracting organization for this role. OpenTrain is the #1 platform for finding and building careers in AI training and data labeling, helping people discover opportunities to teach and evaluate artificial intelligence.

This contractor position is fully remote and open worldwide. The role requires a reliable laptop, stable internet connection, and consistent availability.

• Contractor and part-time engagement
• Worldwide remote opportunity
• 20+ hours per week
• Compensation of $18 to $24 per hour

About AI Agent Evaluation Work

AI training is the human side of building modern artificial intelligence. People create examples, review model behavior, and provide structured feedback that helps AI systems become more accurate, useful, and reliable.

In this role, you will help evaluate LLM-based agents by designing realistic tasks, defining expected behavior, and documenting how outputs should be judged. Your work will support the testing and refinement of systems that perform complex, real-world activities.

• Work directly with AI-generated outputs and agent logs
• Turn real-world tasks into structured, reusable evaluations
• Help improve the clarity and coverage of AI testing frameworks

The Role

OpenTrain AI is seeking an analytical Evaluation Scenario Writer and AI Agent Testing Specialist with strong QA-style thinking and excellent written English. You will create reproducible evaluation scenarios for LLM-based agents and establish the standards used to assess their performance.

This intermediate-level role is a strong fit for someone with experience in software testing, QA, test case design, data analysis, or NLP annotation. You should be comfortable working with structured formats such as JSON or YAML and reading or editing simple Python and JavaScript scripts.

• Subject matter: LLM agent testing and evaluation design
• Experience level: Intermediate
• Language: English
• Data type: Text
• Labeling focus: Evaluation rating

What You'll Do

You will design realistic, reusable scenarios that simulate real-world tasks for LLM-based agents. Each scenario should clearly communicate the task, expected outcomes, acceptable variations, and conditions for evaluating agent performance.

You will also review agent outputs, identify gaps in scenario coverage, and refine evaluation materials for clarity and consistency. The work involves collaborating with developers and other contributors to test and improve evaluation frameworks.

• Design structured evaluation scenarios for LLM-based agents
• Define the golden path and gold-standard agent behavior
• Annotate task steps and expected outputs
• Document edge cases and scoring logic
• Specify acceptable variations in agent behavior
• Review agent outputs and agent logs
• Iterate on scenarios to improve clarity and coverage
• Collaborate with developers and other contributors

Requirements

Applicants should be able to follow complex guidelines accurately, switch between topics quickly, and produce documentation that is clear, precise, and unambiguous. A background in QA, software testing, data analysis, or NLP annotation is strongly preferred.

A bachelor's and/or master's degree in computer science, software engineering, data science or analytics, artificial intelligence or machine learning, computational linguistics or NLP, information systems, or a related field is required. Basic working experience with Python and JavaScript is also required.

• Bachelor's and/or master's degree in a relevant technical or analytical field
• Prior experience in QA, software testing, test case design, data analysis, or NLP annotation
• Ability to design reproducible test scenarios with strong coverage and edge cases
• Comfort using JSON and/or YAML to describe scenarios
• Ability to define gold-standard behavior, acceptable variations, and scoring logic
• Basic Python and JavaScript experience for reading or editing simple scripts
• Strong written English skills
• Comfort working with AI-generated outputs, agent logs, and prompt-based behaviors
• Reliable laptop, stable internet connection, and consistent availability

Why Join AI Training Work

AI training and data labeling are growing fields that offer remote ways to contribute to cutting-edge technology. Contributors apply analytical, language, and technical skills to the examples and evaluations used to shape how AI systems behave.

Through OpenTrain, you can build experience in an industry where human judgment remains essential to testing and improving modern AI. This opportunity offers part-time flexibility while engaging with advanced LLM evaluation work.

• Fully remote work from anywhere worldwide
• Part-time schedule of 20+ hours per week
• Hands-on experience with LLM agent evaluation
• Opportunity to apply QA, testing, writing, and technical skills