Skip to content
OpenTrain AIFor AI Companies

AI Agent Evaluator

Apply now

Job details

Pay

$25–$45 USD/hr

Engagement

Contract, part-time

Location

Remote

United States only

Posted

Nov 16, 2026

OpenTrain on Trustpilot

The work

About OpenTrain
OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control. We make specialized AI training work easier to track, apply for, and grow into a durable freelance career.

About this role
OpenTrain is recruiting for an AI Agent Evaluator to assess how two AI agents handle realistic errands and digital tasks from a user's perspective. You will test agents side by side on a smartphone, make decisions at important checkpoints, capture screen recordings, and evaluate the quality and reliability of each experience. Your observations will help improve AI agents that assist with everyday activities such as payments, bookings, shopping, communications, and forms.

What you will do
You will run realistic scenarios involving bills, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms. You will approve or stop an AI agent at key steps, screen record every attempt using a smartphone, and score completion, quality, control issues, satisfaction, and time for each run. You will also write clear feedback describing what worked, what failed, and how the interaction felt. Some tests may require using personal AI accounts and making real purchases with your own card, with reimbursement afterward.

Required skills
This role requires U.S. residence, a bachelor's degree, two to three years of experience preferably in engineering, business management, or office operations, regular personal use of AI tools, strong attention to detail, and the ability to follow instructions precisely. You should have clear written communication skills, be comfortable documenting each step, and have a smartphone capable of screen recording with an active personal AI account.

Pay and schedule
This remote contractor role pays $25‑$45 per hour. Work is completed on a flexible schedule, and each task run takes approximately 1.5 hours, including evidence capture and scoring.

Helpful background
Generalists are welcome, while experience in a relevant domain can be helpful. Coding experience and a formal AI background are not required. The role is designed for independent contractors who can complete structured mobile testing and provide thoughtful, reliable evaluations.

Why work with OpenTrain
OpenTrain gives freelancers one place to manage AI training opportunities instead of rebuilding proof of work on every platform. A stronger OpenTrain profile helps you show credible experience, discover roles that match your skills, and turn AI training and data-labeling work into a long-term portfolio.

You apply here in about two minutes and finish on the hiring platform. The next step is a short AI interview (technical roles include a coding challenge). Passing it is how you qualify for paid tasks, so take it soon after you are invited.

The hiring platform's application for this job asks for your resume, your LinkedIn profile, your phone number, your location and answers to 8 screening questions.

Requirements

  • Languages: English
  • Skills: ChatGPT, Muse, AI agents, Attention to Detail, Following Instructions, Screen Recording, Dots, Mobile

Open to applicants in

  • United States

How to apply

  1. Apply here on OpenTrain. You create a free account, and we send you to the hiring platform.
  2. Complete your application on the hiring platform.

OpenTrain on Trustpilot

Job ID: cmwjfx0rk004108i3klmd5o92

Keep exploring

Similar open roles

View all AI training jobs

Applied AI Agent Evaluation Research Engineer

About OpenTrain. OpenTrain is the unified marketplace where AI training freelancers find and consolidate opportunities across platforms, build an AI training portfolio, and develop a durable freelance career. We help ca…

AI Agent Evaluation & Benchmark DesignPython & Research EngineeringAgent Failure Analysis & ExperimentationLLM & Agentic Systems
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Flexible hours
Hourly · $96.15–$125/hr

Posted Nov 17, 2026

Civilian Intelligence Agency AI Evaluation Expert

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

intelligence analysisprogram managementorganizational decision-makingcybersecuritystrategic planningscenario evaluation
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Mid-Senior level
Hourly · $80–$170/hr

Posted Oct 30, 2026

AI Agent Tester for Real-World Task Evaluation

About OpenTrain. OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

ChatGPTMuseAI agentsAttention to DetailFollowing InstructionsScreen Recording
Video
Remote · United States
English
Part-time · Flexible
Hourly · $25–$45/hr

Posted Oct 24, 2026

AI Agent Tester

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

ChatGPTMuseAI agentsAttention to DetailFollowing InstructionsScreen Recording
Video
Remote · Austria, Belgium, Switzerland +9 more
English
Part-time · Flexible
Hourly · $25–$45/hr

Posted Nov 16, 2026

Engineering Expert, Simulation Design

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

Classical MechanicsElectrical Networks
Code
Remote · Worldwide
English
Part-time · Flexible
Intermediate level

Posted Nov 11, 2026