Skip to content
OpenTrain AIFor AI Companies

AI Agent Evaluator

Apply now

Job details

Pay

$25–$45 USD/hr

Engagement

Contract, part-time

Location

Remote

United States only

Posted

Nov 16, 2026

OpenTrain on Trustpilot

The work

You will evaluate how two AI agents handle everyday errands and digital tasks from a user's point of view. You will test them side by side on a smartphone, make decisions at key checkpoints, and document the results.

You will provide evidence and feedback that helps improve how AI agents handle practical activities such as payments, bookings, shopping, communication, and forms.

  • Run scenarios involving bills, returns, email follow-ups, bookings, shopping, groceries, travel, family logistics, outings, and forms.
  • Approve or stop an AI agent at important steps during each task.
  • Capture a screen recording of every attempt with a smartphone.
  • Score completion, quality, control issues, satisfaction, and time for each run.
  • Write clear feedback about what worked, what failed, and how the interaction felt.
  • Some tests may involve personal AI accounts and real purchases with your own card. Reimbursement is provided afterward.

Requirements

This is a remote, part-time contractor role for people who can complete structured mobile testing and provide careful, reliable evaluations.

  • Schedule: Flexible; each task run takes about 1.5 hours, including evidence capture and scoring.
  • Language: Professional fluency in English.
  • Education: Bachelor's degree required.
  • Experience: Two to three years of experience, preferably in engineering, business management, or office operations.
  • Skills: Regular personal use of AI tools, strong attention to detail, precise instruction following, and clear written communication.
  • Equipment: A smartphone that can record its screen and an active personal AI account.
  • Coding experience and a formal AI background are not required. Generalists are welcome, and relevant subject experience can help.
  • Skills: ChatGPT, Muse, AI agents, Attention to Detail, Following Instructions, Screen Recording, Dots, Mobile

Open to applicants in

  • United States

How to apply

  1. Press Apply now. You create a free OpenTrain account and add your resume.
  2. Finish your application on the hiring company's site. The company decides who it hires, so it runs its own form. It asks for your resume again, your LinkedIn profile, your phone number, your location and answers to 8 screening questions.
  3. Take a short AI interview.

OpenTrain on Trustpilot

Job ID: cmwjfx0rk004108i3klmd5o92

Keep exploring

Similar open roles

View all AI training jobs

Trade Compliance Expert AI Evaluator

Use trade compliance expertise to create expert tasks, assess AI-generated answers, and give clear feedback on reporting, export controls, sanctions, and audits. This ongoing remote role pays up to $80 per hour with no set schedule.

Generative AI & RLHF
Text
Remote · United States
English
Flexible hours
Hourly · $80/hr

Posted Nov 16, 2026

Hardware Expert AI Training Specialist

Create, solve, and evaluate rigorous hardware engineering problems for AI training from anywhere, with a flexible commitment of about 15 hours per week. Pay is $60 to $120 per hour equivalent, paid per task that meets the project specifications.

RTLHardware
Generative AI & RLHF
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Per task · $60–$120/hr equivalent

Posted Nov 16, 2026

STEM Expert for AI Problem Evaluation

Solve scientific and technical problems, use Python for calculations or simulations, and review AI-generated answers for accuracy. This flexible role offers approximately 15 hours per week and pays $60 to $120 per hour equivalent, paid per task that meets project specifications.

Biology
Generative AI & RLHF
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Per task · $60–$120/hr equivalent

Posted Nov 16, 2026

AI Agent Tester

Test two AI agents on realistic tasks such as bookings, shopping, forms, and follow-ups. Record each attempt, score performance, and write clear feedback from a real user’s perspective.

ChatGPTMuseAI agentsAttention to DetailFollowing InstructionsScreen Recording
Generative AI & RLHF
Video
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $25–$45/hr

Posted Nov 16, 2026

Physics Expert (Gravity and Holography)

Apply advanced physics knowledge to solve, review, and explain challenging problems in gravity, cosmology, and holography. This part-time contract role pays $100 to $170 per hour.

General RelativityCosmologyHolographyQuantum Field TheoryString TheoryLaTeX
Generative AI & RLHF
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Part-time · Flexible
Hourly · $100–$170/hr

Posted Nov 14, 2026