Evaluate JSON-formatted AI model outputs against written task instructions and scoring rubrics in a short-term, US-only remote project; pay ranges $50–$175/hr with a default 40-hour weekly commitment and 20+ hours/week availability required.
Generative AI & RLHF
Remote Hourly · $50–$175/hr
$50–$175/hr
Compensation
1 country
Eligibility
Intermediate
Experience
Jul 25, 2026
Posted
Open to applicants in
United States
Interested in this role?
Create a free OpenTrain account and apply in minutes.
OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. We help freelancers discover projects, build a single portfolio of verified AI training work, and grow a durable freelance career in this fast-growing field.
We are the hiring organization for this role. Working with OpenTrain connects your work history to future AI-training opportunities and helps you show credible experience across projects.
About AI training work
AI training (also called data labeling or human feedback work) is the human side of building modern AI systems. Contributors prepare and evaluate examples—like annotated data, transcriptions, or graded model outputs—that models learn from.
This role sits at the intersection of quality assurance and evaluation: your judgments directly influence how models are scored and improved.
The role
You will act as a JSON Rubric Grader: read task instructions and scoring rubrics, inspect JSON-formatted model outputs, and apply the rubric to decide whether each output meets the required standard.
This is detail-oriented, judgment-based work with fast turnaround expectations. The project is short-term and requires careful, consistent evaluations completed without AI assistance.
What you'll do
Read written task descriptions and detailed scoring rubrics carefully before grading.
Parse and review JSON-based model outputs to check structure, fields, and content against rubric criteria.
Apply rubric rules consistently to make binary or scaled quality judgments (evaluation ratings).
Record scores and any required annotations or comments in the provided system.
Work quickly while maintaining high accuracy and consistency; meet rapid turnaround deadlines.
Complete assignments without using AI tools to assist your judgments.
Requirements
Comfortable reading and interpreting JSON files and common JSON structures.
Experience applying written rubrics, grading criteria, or evaluation guidelines.
Strong attention to detail, consistent judgment, and the ability to follow instructions precisely.
Access to a desktop or laptop computer with a reliable internet connection.
US-based eligibility is required for this role.
Compensation & schedule
Pay: $50 to $175 per hour (USD). You will be engaged as a contractor and this role is part-time/contractor by employment type.
Time commitment: the listing shows 20+ hours/week availability and the project indicates a default commitment of 40 hours per week. The project is short-term and expects rapid turnaround on tasks.
Who should apply
Apply if you have prior rubric-based evaluation experience, strong JSON literacy, and prefer remote, flexible contract work that demands accuracy and speed.
This role is a good fit for evaluators, QA professionals, testers, or anyone experienced with structured data review and standardized grading processes.
How it works / next steps
If selected, you'll receive task instructions, rubrics, and access to the grading workflow. Complete graded assignments to the standards in the rubric and submit within the expected turnaround windows.
Building verified work on OpenTrain helps you show experience for future AI training roles; strong performance may lead to more projects through the platform.
Join OpenTrain to build evaluation tasks, prompts, and clear grading rubrics that measure AI agents on practical workflows; part-time remote work at $20–$35/hr for experienced writers and rubric designers. Work 20+ hours/week producing concise, structured reports and evolving benchmarks.
Design grading rubrics and score AI and human consulting deliverables with evidence-based written justifications. Remote contractor role, 20+ hours/week, pay $150–$220/hr working for OpenTrain AI on evaluation and quality tasks.
Remote contract role reviewing AI-generated Rust code, writing model solutions, and providing expert feedback at $60/hr for 20+ hours/week. Join OpenTrain AI to shape how Rust-capable models reason about ownership, lifetimes, concurrency, and idiomatic style.
United States 
