Expert LLM Data Labeler — Text Annotation & Review

About OpenTrain

OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. Our mission is to connect skilled contributors with projects that directly shape how cutting-edge AI behaves, while offering flexible, remote work that fits around your life.

This role is posted by OpenTrain AI and will run through the OpenTrain platform — you can create a free profile, discover projects, and apply in minutes.

About AI Training Work

AI training (also called data labeling or human feedback) is the human side of building modern AI. People prepare, annotate, and review examples — like LLM responses — so models learn from high-quality data. Work is often remote, flexible, and accessible: many projects need no prior experience, while specialist tasks pay more for domain knowledge.

This project focuses on text-based English documents and LLM response labeling, giving contributors a direct role in improving conversational and generative systems.

The Role

We need experienced annotators to produce and verify about 500–1,000 rows of LLM response data. Each row contains context, a query, and auto-generated labels that must be confirmed or corrected through a multi-step human review.

Work is contract, part-time, remote, worldwide, and requires a minimum commitment of 20+ hours per week. Compensation is PAY_PER_HOUR at USD $5.00/hour. Labeling will be done in AWS SageMaker.

• Project size: ~500–1,000 labeled rows to produce and verify.
• Schedule: 20+ hours/week (part-time contractor).
• Pay: $5.00 per hour (PAY_PER_HOUR).
• Platform: AWS SageMaker for annotation and review.
• Open to candidates worldwide.

Workflow and Quality Assurance

This project uses a two-step human review process with a third QA pass. Step one: two independent agents review the exact same row and provide labels. Step two: a third agent performs quality assurance and resolves disagreements according to guidelines.

For rows containing technical computer-science or financial content, non-expert reviewers should apply best-effort labeling and defer to the expert reviewer. At least one of the three reviewers assigned to each row must be an expert in assessing technical and/or financial content.

• Two agents independently label the same row (primary reviewers).
• A third agent performs QA and final verification.
• At least 1 of 3 reviewers per row must be an expert for technical/financial items.
• Non-experts do best-effort labeling on specialized content and rely on experts for final judgment.

What You'll Do

Annotate, verify, and rate LLM responses in English using provided guidelines and AWS SageMaker. Labels include classification, entity/NER classification, and evaluation/rating tasks.

Collaborate with other reviewers and follow the two-step review + QA workflow to ensure high-quality, consistent outputs.

• Review rows containing context, queries, and model-generated labels; confirm or correct labels.
• Apply CLASSIFICATION, ENTITY_NER_CLASSIFICATION, and EVALUATION_RATING labels as specified.
• Use AWS SageMaker to enter annotations and track review status.
• Communicate clearly with teammates and escalate ambiguous or technical items to expert reviewers.
• Adhere to project guidelines and weekly time commitments (20+ hrs/week).

Requirements

This is an expert-level role: you must be comfortable making judgment calls on text, following detailed annotation guidelines, and working consistently within the multi-review workflow. Familiarity with the English language is required.

You must be available for 20+ hours per week and able to work as a contract, part-time contributor. Experience using annotation tools (AWS SageMaker preferred) is expected.

• Required: Expert experience level with text annotation or related review work.
• Required: Strong familiarity with English (writing and reading).
• Required: Availability 20+ hours per week; remote worldwide.
• Required: Ability to use AWS SageMaker for labeling and review.
• Preferred: Experience with technical computer-science content and/or financial documents.
• Preferred: Prior work in multi-review QA workflows or RLHF-style evaluations.

Who Should Apply

Apply if you are an experienced annotator or reviewer with excellent English skills and reliable availability for part-time contract work. If you have domain expertise in computer science or finance, you can serve as an expert reviewer on specialized rows.

We welcome contributors worldwide; the role is remote and suited to people who prefer flexible schedules and collaborative quality-focused workflows.

• Ideal for annotators with expert-level judgment on text quality and labeling.
• Great fit for those with domain experience in technical or financial documentation.
• Suitable for contractors who want steady, part-time hours and to influence real AI behavior.

How It Works / How To Apply

Create a free OpenTrain account to build your profile and apply. When you apply, you will be asked about your English proficiency, annotation experience, availability (20+ hrs/week), and any technical or financial expertise.

Qualified applicants will be invited to a short onboarding and test assignment to confirm ability to use AWS SageMaker and follow the two-step review process. If accepted, you'll be assigned rows to label and collaborate with other reviewers.

• Sign up on OpenTrain and complete your profile to apply in minutes.
• Be prepared for a short onboarding test using AWS SageMaker.
• Accepted contractors will receive project guidelines, access to the labeling interface, and work assignments.