Structured Data LLM Evaluation Annotator

As a Structured Data LLM Evaluation Annotator, you will help assess how well large language models retrieve, analyze, and reason over structured information. You will review AI-generated text and explain your judgments using clear, specific evidence.

The work includes creating challenging prompts, checking model outputs against datasets and supporting evidence, and documenting errors or gaps. You will follow annotation guidelines and maintain consistent quality across your evaluations.

• Create prompts that test an LLM's ability to work with structured data.
• Evaluate responses for factual accuracy, logical reasoning, completeness, and consistency.
• Identify hallucinations, reasoning gaps, model failures, and edge cases.
• Validate AI outputs against provided datasets and evidence.
• Write concise explanations showing why a response is correct or incorrect.
• Apply annotation guidelines consistently.

What It Pays and Takes

This is a remote contractor assignment for four weeks. Compensation is described as competitive, but the listing does not provide a specific rate or payment unit.

• Commitment: 20, 30, or 40 hours per week.
• Time zone: At least four hours of overlap with Pacific Time.
• Location: Remote and open worldwide.
• Language: Excellent written English.
• Education: A master's degree or higher in any discipline.
• Experience: At least three years of professional, research, or teaching experience.
• Required strengths: Analytical thinking, critical thinking, information validation, clear explanations, and close attention to detail.
• Helpful background: Large language models, generative AI, prompt engineering, AI evaluation, data annotation, model testing, or structured datasets such as CSV files, Excel workbooks, and databases.

How It Works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI Training Work

AI training work uses human-written examples, reviews, and ratings to improve how artificial intelligence systems respond. People with strong subject knowledge and careful reasoning are paid to spot errors, explain judgments, and help make model outputs more accurate.