Legal AI Evaluation and Rubric Design Expert

You will create challenging legal and compliance scenarios, evaluate AI-generated analysis, and turn professional legal judgment into clear scoring criteria. Your work will help assess whether responses identify important issues, apply the right rules, recognize exceptions, separate facts from assumptions, address risks, and explain uncertainty responsibly.

The work may cover contracts, privacy, financial regulation, employment matters, corporate compliance, regulatory interpretation, policy analysis, and related legal specialties.

• Write realistic legal and compliance questions that require expert analysis and professional judgment.
• Design rubrics covering issues, standards, reasoning steps, completeness, exceptions, jurisdiction, risk, and critical errors.
• Rate AI responses for accuracy, relevance, completeness, reasoning quality, and professional rigor.
• Find unsupported conclusions, incorrect rule application, missing exceptions, faulty assumptions, and poor risk prioritization.
• Review peer rubrics for ambiguity, repetition, excessive subjectivity, hidden assumptions, and inaccurate legal expectations.
• Create reference analyses, answer frameworks, critiques, preference data, and reference-answer pairs for reinforcement learning and supervised fine-tuning.
• Keep clear distinctions between factual analysis, legal interpretation, professional judgment, and uncertainty.

What it pays and takes

This is a part-time contractor role for a US-based specialist. The listing is marked entry level, but the role requires substantial legal, regulatory, or compliance experience.

• Pay: $120 per hour.
• Time: 20+ hours per week.
• Location: United States.
• Language: Professional English proficiency at C1 level or higher.
• Education: JD, LLB, LLM, law degree, compliance qualification, or equivalent professional background.
• Experience: At least three years in law, regulatory compliance, legal research, policy, risk, privacy, financial compliance, contracts, or a related field.
• Expertise: A clearly defined legal, regulatory, or compliance domain.
• Skills: Identify controlling issues, assess rule application, weigh material risks, and recognize legitimate alternative interpretations.
• Rubric writing: Define required issues, standards, exceptions, omissions, and reasoning quality.
• Attention to detail: Review terminology, authority, jurisdiction, factual assumptions, materiality, and conclusions carefully.
• Helpful experience includes legal education, examination writing, peer review, legal editing, quality assurance, policy drafting, compliance testing, supervisory review, AI evaluation, legal AI, data annotation, reinforcement learning from human feedback, model benchmarking, or rubric-based scoring.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including writing and rating model responses, evaluating reasoning, and preparing structured feedback. People with legal and compliance expertise are needed to judge whether AI outputs apply rules accurately, handle exceptions, and communicate risk responsibly.