About OpenTrain
OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control. We make specialized AI training work easier to track, apply for, and grow into a durable freelance career.
About this role
OpenTrain is recruiting for a STEM Rubric Design Specialist to create expert-level technical questions, evaluate AI-generated solutions, and design detailed evaluation rubrics for frontier model training. In this hourly, remote contractor position, you will apply your deep technical expertise to probe the reasoning boundaries of advanced AI systems and improve their accuracy and rigor. This is a role for experienced scientists, engineers, and mathematicians who enjoy translating complex technical judgment into measurable criteria.
What you will do
- Create expert-level technical questions that test genuine multi-step reasoning and expose weaknesses in advanced AI systems
- Design and write structured evaluation rubrics defining required reasoning, correct methodology, acceptable alternatives, partial-credit criteria, and critical errors
- Evaluate AI-generated solutions for correctness, reasoning quality, completeness, methodology, and technical precision
- Identify hidden reasoning errors, including cases where an answer reaches the correct conclusion via invalid assumptions or flawed logic
- Critique peer rubrics for technical accuracy, clarity, completeness, and scoring consistency
- Develop reference content including expert answers, explanations, and gold-standard evaluations for RL and SFT workflows
Required skills
- Bachelor’s degree or higher in Mathematics, Physics, Chemistry, Engineering, Computer Science, Statistics, Applied Science, or another technical discipline
- Strong professional proficiency in English, minimum C1, with the ability to write precise technical explanations and evaluation criteria
- 3+ years of professional, academic, research, or industry experience in your stated technical domain
- Demonstrated advanced expertise in a clearly defined technical field or specialization
- Strong ability to design detailed evaluation rubrics defining required reasoning, correct methodology, acceptable alternatives, partial-credit criteria, and critical errors
- Ability to distinguish between a correct final answer and an answer supported by valid versus flawed reasoning
- Comfortable reviewing and critiquing other experts’ rubrics for ambiguity, missing criteria, redundancy, or technical inaccuracies
Helpful background
- Experience with exam writing, academic grading, peer review, research review, technical QA, standards development, or assessment design is strongly preferred
- Prior experience with AI evaluation, RLHF, SFT, benchmarking, data annotation, prompt design, or LLM evaluation is preferred