Independent Generalist Expert (SxS & Instruction Evaluation) - MERCOR (Remote)
Performed instruction and scenario evaluation work by annotating and categorizing data under detailed guidelines and rubrics. Flagged dataset and output issues such as inconsistencies, ambiguities, and errors to improve dataset and system quality. Crafted tasks that tested an AI agent’s ability to reason, filter, and prioritize, then evaluated agent performance against guided hints. • Applied predefined rubrics and taxonomies for structured outputs • Detected and reported inconsistencies, ambiguities, and errors • Authored agent challenge tasks emphasizing reasoning and prioritization • Evaluated agent behavior by running scenarios and providing structured guidance