Lead AI Evaluator & Data Annotator | Scale AI (Remote)
Conducted RLHF-based evaluation by ranking and rating AI model outputs across safety, accuracy, tone, and helpfulness using structured rubrics. Provided nuanced, reasoned feedback to drive model improvement and maintain high-quality ground truth. Led annotation quality oversight and dispute resolution to reduce disagreement and improve rubric clarity. • Ranked and rated model outputs (safety/helpfulness/accuracy/tone). • Reviewed ground truth submissions and resolved labeling disputes. • Refined annotation guidelines and ontology definitions to reduce inter-annotator disagreement. • Maintained 99.1% accuracy across 200,000+ labeled items for text classification, NER, and document parsing tasks.