Senior Data Annotation Specialist (Remote Contract) | Appen
Served as a Senior Data Annotation Specialist performing RLHF labeling and evaluations for LLM training. Worked on ranking model responses, writing ideal responses, and flagging harmful or policy-violating content according to client safety rubrics. Ensured consistent scoring outcomes through calibration and guideline updates while meeting high throughput requirements. • Labeled prompt-response pairs for helpfulness, harmlessness, honesty, and factual accuracy. • Performed response ranking, rating, and policy-violation identification. • Maintained inter-annotator agreement (IAA) of 96%+ across projects. • Conducted weekly calibration to apply updated RLHF guidelines without productivity drops.