Senior Content Analyst & Quality Evaluator (AI Response Evaluator)
TextEvaluated 50–80 long-form AI-generated texts per week using structured rubrics across coherence, factual accuracy, logical completeness, and implicit reasoning gaps. Produced detailed written rationales for each assessment and identified systematic model failure patterns related to counterfactual claims. Translated quality criteria into reproducible decision trees and authored an internal evaluation handbook used by new evaluators. • Applied calibrated scoring with inter-rater reliability consistently above 92% across 14 evaluators. • Authored internal evaluation handbook (38 pages) documenting rubric-driven decision processes. • Contributed methodology notes to inform client NLP fine-tuning based on recurring error patterns. • Annotated and documented assessments independently without using AI drafting tools to meet data integrity requirements.