LLM Response Quality Evaluation & Preference Ranking for Conversational AI Training
Text
RLHFWorked on a preference ranking project designed to improve the response quality of a large language model through human feedback. The work involved reviewing pairs and triplets of AI-generated responses to the same prompt and ranking them based on accuracy, coherence, tone appropriateness, and how well each response actually addressed what the user was asking. A significant portion of the prompts covered professional writing, business communication, and domain-specific knowledge tasks, which is where my background gave me a real edge over general annotators who were largely guessing at quality in those areas. The project ran across roughly 1,800 evaluated response sets. Each ranking submission included a written rationale explaining the preference decision, not just a numeric score. The team leads specifically flagged that annotators who could articulate why one response was better than another were far more valuable to the pipeline than those who ranked without explanation, because the written reasoning fed directly into a secondary review layer. My rationale acceptance rate across all submissions was 91%, meaning the senior reviewers agreed with both my rankings and my reasoning in over nine out of ten cases. One area I contributed beyond the core task was flagging a consistent pattern where the model performed well on surface-level phrasing but gave structurally weak answers to multi-part questions, treating them as single questions and ignoring secondary clauses. I documented roughly 60 examples of this across different prompt categories and submitted them as a structured observation report. The project lead confirmed this was incorporated into the next prompt design cycle as a targeted evaluation category.