CYU
Text
Text SummarizationProject Scope: Code-named Navy Zinc, this project evaluates the "Catch You Up" (CYU) feature. The feature uses a cluster algorithm to group three semantic-related webpages into a "Cluster" and utilizes an LLM to generate a concise summary point (up to 3 words) to help users organize and relocate websites. The evaluation focuses on ensuring these summaries are correct, grounded, safe and relevant. Evaluation Workflow (5 Steps): Review Input: Read the website titles and feature text to identify the shared core idea using a digital notepad. Skip ungradable tasks (e.g., gibberish or foreign language). Evaluate Input: Assess the cluster's safety/harmfulness, flag structural irregularities and determine if the 3 websites share a consistent core idea. Evaluate Responses: Judge the LLM summary based on comprehensiveness (capturing the core theme and entities), groundedness (absence of hallucinations/inaccuracies) and overall satisfaction. Pairwise Comparison: Compare and rank two model responses side-by-side. Comments: Provide written justification and notes on edge cases. Project Structure & Size: Each evaluation task strictly consists of 1 cluster containing 3 websites. Each website provides 2 text elements (Website Title and Feature Text). The generated output is highly constrained, with a maximum length of 3 words. Quality Measures: Labelers must penalize summaries that hallucinate ungrounded information or strike a poor balance in precision (i.e being too broad/vague or overly narrow). Conversely, summaries must be rewarded for retaining shared unique entities (like specific names or locations). If a cluster is inconsistent, a broader, valid overarching theme is still considered acceptable.