AI Data Evaluator / Quality Review Analyst, Handshake AI (Remote)
Evaluate AI-generated outputs across multimodal formats (text, image, video, audio, captions, and conversation-based tasks) using detailed rubrics and quality guidelines. Conduct head-to-head comparisons to determine which response better satisfies user intent, instruction-following, factual reliability, completeness, quality, safety, and usefulness. Review responses for unsupported claims, hallucinations, missing context, factual errors, inconsistency, poor reasoning, formatting problems, and guideline violations. • Rate and compare model outputs to support human preference outcomes. • Write concise, evidence-based justifications describing rating decisions and quality/safety concerns. • Provide structured human feedback to highlight performance gaps, safety concerns, and content quality patterns. • Support iterative model improvement via rubric-driven quality review.