T3 labeler
Video
RLHFProject Scope: The primary objective of the Atlas project is to accelerate and refine the training of multi-modal foundational models—specifically Large Language Models (LLMs) and Vision-Language Models (VLMs). The scope encompasses processing massive streams of unstructured data across text, code, image, and video formats. It bridges the gap between raw data collection and production-ready AI capabilities. The project scales across distinct operational tiers (Tier 1 through Tier 3 Senior Review) to handle tasks ranging from baseline data tagging to complex behavioral and cognitive alignment via human feedback loops. Tasks Performed & Label Types Tasks on Atlas are categorized by data modality and the cognitive complexity required for annotation: A. Text & Reasoning Modalities (LLM Alignment) RLHF & DPO Pair-wise Ranking: Reviewing a single prompt alongside two distinct model-generated responses (Response A vs. Response B). Reviewers evaluate, rank, and explicitly annotate which model output is superior based on specific criteria like style, compliance, and helpfulness. Safety & Red-Teaming Audits: Identifying and categorizing subtle algorithmic vulnerabilities, implicit biases, hate speech, or adversarial jailbreak vectors across text and code. Violations are flagged against exact compliance taxonomies (e.g., illicit_behavior, cyber_risk). Code Critique & Reward Modeling: Reviewing code generation outputs for syntax accuracy, runtime complexity, and security vulnerabilities, accompanied by programmatic unit test validations. B. Video & Computer Vision Modalities (VLM Fine-Tuning) Atomic Action & Egocentric Labeling: Frame-by-frame annotation of first-person (egocentric) or third-person video assets. Reviewers explicitly map human-object interactions (e.g., separating continuous movements into discrete event phrases using strict linguistic structures like action verbs: "Grab the cup, place it on the counter"). Dense Segment Splitting: Taking raw, unedited recording blocks and partitioning them into precise episodic timeline slices, allowing the vision model to understand chronological tasks and event transitions. Quality Measures Adhered ToTo maintain the integrity of datasets destined for frontier AI models, the project enforces a strict, multi-layered quality control ecosystem:SLA & Latency Constraints: High-priority queues operate under rigorous Service Level Agreements (SLAs)—often requiring critical red-teaming or verification tasks to achieve specific recall targets within live 4-to-24-hour processing windows. Inter-Annotator Agreement (IAA): Tasks utilize blind multi-reading. High-tier ranking and preference datasets mandate a Pairwise Consensus Agreement Rate of $\ge 97\%$ across independent data reviewers to minimize subjectivity. Strict Syntax & Formatting Rules: Annotation protocols explicitly penalize formatting variations. In dense video labeling, for instance, strict syntax is enforced (such as using precise active verbs rather than gerunds like "holding", separating actions explicitly with commas, and entirely banning trailing periods)