Data labeling experience
Image
Segmentation1. COCO Dataset - Image Object Detection & Segmentation Scope of project Build a large-scale dataset for training AI to detect and segment objects in everyday scenes. Used for computer vision research and products like autonomous driving, robotics, and image search. Specific labeling tasks performed 1. Bounding Box Annotation: Draw rectangular boxes around every instance of 80 object categories - person, car, bicycle, chair, etc. 2. Instance Segmentation: Outline the exact pixel boundary of each object, not just the box. 3. Keypoint Annotation: Mark key points on people for pose estimation - 17 joints like nose, shoulders, knees. 4. Stuff Segmentation: Label amorphous regions like sky, road, grass with a single mask. 5. Captioning: Write 5 short captions describing the scene for image-text training. Project size - 330k+ images total - 2.5M+ labeled instances - 80 object categories + 91 stuff categories - Done over multiple years with hundreds of annotators Quality measures adhered to - Multi-annotator consensus: Each image labeled by 3+ people. Conflicts resolved by senior annotators. - Guideline tests: Annotators must pass a qualification test with >90% accuracy before working on real data. - Inter-annotator Agreement (IAA): Measured using IoU for boxes/masks. Target IAA >0.85. - Spot audits: 10% of work randomly reviewed by QA team. Below-threshold work sent back for re-labeling. - Clear annotation guidelines: 50+ page manual with edge cases for occlusion, truncation, small objects. 2. SQuAD Dataset - Reading Comprehension Q&A Labeling Scope of project Create a benchmark for machine reading comprehension. Goal: can an AI read a passage and extract the correct answer span. Specific labeling tasks performed 1. Question Writing: Annotators read a Wikipedia paragraph and write 3-5 natural questions that can be answered from the text. 2. Answer Span Selection: Highlight the exact start/end character position in the paragraph that answers the question. 3. Unanswerable Question Creation: For SQuAD 2.0, annotators wrote plausible questions that _cannot_ be answered from the paragraph to test model refusal. 4. Paraphrase Validation: Ensure questions sound natural and don’t just copy the answer text. Project size - SQuAD 1.1: 107k Q&A pairs from 536 articles - SQuAD 2.0: 150k Q&A pairs, with 50k unanswerable questions added - 500+ crowd workers involved Quality measures adhered to. - Pilot phase: First 5k questions reviewed manually to lock down guidelines. - Answer span verification: Auto-check that selected span actually contains the answer text. - Question quality checks: Rejected questions that were too easy, ambiguous, or grammatically poor. - Adjudication: Disagreements on answer span resolved by majority vote or expert review. - Bias filtering: Removed questions that relied on annotator background knowledge not in the text.