Joshua B.

Key Skills

Software

Scale AI

Don't disclose

Top Subject Matter

LLM / AI evaluation

structured annotation

model safety and alignment

Top Data Types

Text

Document

Top Task Types

RLHF

Tracking

Classification

Fine-tuning

Data Collection

Freelancer Overview

Senior AI Customer Service Evaluation Specialist — Handshake AI. Brings 5+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Internal, Proprietary Tooling, and Scale AI. Education includes Doctor of Philosophy, Massachusetts Institute of Technology (MIT) (2026) and Master of Science, Massachusetts Institute of Technology (MIT) (2023). AI-training focus includes data types such as Text and labeling workflows including Evaluation, Rating, and Tracking.

Labeling Experience

Senior AI Customer Service Evaluation Specialist — Handshake AI

Text

Developed and optimized complex customer service questions and multi-turn case studies to stress-test generative AI model behavior in real support scenarios. Graded model outputs for correctness, conversational tone, adherence to support guidelines, and technical troubleshooting accuracy while enforcing quality frameworks. Engineered edge cases for de-escalation, policy interpretation, and empathetic complaint handling with written calibration feedback for production improvement. • Support scenario modeling and edge-case design • Expert scoring and rubric-based evaluation of model responses • Human-in-the-loop style validation and calibration documentation • Fine-tuning guidance via structured written technical explanations

2025 - Present

Senior Full-Stack / AI Engineer - Handshake AI

Scale AIRLHF

Senior full-stack engineer supporting AI evaluation and quality-focused review of complex model outputs, code, and machine learning responses. The role requires advanced technical understanding and the ability to apply expert judgment to evaluate accuracy, safety, and appropriateness. Responsibilities also include using structured processes to ensure consistent decisions and clear escalation documentation. • Evaluate model outputs and related artifacts for correctness, safety, and alignment with project criteria • Apply structured labeling frameworks and rigorous guideline compliance to grade findings • Detect ambiguous and out-of-scope edge cases and escalate with clear technical explanations • Collaborate on evaluation workflows that emphasize repeatability and decision traceability

2025 - Present

Senior Full-Stack / AI Engineer — Handshake AI (AI evaluation and annotation)

Text

Evaluated and reviewed the accuracy, quality, and appropriateness of complex AI model outputs and code-related machine learning responses using structured labeling frameworks. Applied rigorous annotation guidelines to grade model safety, structural soundness, and alignment of text content while documenting reproducible evaluation decisions. Escalated ambiguous or out-of-scope edge cases with clear written technical explanations to support downstream model and data alignment work.• Reviewed model-generated outputs and associated artifacts against project-specific guidelines.• Performed human-in-the-loop quality checks and ambiguity detection.• Graded outputs using structured safety and alignment rubrics.• Authored escalation notes describing subtle model errors and rationale.

2025 - Present

Customer Experience AI Evaluation Specialist — Outlier AI

TextTrackingClassification

Reviewed, graded, and annotated complex multi-turn support dialogues generated by frontier LLMs using internal dashboards and evaluation tools. Produced intensive, citation-backed analyses that explain grading decisions and assess alignment with customer experience best practices. Maintained high-volume quality assurance benchmarks while adhering to strict annotation frameworks and project guidelines. • Dialogue interaction assessment workflows for multi-turn transcripts • Structured annotation and grading with justification memos • High-volume QA across large text-based customer interaction datasets • Compliance with project-specific annotation guidelines

2024 - Present

Full-Stack Software Engineer — AI Evaluation Specialist — Outlier AI

Text

Conducted intensive evaluation workflows on text and model-generated outputs, producing structured, citation-supported annotations and detailed justifications for grading decisions. Maintained high accuracy across large volumes of guideline-driven validation tasks using project-specific multi-page annotation standards. Evaluated complex technical documents and content for nuance, logical errors, and overall quality to generate reproducible evaluation data.• Produced citation-backed annotations and reasoning memos for each grading decision.• Executed high-volume review with strict adherence to multi-page guidelines.• Performed content review and quality control on complex technical text and documents.• Translated domain expertise into consistent, reproducible evaluation outputs.

2024 - 2025

Show more

Education

Work History