James A.

Key Skills

Software

Scale AI

Other

Top Subject Matter

RLHF training data annotation and LLM evaluation

Reward model training data review and prompt writing

Top Data Types

Text

Top Task Types

RLHF

Freelancer Overview

AI Trainer — Data Annotation & Model Evaluation (ScaleAI Contractor, Remote). Brings 4+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Scale AI and Surge HQ. Education includes Bachelor of Science, University of Oklahoma (2023). AI-training focus includes data types such as Text and labeling workflows including RLHF, Evaluation, and Rating.

Labeling Experience

AI Trainer — Data Annotation & Model Evaluation (ScaleAI Contractor, Remote)

Scale AITextRLHF

Annotated and evaluated 12,000+ AI-generated responses to support RLHF training pipelines for LLM behavior refinement. Built high-quality prompts and adversarial test cases, then reviewed outputs for safety and factual accuracy failure modes. Worked with quality leads to maintain annotation consistency and documented policy-violating responses for alignment improvement cycles. • LLM response annotation across text summarization, Q&A, and code generation tasks • Adversarial testing to identify safety and factuality issues • Quality assurance focused on inter-rater agreement (consistently above 92%) • Policy-violation flagging and documentation for model alignment

2024 - Present

AI Content Reviewer & Prompt Engineer (Outlier AI via Surge HQ, Remote)

Text

Reviewed and ranked AI-generated creative and instructional content to train reward models used in generative AI products. Authored detailed structured prompts across STEM, creative writing, and customer support to expand training coverage. Identified edge cases and ambiguous instructions, providing rationales that informed updates to annotation guidelines. • Content review and ranking for reward-model training • Prompt writing across multiple domains to guide training signals • Edge-case identification with written rationale for guideline changes • Maintained 80+ tasks per day while meeting high quality scores

2023 - 2024

Education

Work History