Kelly P.

Key Skills

Software

Other

Top Subject Matter

LLM RLHF scenario design and evaluation for human-AI training

RLHF data pipeline engineering and scenario prompt/label construction for LLM fine-tuning

LLM training data infrastructure quality review and evaluation support

Top Data Types

Text

Audio

Geospatial Tiled Imagery

Top Task Types

RLHF

Fine-tuning

Freelancer Overview

AI Scenario Designer & RLHF Specialist. Brings 8+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Internal, Proprietary Tooling, and Other. Education includes Bachelor of Technology, N/A (2019). AI-training focus includes data types such as Text and labeling workflows including RLHF, Fine-tuning, and Evaluation.

Labeling Experience

AI Scenario Designer & RLHF Specialist

TextRLHF

Designed and refined hundreds of realistic, multi-layered AI training scenarios used in RLHF pipelines, including scheduling conflicts and competing priority dilemmas. Constructed simulated digital environments (emails, calendar invites, Slack/Drive structures) with authentic personas and context noise to stress-test agent reasoning. Evaluated agent responses against nuanced quality criteria and authored structured hints to guide model improvement. • Created training prompts/tasks and scenario specifications for RLHF preference learning. • Authored evaluation rubrics and reasoning-quality critique criteria for model scoring. • Maintained structured documentation using internal/proprietary annotation tooling similar to Airtable and Crucible. • Collaborated with finetuning researchers to iterate complexity and realism and translate research needs into testable challenges.

2023 - Present

RLHF Data Pipeline Engineer & Scenario Architect

OtherTextFine-tuning

Designed and maintained scalable RLHF data pipelines, including prompt scenario creation and preference-pair structuring for 50,000+ examples per quarter used in supervised fine-tuning of domain-specific LLMs. Authored complex, realistic task prompts across professional communication, scheduling, research synthesis, and multi-stakeholder coordination to elicit and evaluate nuanced model behaviors. Collaborated daily with finetuning researchers to iterate prompt design, annotation schema, and feedback tooling. • Built preference datasets (prompt pairs/ratings) for RLHF training. • Developed and refined annotation schemas and feedback mechanisms for evaluation. • Implemented automated regression testing and standardized quality expectations across teams. • Produced prompt libraries and documentation to support repeatable training runs.

2021 - 2022

ML Data & Evaluation Infrastructure Specialist

OtherText

Contributed to large-scale training data infrastructure for transformer-based LLM training across conversational AI, search, and classification tasks, with involvement in prompt design and quality review. Identified systemic data quality issues and implemented monitoring to improve training data reliability across two production model releases. Contributed to finetuning tooling improvements that reduced researcher setup time for new training experiments and demonstrated strong documentation and workflow design. • Performed qualitative/quality review of prompts and training data used in LLM pipelines. • Implemented data quality monitoring to ensure label/data reliability. • Supported fine-tuning infrastructure and workflows for new experiments. • Documented processes to standardize training-data preparation and evaluation expectations.

2019 - 2021

Education

Work History