Andrew G.

Key Skills

Software

AWS SageMaker

Top Subject Matter

Human-in-the-Loop NLP evaluation and annotation

ML ranking dataset labeling and evaluation

Human-in-the-loop evaluation

Top Data Types

Text

Top Task Types

RLHF

Freelancer Overview

Research Engineer & Postdoctoral Fellow, UT Austin (Human-Centered AI Lab). Brings 14+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Internal, Proprietary Tooling, and AWS SageMaker. Education includes Doctor of Philosophy, University of Texas at Austin (2013) and Bachelor of Science, Texas A&M University (2009). AI-training focus includes data types such as Text and labeling workflows including Evaluation, Rating, and RLHF.

Labeling Experience

Principal Software Engineer, Netsmart Technologies

TextRLHF

Architected a human-in-the-loop data validation platform that processed millions of records per month and routed expert evaluation tasks by domain, language, and prior quality history. Integrated LLMs into clinical documentation workflows by creating standardized prompt templates, output scoring rubrics, and inter-rater reliability dashboards for consistent evaluation. Led development of APIs and evaluation tooling that supported large-scale expert validation and quality measurement for AI outputs. • Human expert task routing and data validation • Creation of prompt templates and scoring rubrics • Inter-rater reliability dashboards for evaluator consistency • Large-scale processing of labeled/evaluated clinical records

2020 - Present

Senior Software Engineer - HomeAway

AWS SageMakerTextRLHF

Senior software engineer owning microservices for real-time pricing and availability at large scale. Worked with event streaming and caching layers to support ML ranking and search quality initiatives. Required experience in Kafka, PostgreSQL/Redis, API security (OAuth/JWT), and building evaluation-oriented tooling. • Owned microservices for real-time pricing and availability across 2M+ listings. • Built and maintained backend infrastructure for annotation and QA workflows used by data science teams. • Collaborated on evaluation metrics, A/B frameworks, and automated scoring pipelines for search ranking. • Designed OAuth2/JWT authentication, role-based access control, and audit logging for a multi-tenant platform.

2016 - 2020

Senior Software Engineer, HomeAway (Vrbo / Expedia Group)

Text

Developed an internal annotation and quality-assurance tool used by data science teams to label and review training datasets for ML ranking models. Collaborated with ML engineers to instrument evaluation metrics, run A/B frameworks, and automate scoring pipelines for search ranking features to support model improvement. This role focused on evaluation tooling and data labeling workflows used in production ML iterations. • Annotation and QA of training datasets for ML ranking • Evaluation metric instrumentation and automated scoring • A/B testing frameworks to validate changes • Labeling/review processes to improve ranking quality

2016 - 2020

Research Engineer and Postdoctoral Fellow - The University of Texas at Austin

AWS SageMakerTextRLHF

Research engineer and postdoctoral fellow in the Human-Centered AI Lab developing experimental platforms for human evaluation of NLP systems. Responsible for engineering web-based annotation infrastructure and running research workflows while publishing results. Required skills in Django/React, experiment design, and statistical methods for measuring agreement and quality. • Built a Django/React annotation platform used across multiple published studies. • Supported large-scale evaluation workflows with configurable task templates and real-time quality monitoring. • Co-authored peer-reviewed papers on human evaluation methodology and annotator agreement metrics. • Supervised M.S. students and taught a graduate seminar on human-in-the-loop machine learning.

2013 - 2016

Research Engineer & Postdoctoral Fellow, UT Austin (Human-Centered AI Lab)

Text

Built experimental platforms for crowdsourced and expert human evaluation of NLP model outputs, including human judgment pipelines. Designed and maintained an annotation platform to support large-scale evaluation studies with configurable task templates and real-time quality monitoring. The work emphasized measuring quality and agreement to improve model behavior via feedback-driven iteration. • Human annotation and evaluation of NLP outputs • Crowdsourced and expert evaluator workflows • Inter-annotator agreement and reliability monitoring • Support for 1,200+ annotators across multiple languages

2013 - 2016

Education

Work History