Fidelis A.

Key Skills

Software

No software listed

Top Subject Matter

AI Model Evaluation

LLM Output Auditing

Decision Support AI

Top Data Types

Text

Document

Top Task Types

RLHF

Question Answering

Prompt + Response Writing (SFT)

Classification

Point/Key Point

Fine-tuning

Red Teaming

Transcription

Evaluation/Rating

Data Collection

Object Detection

Segmentation

Entity (NER) Classification

Polygon

Freelancer Overview

Founder & AI Systems Researcher (LLM Evaluation & AI Output Auditing). Brings 5+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Internal and Proprietary Tooling. Education includes Doctor of Philosophy, University of Cincinnati (2026) and Master of Science, University of Cincinnati (2025). AI-training focus includes data types such as Text and labeling workflows including Evaluation, Rating, and RLHF.

Labeling Experience

Founder & AI Systems Researcher (LLM Evaluation & AI Output Auditing)

Text

As Founder & AI Systems Researcher at VitalSearch, led the design, evaluation, and deployment of production-grade agentic AI systems used for structured output validation and auditing. Developed and applied systematic workflows to evaluate LLM outputs for accuracy, reasoning quality, and failure mode identification, used in real-time decision support contexts. Published peer-reviewed research on AI output auditing and contributed directly to RLHF and fairness training data pipelines. • Designed and conducted structured model evaluation for LLMs, including output accuracy and traceability. • Built and deployed evaluation frameworks used in live AI training and auditing systems. • Developed protocols for output provenance and systematic inconsistency detection. • Led research-integrated labeling tasks validated in peer-reviewed venues.

2025 - Present

LLM RLHF Evaluator & Safety Rating Specialist

TextRLHF

Designed and implemented structured RLHF evaluation rubrics for large language models, focusing on accuracy, safety, and output traceability. Scored and analyzed thousands of model responses to identify failure modes, reduce false-positive/negative rates, and refine training signals. Led evaluation with a diverse human cohort to ensure robust reliability and real-world alignment. • Created and iterated on evaluation workflows for safety and RLHF projects. • Performed rubric-driven human rating of LLM outputs. • Led identification and mitigation of bias and error in model outputs. • Benchmarked evaluation processes against published research results.

2024 - Present

AI/Systems Analyst (AI-Assisted Workflow Evaluation)

Text

Assessed and triaged AI system outputs for consistency, error detection, and behavioral validation in production workflow environments. Translated ambiguous requirements into structured evaluation formats, facilitating systematic assessment of AI-assisted systems. Led the process mapping and cross-platform validation of system responses and model outputs. • Performed structured inconsistency identification in cross-platform AI workflows. • Designed evaluation metrics for edge-case error detection in AI systems. • Led validation of system outputs against defined technical criteria. • Authored technical documentation supporting cross-team evaluation work.

2023 - 2024

AI/Policy Analyst (Medicare Advantage Data Validation)

Text

Validated structured datasets and performed comprehensive consistency checks on AI-related data for regulatory reporting pipelines. Translated complex regulatory frameworks into testable technical specifications to ensure high accuracy of annotated outputs. Led data-driven logic design for identifying and addressing output inconsistencies at scale. • Applied SQL-based methods to verify integrity of large structured datasets. • Engineered test cases simulating policy-driven data edge scenarios. • Built output validation protocols for risk adjustment and compliance. • Created technical evaluation reports for regulatory and AI alignment.

2022 - 2022

Structured Data Validator (FDA FAERS Automated Outputs)

Text

Conducted structured data validation and inconsistency detection on pharmaceutical datasets as part of AI evaluation projects. Designed automated tools to flag output errors and validate metadata within structured outputs, directly relevant to high-stakes domains. Developed processes and query logic for ensuring reliable AI-assisted outputs in regulatory contexts. • Implemented logic for flagging output inconsistencies in pharmaceutical datasets. • Automated metadata extraction and date validation for enhanced reliability. • Built synonym expansion pipelines to encapsulate data edge cases. • Applied evaluation frameworks to data from openFDA APIs.

2022 - 2022

Education

Work History