Shreyansh S.

Key Skills

Software

Other

Label Studio

Top Subject Matter

AI/GenAI evaluation and AI-in-Testing practices

Test automation and evaluation-quality infrastructure for AI systems

Community-led AI/LLM quality engineering and evaluation

Top Data Types

Text

Audio

Top Task Types

Text Generation

Text Summarization

Red Teaming

Transcription

Evaluation/Rating

Segmentation

Freelancer Overview

QA Lead at Version 1 (GenAI/LLM evaluation enablement via AI-in-Testing upskilling and AI-QE quality metrics). Brings 12+ years of professional experience across complex professional workflows, research, and quality-focused execution. Core strengths include Other. Education includes Bachelor of Engineering, Mahakal Institute of Technology (2014) and Certification (ISTQB Certified Tester Foundation Level), ISTQB. AI-training focus includes data types such as Text and labeling workflows including Evaluation and Rating.

Labeling Experience

Founder of AI-SDETS-HUB (AI-QE education and process adoption for LLM evaluation)

OtherText

Founded and leads an AI-QE community that promotes LLM evaluation processes, agentic AI testing, and shift-left/shift-right QA practices. Focuses on enabling practitioners to adopt evaluation-centric AI quality standards across multiple industry verticals. Acts as an architect of reference processes and toolchains for embedding LLM quality practices in product teams. • Builds a practitioner network to share AI quality engineering methods and evaluation frameworks. • Promotes adoption of agentic AI testing and LLM/RAG evaluation approaches. • Provides guidance on observability and traceability to compare framework impact vs. model changes. • Supports community-driven dissemination of AI-QE operating procedures.

2026 - Present

AI-QE Manager (Acting Director) at Logility (LLM evaluation framework architecture and evaluation pipeline operations)

OtherText

Built and led AI-QE capabilities for AI-first supply chain products by architecting LLM evaluation frameworks and operational test suites. Implemented evaluation pipelines and observability/traceability to attribute defect and regression outcomes to model changes. Reduced LLM defect leakage and improved release predictability by defining and owning AI-QE quality metrics delivered to leadership. • Established LLM evaluation frameworks using Arize Phoenix and DeepEval (plus GuideLLM) and reduced LLM defect leakage by ~60%. • Architected an LLM evaluation suite with 200+ tagged tests across LLM, API, frontend, and security layers using Arize traceability. • Defined quality metrics (defect leakage, MTTR, automation coverage, stability trends) and reported them for ROI on AI investment. • Reduced regression bugs per cycle by lowering regression issues from ~50–60 to ~15–20 via evaluation improvements.

2024 - 2026

QA Lead at Version 1 (GenAI/LLM evaluation enablement via AI-in-Testing upskilling and AI-QE quality metrics)

OtherText

Led QA leadership for GenAI/LLM evaluation practices by building and operationalizing quality metrics and reporting loops to support AI-first product testing outcomes. Focused on upskilling QA engineers in AI-in-Testing to accelerate automation development and improve defect lifecycle management. Drove structured debugging and root cause analysis ceremonies to improve attribution and reliability of evaluation results. • Built quality dashboards tracking defect leakage, MTTR, and automation coverage for stakeholders. • Mentored 10 QA engineers in AI-in-Testing practices to reduce automation cycle time by ~30%. • Enhanced MTTR through joint debugging and RCA sessions with cross-functional teams. • Established Automation COE engagement via monthly meetings for AI-oriented QA capability building.

2022 - 2024

QA Automation Lead at Mitratech (foundation for scalable QA pipelines supporting AI evaluation)

OtherText

Owned automation and quality engineering practices that enabled scalable testing infrastructure used for advanced AI evaluation workflows. Reduced regression cycle time and expanded non-functional testing coverage to improve overall model and system reliability signals for downstream AI/LLM assessment. Implemented CI-integrated build validation to reduce flaky behavior that can corrupt evaluation outcomes. • Managed an Automation COE and monthly upskilling cadence for QA engineers. • Reduced regression cycle time by ~40% via scalable automation frameworks and stable CI/CD pipelines. • Expanded non-functional test coverage to ~50% using new API and performance suites. • Reduced flaky tests and maintenance overhead by ~60% via PR build validation workflows.

2020 - 2022

Education

Work History