Skip to content
OpenTrain AIFor AI Companies

ML Engineer, LLM & Agentic Evaluations

Apply now

Job details

Engagement

Contract, part-time

Location

Remote

23 countries

Posted

Nov 18, 2026

OpenTrain on Trustpilot

The work

About OpenTrain
OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control. We make specialized AI training work easier to track, apply for, and grow into a durable freelance career.

About this role
OpenTrain is recruiting for an ML Engineer focused on evaluating large language models and agentic AI workflows. You will help measure model capabilities, identify failure modes, and improve how AI systems are trained and benchmarked through hands on evaluation work involving reasoning, coding, mathematics, tool use, and long context tasks. This is a remote contractor assignment for contributors based in Latin America.

What you will do
Design, build, and run evaluation suites for large language models across reasoning, coding, mathematics, tool use, and long context tasks. Develop benchmarks and rubrics for multi step planning, function calling, code execution, and browser or computer use agents. Curate, clean, and annotate evaluation datasets while writing precise task specifications and grading criteria. Compare open source and frontier models, analyze results, and produce structured reports with actionable recommendations. Implement programmatic and LLM based graders, validate them against human reference judgments, and investigate recurring error patterns. Integrate evaluations into engineering harnesses, CI pipelines, and dashboards, and contribute to documentation and tooling improvements.

Required skills
Pursuing or recently completed a Bachelor's, Master's, or PhD in computer science, machine learning, data science, mathematics, or a related quantitative field. Strong Python programming skills with familiarity with NumPy, Pandas, Jupyter, and standard machine learning workflows are required. Candidates should understand how modern LLMs are trained, prompted, and evaluated and have experience interacting with hosted or open source LLM APIs. Strong analytical and technical writing skills and the ability to work independently in a remote environment are important.

Helpful background
Experience designing or running LLM benchmarks, working with agent frameworks or tool using agents, and applying LLM as a judge methods is helpful. Familiarity with inter annotator agreement, statistical analysis, prompt engineering, structured outputs, RLHF, RLAIF, or post training data pipelines is also valuable.

Engagement details
This is a three month contractor assignment, adjustable based on engagement. The role supports specialized evaluation work for advanced AI systems.

Why work with OpenTrain
OpenTrain gives freelancers one place to manage AI training opportunities instead of rebuilding proof of work on every platform. A stronger OpenTrain profile helps you show credible experience, discover roles that match your skills, and turn AI training and data-labeling work into a long-term portfolio.

Requirements

  • Languages: English

Open to applicants in

  • Argentina
  • Belize
  • Bolivia
  • Brazil
  • Chile
  • Colombia
  • Costa Rica
  • Dominican Republic
  • Ecuador
  • El Salvador
  • Guatemala
  • Guyana
  • Haiti
  • Honduras
  • Mexico
  • Nicaragua
  • Panama
  • Paraguay
  • Peru
  • Puerto Rico
  • Suriname
  • Uruguay
  • Venezuela

How to apply

  1. Apply here on OpenTrain. You create a free account, and we send you to the hiring platform.
  2. Complete your application on the hiring platform.

OpenTrain on Trustpilot

Job ID: cmwmk86w2001c07jplth6amdh

Keep exploring

Similar open roles

View all AI training jobs

Applied AI Agent Evaluation Research Engineer

About OpenTrain. OpenTrain is the unified marketplace where AI training freelancers find and consolidate opportunities across platforms, build an AI training portfolio, and develop a durable freelance career. We help ca…

AI Agent Evaluation & Benchmark DesignPython & Research EngineeringAgent Failure Analysis & ExperimentationLLM & Agentic Systems
Text
Remote · United Arab Emirates, Argentina, Austria +49 more
English
Flexible hours
Hourly · $96.15–$125/hr

Posted Nov 17, 2026

LLM Evaluation Software Engineer (Ruby)

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

Ruby
Text
Remote · India, Pakistan, Nigeria +6 more
English
Part-time · Flexible

Posted Jul 20, 2026

Senior LLM Code Evaluation Engineer

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

Go
Code
Remote · India, Pakistan, Nigeria +6 more
English
Part-time · Flexible

Posted Jul 17, 2026

ML Engineer, LLM and Agentic Evaluations

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

PythonLLM
Text
Remote · India, Pakistan, Türkiye
English
Part-time · Flexible

Posted Nov 18, 2026

LLM Evaluation Analyst (Malay)

About OpenTrain OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control.…

Critical ThinkingBusiness Analysis
Text
Remote · Worldwide
Malay, English
Part-time · Flexible

Posted Jul 17, 2026