AI Coding Agent Evaluation Expert

About OpenTrain

OpenTrain AI is the #1 platform for finding and building careers in AI training and data labeling. We help people discover opportunities, build a professional AI-training profile, and grow their careers in a fast-expanding field where human expertise directly shapes how AI systems work.

About AI Coding Agent Evaluation

AI training is the human side of building artificial intelligence. In this role, you will assess coding assistants and provide structured feedback that helps improve how AI models reason about software engineering problems, technical workflows, and open-ended development tasks.

• Remote work available worldwide
• Part-time contractor opportunity
• Work with cutting-edge AI coding technology
• Help evaluate and improve AI-generated technical solutions

The Role

OpenTrain AI is seeking an AI Coding Agent Evaluation Expert to contribute detailed accounts of real-world technical workflows, structured evaluations, and written and verbal feedback. You will bring practical STEM experience to the assessment of how coding agents handle complex problems, including large codebases, advanced debugging, architectural changes, end-to-end feature work, and ambiguous requirements.

This role is listed as entry level, while requiring regular hands-on experience with AI coding agents and the ability to evaluate sophisticated technical work. It is designed for practitioners who use AI coding agents as a meaningful part of their technical toolkit.

• Compensation: $80–$100 per hour
• Time commitment: 20+ hours per week
• Work arrangement: Part-time contract
• Location: Worldwide
• Primary language: English

What You'll Do

You will turn your experience using AI coding agents into clear, evidence-based guidance for model improvement. Your work will include written evaluations, verbal feedback, structured discussions, surveys, and knowledge-sharing sessions.

• Describe the AI coding agents, tools, workflows, and technical challenges involved in your day-to-day work.
• Explain step by step how AI agents support end-to-end feature development, complex project collaboration, and resolution of unclear requirements.
• Evaluate agent performance on intricate technical problems, including large-codebase navigation, debugging, and architectural refactoring.
• Identify the strengths and limitations of AI coding agents across varied real-world scenarios.
• Provide concrete observations and data-driven suggestions for improving AI models and coding assistants.
• Share your experience through structured discussions, surveys, and knowledge-sharing sessions.

Required Skills and Experience

Applicants should have current or extensive professional experience in a STEM-related field and regular hands-on experience using AI coding agents for sophisticated technical work beyond basic code completion. You must be able to explain complex technical workflows, decisions, challenges, and solutions clearly in writing and verbally.

• Current or extensive experience as a STEM professional
• Regular hands-on use of AI coding agents beyond basic code completion
• Experience applying AI assistants to end-to-end feature development, advanced debugging, architectural refactoring, or comparable open-ended problems
• Ability to assess, test, and explain the quality and limitations of AI-enabled technical solutions
• Familiarity with popular AI coding tools and emerging practices in AI-assisted technical work
• Ability to critically evaluate AI-generated technical work and communicate its strengths and limitations

Helpful Background

Experience communicating AI-enabled solutions to both technical and non-technical stakeholders is helpful. A track record of using coding agents across dynamic, real-world projects is also valuable.

Why Work in AI Training

People who contribute to AI training help shape how modern AI systems behave. Depending on the project, contributors may evaluate model outputs, write feedback, or assess technical solutions—all work that gives human experts a direct role in developing the next generation of AI tools.

• Work remotely with a flexible part-time schedule
• Apply your existing technical experience to emerging AI systems
• Contribute to the development of advanced coding assistants
• Build experience in a rapidly growing AI-training industry