SWE-Bench C++ Evaluation Engineer

You will evaluate C++ code produced or reviewed by language models using public GitHub repositories, issue reports, and unit-test codebases. You will apply project guidelines, document evidence for your decisions, and provide clear feedback that can improve model behavior.

The work also involves navigating complex codebases, running projects locally, modifying code, and testing real-world software projects.

• Evaluate model outputs and apply project rubrics.
• Review source materials, GitHub issues, code, and model outputs against project guidelines.
• Explain judgments clearly with evidence from the code or tests.
• Maintain consistent quality across repeated evaluation tasks.
• Run, modify, and test C++ projects locally.

What it pays and takes

The project requires at least two years of overall experience and strong practical C++ skills.

• Pay: $25 per label.
• Time: 20+ hours per week.
• Schedule: Four hours of overlap with PST is required.
• Language: Fluent English.
• Location: Open to candidates in Argentina, Bangladesh, Belize, Bolivia, Brazil, Chile, Colombia, Costa Rica, the Dominican Republic, Ecuador, Egypt, El Salvador, Ghana, Guatemala, Guyana, Haiti, Honduras, India, Mexico, Nicaragua, Nigeria, Pakistan, Panama, Paraguay, Peru, Puerto Rico, Suriname, Turkey, Uruguay, and Venezuela.
• Required: Strong C++ experience, plus proficiency with Git, Docker, and basic software pipeline setup.
• Required: Ability to understand and navigate complex codebases.
• Helpful: Experience with open-source projects, LLM research or evaluation, developer tools, or automation agents.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including code reviews, model evaluations, and written feedback. People with strong software knowledge help make these systems more accurate and useful by applying careful judgment to real-world technical tasks.