English-Speaking C++ LLM Evaluation Engineer

You will help build and evaluate datasets for large language models, or LLMs. Your work will focus on real C++ software projects, GitHub issues, development environments, and bug-fixing tasks.

You will work with researchers to find repositories and issues that test how well LLMs understand and modify complex code. The role may also include leading a team of junior engineers.

• Analyze and triage GitHub issues in popular open-source libraries.
• Create verifiable software engineering tasks from public repository histories.
• Set up and configure code repositories, including Docker environments.
• Review unit test coverage and assess test quality.
• Modify and run codebases locally to evaluate LLM performance on bug-fixing problems.
• Work with researchers to identify challenging repositories and software issues.
• Potentially lead a team of junior engineers.

What it pays and takes

This is a remote, part-time contractor role. The listing is marked entry level, and the role requires at least three years of overall software engineering experience.

• Hours: 20 or more hours per week, with flexible options of 20, 30, or 40 hours.
• Location: Open to candidates in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and Mexico.
• Language: Professional fluency in English.
• Required: Strong C++ experience and the ability to understand complex codebases.
• Required: Proficiency with Git, Docker, and basic software pipeline setup.
• Required: Comfort running, modifying, and testing real-world projects locally.
• Helpful: Experience contributing to or evaluating open-source projects.
• Helpful: Previous LLM research or evaluation work.
• Helpful: Experience building or testing developer tools or automation agents.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including code tasks and reviews of model responses. Experienced software engineers help create difficult, accurate examples so AI systems can improve at real programming problems.