LLM Code Evaluation Engineer

You will evaluate large language models against realistic coding problems and validate repositories used to build AI training datasets. The work combines hands-on software development, repository review, and structured assessment of model performance.

You will work with public open-source codebases, investigate bug-fixing scenarios, and help create software-engineering tasks that can be checked through tests and code review.

• Analyze and triage issues in widely used open-source libraries.
• Configure repositories and development environments, including Docker-based setups.
• Review unit-test coverage and assess whether tests are well designed.
• Modify and run codebases locally to evaluate LLM performance on bug-fixing tasks.
• Work with researchers to identify repositories and issues that provide meaningful challenges for language models.
• Help coordinate junior engineers on project work when needed.

What it pays and takes

This is a part-time contractor role listed at entry level. You should be comfortable working independently with real software projects and communicating clearly in English.

• Time requirement: 20 or more hours per week.
• Location: Open worldwide.
• Language: English proficiency required.
• Programming: Strong experience in at least one of Python, JavaScript, Java, Go, Rust, C, C++, C#, or Ruby.
• Tools: Proficiency with Git, Docker, and basic software pipeline setup.
• Technical ability: Understand and navigate complex codebases, and run, modify, and test real-world projects locally.
• Useful experience: Contributing to or evaluating open-source projects, especially well-maintained public repositories.
• Helpful background: LLM research or evaluation, developer tools, automation agents, or familiarity with public repositories that have 500 or more stars.

How it works

Apply on OpenTrain with your resume and then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including code reviews, model evaluations, and labeled data. OpenTrain is a free platform where people find AI training and data-labeling projects, build a profile, and apply in minutes.