Software Engineering LLM Evaluator

As a Software Engineering LLM Evaluator, you will create and improve coding datasets used to train and benchmark large language models. You will use practical software engineering judgment to assess how well AI systems handle real development tasks.

The work covers code review, solution writing, automated verification, and structured evaluation. You will help identify recurring model errors and improve coding solutions against industry performance standards.

• Curate code examples and build solutions for AI model training projects.
• Review and refine AI-generated code for correctness, efficiency, scalability, reliability, and maintainability.
• Work with Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go.
• Build agents that verify code quality and identify recurring error patterns.
• Evaluate model performance across prototyping, architecture, API design, production implementation, launch, experimentation, monitoring, and operational maintenance.
• Design automated checks for software engineering tasks.
• Collaborate with cross-functional teams to improve AI coding solutions against industry benchmarks.

What it pays and takes

The listing does not provide a pay rate. This is a contractor, part-time engagement with a starting duration of one month and possible extensions based on performance and fit.

• Pay: Not specified in the listing.
• Hours: Flexible participation from 10 to 40 hours per week; the structured listing indicates 20 or more hours per week.
• Location: Worldwide, with partial Pacific Time overlap required.
• Language: English.
• Experience: Listed as entry level, while the role description requires several years of professional software engineering experience, including at least two continuous years full time at a top-tier product company.
• Required background: Strong full-stack application development and production deployment experience.
• Technical knowledge: Software architecture, design, development, debugging, and code quality assessment.
• Communication: Ability to provide clear, structured written and spoken rationales for evaluations.
• Programming: Proficiency in at least one of Python, JavaScript, ReactJS, C/C++, Java, Rust, or Go.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training work uses human judgment to help models learn from examples and produce better results. Experienced software engineers are paid to review code, explain what is wrong, and show models how reliable software should be designed and built.