AI Code Evaluation and Benchmarking Engineer

You will evaluate AI-generated software against functional and technical requirements. The work includes testing proposed solutions, debugging code, reproducing issues, and checking fixes across different programming environments.

You will also assess model-generated explanations and implementation approaches for technical accuracy. Your findings will help create reliable evaluation data for systems that generate and reason about code.

• Review AI-generated code for correctness, efficiency, maintainability, and compliance with requirements.
• Analyze software engineering tasks and confirm whether proposed solutions produce the expected results.
• Debug code, reproduce technical issues, and verify fixes across programming environments.
• Assess AI-generated explanations, reasoning, and implementation approaches.
• Create, improve, and maintain coding evaluation datasets, benchmarks, and grading rubrics.
• Identify edge cases and technical failure modes in AI software engineering outputs.
• Document findings and provide structured feedback for consistent evaluations.
• Work with project teams to establish quality standards and evaluation methods.

What It Pays and Takes

Compensation details are not disclosed. This is a remote contractor assignment for freelancers located in the United States.

• Contract length: One month.
• Time requirement: At least 20 hours per week, including at least four hours per day.
• Schedule requirement: At least four hours of overlap with Pacific Time.
• Language: English.
• Education: Bachelor's or master's degree in computer science, software engineering, or a related technical field.
• Experience: At least three years of professional software engineering experience.
• Programming: Proficiency in one or more of Python, Java, C, C++, Go, Swift, Objective-C, PHP, or SQL.
• Technical knowledge: Data structures, algorithms, software design principles, and debugging methods.
• Additional requirements: Code review experience, production or large-scale codebase evaluation experience, and familiarity with Git or comparable version control systems.
• You should have strong written communication, careful attention to detail, and the ability to assess complex technical solutions with minimal supervision.

Helpful Background

Experience with AI or machine learning data annotation, natural language processing, prompt engineering, model evaluation, large language model projects, AI-generated code evaluation, benchmark creation, or software quality assessment is valuable.

How It Works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI Training Work

AI training work is the human side of building artificial intelligence. People review examples such as code, text, images, and model responses so AI systems can learn to produce more useful and accurate results.