Software Engineering AI Evaluation Specialist

About OpenTrain

OpenTrain AI is the hiring and contracting organization for this role and the #1 platform for finding and building careers in AI training and data labeling. OpenTrain helps contributors discover cutting-edge projects, build a professional profile, and apply in minutes.

• Worldwide contractor opportunity
• Part-time engagement requiring 20 or more hours per week
• English-language work

About AI Training and Code Evaluation

AI training is the human work behind modern artificial intelligence. For coding systems, experienced engineers prepare examples, test model-generated solutions, and judge whether responses are accurate, reliable, and useful. Your technical decisions can help shape how software-focused AI systems perform on real engineering problems.

• Work with real-world software and open-source codebases
• Evaluate language models on software engineering and bug-fixing tasks
• Create rigorous scenarios for coding-agent benchmarks

The Role

OpenTrain is recruiting a Software Engineering AI Evaluation Specialist to combine hands-on software engineering with AI evaluation. You will prepare public GitHub repositories for dependable local execution, examine software quality, and assess how effectively language models complete software engineering and bug-fixing tasks.

The role requires practical coding judgment, strong debugging ability, and the confidence to navigate complex codebases. You may also review code, provide technical guidance, and coordinate small teams when needed.

• Contractor position
• Part-time workload of 20 or more hours per week
• Open-source software code evaluation focus

What You’ll Do

You will work directly with repositories and development environments to make software projects suitable for reliable evaluation. Your work will span issue triage, environment setup, testing, code review, and assessment of AI-generated engineering outcomes.

• Analyze and triage issues across open-source repositories
• Configure development environments and Dockerize repositories
• Run, modify, debug, and test projects locally
• Evaluate unit test coverage, reliability, and overall test quality
• Assess model performance on software engineering and bug-fixing tasks
• Identify challenging repositories and issues for AI evaluation benchmarks
• Collaborate with technical researchers and engineers on evaluation scenarios
• Review code, provide technical guidance, and coordinate small teams when needed

Required Skills and Experience

This opportunity is intended for an experienced software engineer who can make sound technical judgments across complex, real-world projects. You should be comfortable configuring, modifying, testing, and debugging software locally while communicating your findings clearly.

• Five or more years of professional software engineering experience with complex codebases
• Strong proficiency in Python and JavaScript
• Practical experience with Git and Docker
• Working knowledge of software development pipelines and CI workflows
• Ability to understand, navigate, and modify complex codebases
• Experience running and testing real-world software projects locally
• Strong debugging, code review, problem-solving, and technical communication skills
• Sound judgment when evaluating test quality and language model performance

Why Build a Career in AI Training?

AI training and data-labeling work is a growing part of the technology industry. Contributors with specialized skills, including software engineering, help improve the systems that power the next generation of AI tools while building experience in an emerging field.

OpenTrain gives you a place to build a lasting AI training portfolio, show credible experience, and find projects aligned with your expertise. Creating an OpenTrain account is free.

• Apply your professional engineering judgment to cutting-edge AI work
• Build a portfolio around software evaluation and model quality
• Work remotely with a flexible part-time structure

How to Apply

Create a free OpenTrain account and apply through OpenTrain. Include your relevant software engineering experience and be ready to demonstrate your ability to work with Python, JavaScript, Git, Docker, complex codebases, and local testing workflows.

• Apply through OpenTrain
• Highlight five or more years of professional engineering experience
• Showcase debugging, code review, testing, and technical communication strengths