Software Engineering AI Evaluation Specialist

You will help train AI systems by testing and assessing software-development workflows. You will verify model outputs in hands-on environments and give consistent technical judgments using structured evaluation criteria.

• Run evaluation assignments involving version control, pull requests, code review, issue tracking, and CI/CD workflows.
• Check whether AI-generated outputs are technically correct in relevant development environments.
• Create and maintain realistic repositories with commit histories, CI workflows, and reliable task contexts.
• Configure, document, and troubleshoot software integrations and authentication flows.
• Systematically test undocumented product behavior and document your findings.
• Take part in calibration activities so technical grading and rubric use stay consistent.

What it pays and takes

This is a part-time contractor role for candidates with professional software engineering experience and strong technical communication. The work is open to candidates in the listed countries and requires English communication skills.

• Pay: $50 to $70 per hour.
• Work type: Contractor and part-time.
• Language: Strong written and verbal English communication.
• Experience: At least three years of professional software engineering experience.
• Location: United Arab Emirates, Argentina, Austria, Bangladesh, Belgium, Bahrain, Brazil, Canada, Switzerland, Chile, Colombia, Germany, Denmark, Algeria, Egypt, Spain, Ethiopia, France, United Kingdom, Ghana, Greece, Indonesia, India, Italy, Jordan, Japan, Kenya, Kyrgyzstan, Kuwait, Kazakhstan, Lebanon, Sri Lanka, Morocco, Mexico, Malaysia, Nigeria, Netherlands, Oman, Peru, Philippines, Pakistan, Palestine, Portugal, Qatar, Saudi Arabia, Singapore, Thailand, Tunisia, Taiwan, United States, Uzbekistan, Vietnam, and South Africa.
• Technical skills: Git and GitHub workflows, including branching, pull requests, code review, diff analysis, and CI log diagnostics.
• Build skills: CI/CD systems, preferably GitHub Actions, including custom workflow authoring and reproducible builds.
• Scripting and integration: Advanced Python or Bash, environment setup and reset automation, REST APIs, OAuth, and webhook integrations.
• Useful experience: Connectors across Claude and ChatGPT, AI assistants, developer agent tools, Slack, Google Workspace, Microsoft 365, or cloud-workspace administration.
• Additional background: Technical QA, model evaluation, data annotation, consistent rubric use, nuanced edge-case detection, agent tool-calling, or the Model Context Protocol.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training is the human work behind systems that learn from examples, including reviewing software outputs and rating how well models complete technical tasks. People with software engineering experience are needed to spot subtle errors, test realistic workflows, and apply detailed evaluation standards.