Python Software Engineering LLM Evaluator

You will work with public GitHub repositories and realistic software engineering tasks to assess how language models handle real code and bug-fixing scenarios. The work includes setting up projects locally, testing code, and judging the quality of model performance.

You will also work with researchers to find repositories and issues that create useful challenges for language model evaluation.

• Analyze and triage issues in open-source Python libraries.
• Configure repositories and Docker-based development environments.
• Run, modify, and test real codebases locally.
• Evaluate unit-test coverage and test quality.
• Assess language model performance on software engineering and bug-fixing tasks.
• Help researchers identify repositories and issues for meaningful evaluations.

What It Pays And Takes

This is a fully remote contractor assignment lasting three months. The role is marked as entry level and is listed as part-time work.

• Pay: $100 per task.
• Time: 20+ hours per week.
• Location: Open to candidates in India, Pakistan, Nigeria, Egypt, Ghana, Bangladesh, Turkey, or Mexico.
• Language: English fluency required.
• Experience: At least three years of software engineering experience.
• Skills: Strong Python experience, plus proficiency with Git, Docker, and basic software pipeline setup.
• You must be able to understand complex codebases and run, modify, and test real-world projects locally.
• Experience contributing to or evaluating open-source projects is a plus.
• Previous language model research or evaluation experience is a plus.

How It Works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI Training Work

AI training work is the human side of building artificial intelligence. Contributors review code, write or rate model responses, and test how well models handle real tasks, helping improve the systems people use.