Senior Python Engineer for LLM Evaluation

You will help build and review training tasks for language models working on real software engineering problems. The role combines repository setup, issue analysis, test review, and local code execution.

You may also help researchers choose difficult repositories and issues, and support junior engineers during collaborative project work.

• Analyze and triage GitHub issues in open-source libraries.
• Set up repositories and development environments with Docker and automation tools.
• Review unit test coverage and assess whether tests are useful and well designed.
• Run and modify codebases locally to evaluate model performance on bug-fixing tasks.
• Work with researchers to select challenging repositories and software issues.
• Support junior engineers on project work when needed.

What it pays and takes

The role details do not specify a pay rate. It is a part-time contractor position with a minimum weekly commitment and specific country and language requirements.

• Pay: Not specified in the role details.
• Time: 20+ hours per week.
• Work arrangement: Part-time contract work.
• Location: Open to candidates in India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Türkiye, or Mexico.
• Language: English.
• Experience: At least 3 years of overall engineering experience.
• Required skills: Strong Python, practical Git and Docker skills, and experience setting up software pipelines.
• You should be comfortable navigating complex codebases and testing real-world projects locally.
• Helpful background includes contributing to or evaluating open-source projects, building or testing developer tools or automation agents, and prior LLM research or evaluation experience.
• Familiarity with high-quality public GitHub repositories is helpful.

How it works

Apply on OpenTrain with your resume and then complete the application on the hiring site.

About AI training work

AI training work uses examples, tests, and human reviews to improve how AI systems perform. In this role, your software engineering judgment helps assess whether language models can understand repositories and fix real bugs.