Senior Python Software Engineer LLM Evaluation

About OpenTrain

OpenTrain is the #1 platform for finding and building careers in AI training and data labeling. It connects contributors with specialized projects, helps them build an AI training profile, and makes it easy to apply and grow in a rapidly developing field. Creating an OpenTrain account is free.

About AI Training and LLM Evaluation

AI training is the human work behind modern artificial intelligence. For language models, experienced contributors help create and evaluate examples, test model outputs, and assess whether systems can complete realistic tasks. In this role, your software engineering judgment will help measure how effectively LLMs diagnose and fix bugs in real codebases.

• Work remotely on practical AI evaluation projects.
• Contribute to the development of software engineering datasets and model evaluations.
• Use your technical experience to help shape how AI systems work with code.

The Role

OpenTrain AI is seeking a Senior Software Engineer specializing in Python for LLM evaluation and repository validation. You will work with realistic software engineering problems across open-source libraries, analyzing GitHub issues, configuring repositories and development environments, reviewing unit test quality, and running code locally to assess LLM bug-fixing performance.

The role may also involve helping researchers identify challenging repositories and issues and supporting junior engineers during collaborative project work. It is offered as a part-time contractor opportunity requiring 20 or more hours per week.

• Role focus: Software Engineering LLM Evaluation
• Work arrangement: Remote contractor
• Schedule: Part time, 20+ hours per week
• Working language: English
• Eligible countries: India, Pakistan, Nigeria, Kenya, Egypt, Ghana, Bangladesh, Turkey, and Mexico

What You'll Do

• Analyze and triage GitHub issues across open-source libraries.
• Set up repositories using Docker and development environment automation.
• Evaluate unit test coverage and overall test quality.
• Run and modify codebases locally to assess model performance on bug-fixing tasks.
• Navigate complex repositories and test real-world projects in local environments.
• Collaborate with researchers on repository and issue selection.
• Support junior engineers on collaborative project work when needed.

Requirements and Helpful Experience

This role requires practical software engineering experience and the ability to work confidently inside unfamiliar, complex codebases. The listing is marked entry level in its structured experience field, while the role requirements specify at least three years of overall engineering experience.

• At least 3 years of overall engineering experience.
• Strong Python experience, including software engineering and code modification.
• Hands-on skill with Git, Docker, and software pipeline setup.
• Ability to triage GitHub issues and evaluate unit test quality.
• Comfort running, modifying, and debugging real-world code locally.
• Experience contributing to or evaluating open-source projects is a plus.
• Prior LLM research or evaluation experience is helpful.
• Experience building or testing developer tools or automation agents is helpful.
• Familiarity with high-quality public GitHub repositories is helpful.

Who Should Apply

Apply if you are a strong Python engineer who enjoys investigating software defects, validating tests, and understanding how production-quality repositories work. This project is especially relevant for engineers with open-source experience or an interest in developer tools, automation agents, and evaluating AI-generated code.

• Python software engineers who enjoy debugging and code review.
• Engineers comfortable setting up repositories and development environments.
• Open-source contributors familiar with GitHub issues and public codebases.
• Developers interested in shaping the next generation of coding-capable AI.

How to Get Started

Create a free OpenTrain account, build your profile around your software engineering experience, and apply in minutes. OpenTrain helps contributors discover flexible AI training work and develop a durable career in a fast-growing industry.

• Create or update your OpenTrain profile.
• Highlight your Python, Git, Docker, testing, and repository experience.
• Apply for this contractor project through OpenTrain.