Senior Software Engineer Python LLM Evaluation

You will help evaluate AI systems on realistic software engineering problems. The work focuses on open-source repositories, GitHub issues, code quality, and local testing.

You may also help researchers choose difficult repositories and issues and support junior engineers during project work.

• Analyze and triage GitHub issues across open-source libraries.
• Set up repositories using Docker and development environment automation.
• Evaluate unit test coverage and test quality.
• Run and modify codebases locally to assess LLM bug-fixing performance.
• Work with researchers to select repositories and issues for evaluation.
• Support junior engineers on collaborative project work when needed.

This is a part-time contractor role with a commitment of at least 20 hours per week.

You should be comfortable working in English and have strong practical software engineering experience, especially with Python and local development environments.

• At least 3 years of overall engineering experience.
• Strong Python experience.
• Practical skill with Git, Docker, and software pipeline setup.
• Comfort navigating complex codebases and testing real-world projects locally.
• Professional fluency in English.