SWE Bench Engineer (Java LLM Code Evaluation)

You will assess Java software projects and help measure how well language models handle real bug-fixing tasks. The work involves running, changing, testing, and reviewing open-source code locally.

• Analyze and triage issues in open-source Java libraries.
• Set up repositories, Docker-based development environments, and basic software pipelines.
• Review unit-test coverage and overall code quality.
• Modify and run projects locally to evaluate LLM performance on bug-fixing tasks.
• Find repositories and issues that create meaningful evaluation challenges for language models.
• Work with researchers and, when needed, guide junior engineers on related projects.

This is a remote contractor assignment for freelancers in Latin America. It is a one-month contract, and the structured time commitment is 20 or more hours per week.

• Schedule: 20 or more hours per week, including four hours of overlap with PST.
• Experience: At least three years of overall software engineering experience.
• Technical skills: Strong hands-on Java experience, plus proficiency with Git, Docker, and basic software pipeline setup.
• You must be comfortable navigating complex codebases and running, modifying, and testing real-world projects locally.
• Language: Professional fluency in English.