Senior Software Engineer, LLM Evaluation

Create and assess software engineering examples used to train and benchmark large language models. You will review AI-generated code and build reliable reference solutions.

• Improve code for efficiency, scalability, reliability, and quality.
• Build Python agents and automated tools to find error patterns and check solutions.
• Evaluate models on prototyping, architecture and API design, production implementation, monitoring, and maintenance.
• Design checks for software engineering tasks and compare results with performance benchmarks.
• Work with researchers and cross-functional teams on AI coding solutions.

Requirements and engagement

• At least three years of software engineering experience and strong Python skills, including production frameworks, tools, and best practices.
• Experience building full-stack applications and deploying scalable software.
• Strong skills in software architecture, design, development, debugging, and code review.
• Able to write and explain clear, structured evaluation rationales.
• Working knowledge of JavaScript, ReactJS, C or C++, Java, Rust, or Go is valuable.
• A background building high-scale production systems and a strong computer science foundation are helpful. Practical engineering ability is valued alongside formal education.
• Remote contractor role open to candidates based in the United States; professional fluency in English is required.
• Plan for 20 or more hours per week, with up to 40 hours available. The initial engagement is one month, with possible extensions based on performance and fit.

How to apply

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

OpenTrain helps people find AI training and data-labeling roles and apply in one place. In AI training, people prepare and review examples that help models learn, and experienced software engineers contribute by creating and assessing code examples.