Senior Software Engineer, LLM Code Evaluation

Create and assess software engineering datasets used to train and benchmark large language models. The work centers on Python and covers backend services, data pipelines, machine learning infrastructure, full-stack development, and related programming languages.

• Curate code examples and write accurate reference solutions.
• Correct AI-generated code for efficiency, scalability, reliability, and quality.
• Build Python agents and automated tools to find error patterns and check solutions.
• Assess models on prototyping, architecture and API design, production implementation, monitoring, and maintenance.
• Create ways to verify software engineering tasks and compare results with benchmarks.
• Work with researchers and cross-functional teams on AI coding projects.

• Time: At least 20 hours per week.
• Engagement: Individual contractor role, initially one month, with possible extensions based on performance and fit.
• Required skills: Strong Python skills, including production frameworks, tools, and best practices; experience building full-stack applications and deploying scalable software; and knowledge of software architecture, development, debugging, and code review.
• You should be able to write clear, structured explanations for your evaluations.
• Experience with high-scale production systems and a strong computer science foundation are helpful. Practical engineering ability is valued alongside formal education.