Senior Python Engineer, LLM Evaluation (US-Based)

Use your software engineering judgment to assess and improve code examples, programming solutions, and AI-generated code used to train and benchmark large language models.

• Curate code examples and create precise programming solutions for training datasets.
• Review and refine AI-generated code for correctness, efficiency, scalability, and reliability.
• Build Python agents and automated tools that find error patterns and check software solutions.
• Assess model capabilities across prototyping, architecture and API design, implementation, launch, experimentation, monitoring, and maintenance.
• Design checks for software engineering tasks and compare coding solutions against performance benchmarks.
• Contribute code in Python, JavaScript including ReactJS, C/C++, Java, Rust, and Go.

• Time: At least 20 hours per week.
• Experience: At least three years of software engineering experience.
• Fluent in English.
• Strong Python skills, including production frameworks, tools, and best practices.
• Experience building full-stack applications and deploying scalable software.
• Strong knowledge of software architecture, development, debugging, and code review.
• Ability to write clear, structured explanations for technical evaluations.
• Familiarity with backend services, data pipelines, or machine learning infrastructure.
• Experience building production systems at high scale and a strong computer science foundation are helpful. Practical engineering skill is valued alongside formal education.