Senior Python Engineer, LLM Evaluation (US-Based)

Create and refine code examples and solutions used to train and benchmark large language models. Evaluate AI-generated code for quality, efficiency, scalability, reliability, and alignment with software engineering benchmarks.

• Build Python agents and automated tools to find error patterns and verify software solutions.
• Assess AI coding work across prototyping, architecture, API design, production implementation, experimentation, monitoring, and maintenance.
• Work mainly with Python across backend services, data pipelines, and machine learning infrastructure. Some projects also use JavaScript, ReactJS, C/C++, Java, Rust, or Go.
• Work with cross-functional teams to improve AI coding solutions against industry benchmarks.

What the role requires

• At least three years of software engineering experience.
• Strong Python skills, including production-grade frameworks, tools, and practices.
• Experience building full-stack applications and deploying scalable software with modern languages and tools.
• Knowledge of software architecture, design, development, debugging, and code-quality assessment.
• Clear written and spoken communication of technical evaluation reasoning.
• Based in the United States.
• Commit at least 20 hours per week. The contract is planned for one month, with possible extensions based on performance and fit.

How to apply

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training uses examples prepared and reviewed by people to help artificial intelligence systems learn and measure their performance. Software engineers bring specialized knowledge to tasks such as creating code examples and evaluating AI-generated solutions.