AI Model Evaluation Data Scientist

You will use Python and data analysis to evaluate AI models and identify ways to improve their performance. The work combines technical analysis, response review, dataset creation, and clear written reasoning.

You will work with researchers and annotators on supervised fine-tuning, reinforcement learning with human feedback, and reward-model refinement. You will also use public datasets to answer business questions and review code and documentation for issues.

• Design, develop, and maintain Python code for training and optimizing AI models.
• Benchmark model performance and analyze evaluation results.
• Evaluate and rank model responses to user queries using set criteria.
• Write clear explanations and rationales for evaluation decisions.
• Create and maintain datasets for supervised fine-tuning.
• Create and refine responses for clarity, relevance, and technical accuracy.
• Review code and documentation and provide constructive feedback.
• Use public datasets from sources such as Kaggle, the United Nations, and the US government.

What it pays and takes

The listing does not specify a pay rate. This is a remote contractor assignment with a one-month contract term.

• Hours: At least 20 hours per week, with options for 20, 30, or 40 hours per week.
• Schedule: Four hours of daily overlap with Pacific Time.
• Location: Fully remote and open worldwide.
• Language: Fluent conversational and written English.
• Education: A bachelor's or master's degree in engineering or computer science, or equivalent experience.
• Experience level: Entry level.
• Requirements: Proficiency in Python, strong data analysis skills, problem-solving ability, and sound business judgment.
• Communication: Explain technical reasoning clearly in Jupyter notebooks or comparable formats and communicate effectively with researchers and other stakeholders.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

OpenTrain is the hiring and contracting organization for this role. AI training work uses human examples and feedback to improve how artificial intelligence systems respond, and people with data science and technical skills help assess model quality and guide those improvements.