AI Conversation Evaluation Specialist

You will review conversations between people and AI models as part of conversational AI evaluation research. Your feedback will focus on whether the AI had enough context to make a useful recommendation.

You will assess real-world conversations involving relationships, work, education, family, planning, wellbeing, and other personal or organizational decisions. You will also work with research teams to keep reviews consistent and well supported.

• Identify information that should have been gathered before an AI recommendation was made.
• Suggest follow-up questions that could clarify a person's needs, limits, or circumstances.
• Assess whether additional information could have changed the AI's recommendation.
• Separate important details from descriptive or non-essential information.
• Document findings through qualitative review and structured evaluation guidelines.

What it pays and takes

This is an entry-level, part-time contractor role with a weekly commitment of at least 20 hours. The role is open to candidates in the countries listed for this opportunity.

• Hours: 20 or more hours per week.
• Work arrangement: Part-time contract work.
• Language: Professional fluency in English.
• Experience level: Entry level.
• Relevant background: Three to five years of professional practice or subject-matter experience.
• Required fields may include psychology, social work, counseling, qualitative research, HCI, communication, conflict resolution, coaching, mediation, student services, organizational consulting, or a related advisory field.
• Relevant experience may include client assessment, interviewing, counseling, case review, conversation analysis, thematic analysis, qualitative coding, user research, or comparable work.
• Helpful qualifications may include an MSW, social-work credential, counseling or therapy license, PhD or PsyD, or advanced training in HCI, communication, anthropology, sociology, or qualitative research.

How it works

Apply on OpenTrain with your resume, then complete the application on the hiring site.

About AI training work

AI training work uses human review to help models understand language, context, and useful responses. This role applies specialist judgment to conversations where personal, social, and practical details can affect the quality of an AI recommendation.