AI Quality Analyst — Personalization

About OpenTrain

OpenTrain is the centralized platform for people who build careers in AI training and data labeling. We help contributors discover projects, track opportunities, and build a unified portfolio of AI training work.

OpenTrain AI is the contracting organization for this role. We recruit and manage short-term contractor projects that let you gain experience, earn, and grow inside a fast-moving industry.

Why AI training work matters

AI training (also called data labeling or human feedback work) is the human side of building artificial intelligence. Contributors create, review, and rate examples that shape how modern models behave.

This work is largely remote, often flexible and part-time, and accessible to people with strong attention to detail and analytical thinking. Contributors directly influence how state-of-the-art conversational systems respond to real users.

The role

As an AI Quality Analyst (Personalization) you will evaluate how well a conversational AI uses personal context to make responses more relevant and helpful. This is a hands-on text-evaluation role combining creative prompt design with rigorous, defensible ratings and written rationales.

Tasks are text-based evaluations and side-by-side response rankings (label type: EVALUATION_RATING). You will design and run multi-turn conversations, judge model grounding and integration of personal context, and explain your rankings with explicit references to turn numbers.

What you'll do

• Design and execute multi-turn conversational prompts (typically 1–5 turns) that require the AI to use personal context and past interactions.
• Evaluate model responses against your intent and check whether personalization was applied appropriately.
• Analyze grounding to ensure claims are supported and not the result of flawed inference or hallucination.
• Assess integration quality so personal data is woven naturally into replies without robotic overnarrating or forced connections.
• Compare and stack-rank two model responses side-by-side to determine which is more helpful, usable, and enjoyable.
• Write clear, defensible rationales for comparisons, explicitly referencing specific turn numbers and observable differences.

Requirements (must have)

This is an entry-level-friendly contract but it requires strong analytical and written skills. You must be able to evaluate nuanced, ambiguous language and produce structured rationales that other reviewers can follow.

• Fluent written English and high reading comprehension.
• Exceptional analytical thinking and ability to assess nuanced AI outputs.
• Experience designing creative multi-turn prompts derived from personal context.
• Strong understanding of personalization concepts and how personal data should be integrated into responses.
• Meticulous attention to detail for side-by-side comparisons and spotting subtle differences in naturalness or overnarration.
• Excellent written communication skills (clear, structured rationales referencing turn numbers).
• Reliable desktop or laptop computer with a stable internet connection.

Helpful background (not required)

• BS/BA or equivalent in Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field.
• Prior experience in data annotation, AI quality evaluation, content moderation, or similar roles.

Engagement, schedule, and compensation

Contract length: 3 months. Employment type: contractor, part-time. Time expectation: 20+ hours per week; the role also describes a commitment of at least 4 hours per day up to 40 hours per week with a required 4-hour daily overlap with Pacific Time (PST).

No hourly or fixed compensation is listed in the role details; OpenTrain will provide contracting information during the hiring process.

How the work is delivered

This is remote, text-based evaluation work. You will be asked to run short multi-turn scenarios, rate outputs, and submit written rationales through the OpenTrain platform tools.

Labeling task type: EVALUATION_RATING on text data. You will not need to provide voice or image data for this role.

How to apply

Create or sign in to your OpenTrain profile and submit an application for this AI Quality Analyst role. Your profile should highlight relevant evaluation, annotation, or prompt-design experience and confirm your availability for the stated schedule.

If selected, OpenTrain AI will contact you with onboarding instructions and the contractor agreement.