About OpenTrain
OpenTrain is the centralized, open platform where AI training and data-labeling freelancers find work, consolidate opportunities across platforms, and build a unified AI training portfolio they control. We make specialized AI training work easier to track, apply for, and grow into a durable freelance career.
About this role
OpenTrain is recruiting for an AI Quality Analyst to evaluate a new personalization feature for Gemini. In this role, you will assess how well the model uses information from your past Gemini conversations, Gmail, Google Search, and YouTube activity to make responses more relevant and helpful. This is a remote, contract position focused on Thai language evaluation.
What you will do
- Design and execute multi-turn conversational prompts (1-5 turns) that require the AI to utilize your personal information and experiences
- Evaluate model responses based on your intent from the starting prompt, checking for appropriate personalization
- Analyze responses for Grounding issues, Integration quality, and overall Helpfulness
- Rigorously evaluate and stack-rank two model responses side-by-side to determine which is more helpful, easy to use, and enjoyable
- Write clear, defensible rationales for your comparisons, referencing specific turn numbers
- Extract and verify debug info to confirm that chat summaries and data sources were properly utilized
- Maintain strict data hygiene by deleting evaluation conversations to prevent contamination of future chat history
Required skills
- Thai Proficiency: Ability to read and write in Thai with a high degree of competence, as Thai is the focus language for this project
- Exceptional Analytical Thinking: Demonstrate ability to evaluate nuanced and ambiguous AI responses, specifically assessing personalization quality
- Creative Prompt Engineering: Experience in designing creative, multi-turn starting prompts based on personal context to thoroughly test the model's capabilities
- Strong Evaluation Acumen: Understanding of personalization concepts, including the ability to identify incorrect personalization, poor inferences, and forced connections
- Meticulous Attention to Detail: The ability to review Side-by-Side (SxS) model responses and spot subtle differences in naturalness and overnarrating
- Excellent Written Communication: Superior ability to write clear, concise, and structured rationales for model rankings
- Willingness to use your primary personal Google account and enable personal data sources for a genuine assessment
- Desktop/Laptop with a good internet connection
Helpful background
- BS/BA degree or equivalent experience in a relevant field (e.g., Policy, Law, Ethics, Linguistics, Journalism, Computer Science, or a related analytical field)
- Experience in data annotation, AI quality evaluation, content moderation, or a related role