LLM Safety Evaluator, Hebrew and English

About OpenTrain

OpenTrain AI is the hiring and contracting organization for this fully remote role. OpenTrain is the #1 platform for finding and building careers in AI training and data labeling, helping people discover meaningful projects, build their profiles, and apply in minutes.

• Hourly contractor position
• Fully remote and open worldwide
• Part-time schedule of 20+ hours per week
• Pay range of $26-$38 per hour

About AI Safety Evaluation

AI training is the human side of building artificial intelligence. Evaluators review model responses, create examples, and provide feedback that helps large language models become more accurate, useful, and safe.

In this fast-growing field, your work will focus on red-teaming and safety evaluation. Your judgments will help identify harmful behavior, clarify policy gaps, and shape how advanced AI systems respond to sensitive requests.

• Work directly with large language model outputs
• Help improve model safety through structured human feedback
• Contribute to cutting-edge AI development from anywhere

The Role

As an LLM Safety Evaluator, you will review AI-generated responses and create safety-focused evaluation content in both Hebrew and English. You will assess whether outputs are accurate, safe, and clearly explained, including in ambiguous or adversarial scenarios.

Some assignments will involve explicit, toxic, violent, sexual, or psychologically disturbing material as part of daily work. Your feedback and evaluations will directly support the training and safety of large language models.

• Language requirement: near-native or native Hebrew reading and writing
• English requirement: minimum C1 proficiency in reading and writing
• Experience level: Intermediate
• Data type: Text
• Workload: 20+ hours per week

What You’ll Do

You will curate and label adversarial or safety-sensitive training examples, review and score model outputs, and document safety failures. You will also stress-test models to uncover policy gaps and explain evaluation decisions consistently.

The work requires careful analysis across a broad range of safety categories, while maintaining clear documentation in a multilingual evaluation environment.

• Review and score AI-generated responses
• Generate safety-focused evaluation content in Hebrew and English
• Curate and label adversarial and safety-sensitive examples
• Document model safety failures and recurring adversarial patterns
• Probe safety boundaries through hands-on LLM red teaming
• Stress-test models for policy gaps
• Evaluate hate and harassment, sexual content, suicide and self-harm, violence, and bias
• Evaluate illegal goods or services, malicious activities, malicious code, and misinformation

Requirements

This role is intended for an experienced safety or trust and safety professional who can apply written policies consistently and communicate decisions clearly. You should be comfortable making careful judgments when cases are ambiguous and when content is sensitive or disturbing.

• Bachelor’s degree or higher in Communications, Linguistics, Psychology, Law or Policy, Security Studies, or a related field, or equivalent professional experience
• Proven experience in Trust & Safety, content moderation, policy enforcement, risk operations, investigations, or safety evaluation
• Required hands-on LLM red-teaming experience
• Strong knowledge of the listed AI safety categories
• Ability to apply written safety policies consistently
• Ability to explain evaluation decisions clearly in ambiguous cases
• Comfort reviewing explicit, toxic, violent, sexual, or psychologically disturbing content daily
• Strong practical experience with Perplexity, Gemini, ChatGPT, or similar AI systems

Preferred Experience

Prior experience with AI data training, annotation, or evaluation workflows is preferred. Familiarity with these processes can help you move efficiently between content review, labeling, scoring, and written feedback.

• Prior AI data training experience
• Prior data annotation experience
• Prior AI evaluation workflow experience

How to Apply

Create a free OpenTrain account to build your profile and apply in minutes. If selected, you will work with OpenTrain AI as a remote, part-time contractor supporting multilingual LLM safety evaluation.

• Apply through OpenTrain
• Work remotely from anywhere worldwide
• Choose a flexible part-time workload of 20+ hours per week
• Earn $26-$38 per hour based on the stated project pay range