Intelligence Analyst, AI Model Evaluation

You will evaluate how well AI systems handle government and defense analysis. The work uses intelligence tradecraft and professional judgment to assess whether model outputs use sources appropriately, reason carefully, express uncertainty, and remain useful in operational settings.

• Map analytic cases to defined taxonomies across several intelligence evaluation areas.
• Write complex test cases covering source handling, analytic rigor, uncertainty, and practical usefulness.
• Review and grade AI-generated military and government reports and related evaluation materials.
• Identify over-refusal when a model declines legitimate analytic exploration, then explain the issue clearly.
• Record edge cases and disagreements to improve evaluation methods.
• Help refine source-evaluation procedures and confidence-level assignments with project leads.
• Share expert insights that help calibrate analytic standards and improve AI benchmarks.

What it pays and takes

This is a remote, part-time contractor role. Prior AI evaluation work can help but is not required.

• Pay: $30 to $80 per hour.
• Location: Open to candidates in multiple countries across North and South America, Europe, Africa, Asia, and the Middle East.
• Language: English proficiency is required.
• Experience: Demonstrated all-source analytic experience or deep expertise in a specific analytic discipline.
• Skills: Strong source evaluation, confidence-level assessment, and knowledge of established analytic standards.
• Knowledge: Familiarity with structured analytic techniques and intelligence tradecraft standards.
• Communication: Exceptional written and verbal communication, with a focus on clarity and precision.
• Valued background: Experience teaching, mentoring, or leading analytic reviews.
• Helpful context: Civilian intelligence community or comparable government and defense analysis experience.
• AI background: Previous participation in AI evaluation or model assessment is helpful, but not required.

How it works

You apply here in about two minutes and finish on the hiring platform. The next step is a short AI interview (technical roles include a coding challenge). Passing it is how you qualify for paid tasks, so take it soon after you are invited.

The hiring platform's application for this job asks for your resume, your LinkedIn profile, your phone number, your location and answers to 8 screening questions.

About AI training work

AI training is the human work behind systems that generate and evaluate text, images, audio, and other outputs. Contributors review examples, rate model responses, and explain errors so AI systems become more accurate and useful; specialized experience matters when the work requires expert judgment. OpenTrain is a platform where people find these projects, build a profile, and apply.