You will create realistic, reusable evaluation scenarios for AI agents that perform real-world tasks. You will define the correct path, acceptable agent behavior, expected outputs, edge cases, and scoring rules.
You will also review agent outputs, improve scenarios for clarity and coverage, and work with developers and other contributors to test and refine evaluation frameworks.
- Design structured evaluation scenarios for LLM-based agents.
- Define gold-standard behavior and acceptable alternatives.
- Annotate task steps, expected outputs, edge cases, and scoring logic.
- Review agent responses and revise scenarios to improve coverage.
- Use structured formats such as JSON and YAML.