AI Evaluation Consultant (AI benchmark review, instruction auditing, oracle validation, agent failure analysis) at Abundant AI
Other
Served as an AI evaluation consultant reviewing and validating benchmark tasks for frontier coding models. Evaluated solution correctness by inspecting task instructions, proposed solutions, patches, and accompanying test suites. Conducted agent failure analysis to diagnose reasoning failures versus infrastructure issues such as timeouts and instruction gaps.• Reviewed benchmark instructions and results for alignment and correctness.• Identified oracle violations, brittle tests, ambiguous requirements, and benchmark quality issues.• Performed agent failure analysis and distinguished failure categories across models.• Validated acceptance criteria and assessed benchmark performance using success/failure metrics.
2025 - Present