Agreement + Automatic Metrics Metric Papers

Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 11 papers for Agreement + Automatic Metrics Metric Papers. Dominant protocol signals include automatic metrics, human evaluation, simulation environments, with frequent benchmark focus on Contentbench, GSM8K and metric focus on agreement, accuracy. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.

Top Papers Reporting This Metric

• GATES: Self-Distillation under Privileged Context with Consensus Gating

  Alex Stein, Furong Huang, Tom Goldstein · Feb 24, 2026

  Automatic Metrics Math

  Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.

• Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference

  Arindam Khaled · Feb 23, 2026

  Automatic Metrics Math

  In this work, we propose "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary.

• Can Large Language Models Replace Human Coders? Introducing ContentBench

  Michael Haman · Feb 23, 2026

  Automatic Metrics Coding

  This paper introduces ContentBench, a public benchmark suite that helps answer this replacement question by tracking how much agreement low-cost LLMs achieve and what they cost on the same interpretive coding tasks.

• Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation

  Muhammad Tayyab Khan, Lequn Chen, Wenhe Feng, Seung Ki Moon · Feb 20, 2026

  Automatic MetricsSimulation Env General

  When deterministic scoring cannot resolve an ambiguity, the system escalates to multimodal and constrained large-language-model reasoning, followed by a single human-in-the-loop (HITL) review step.

• Multi-Objective Alignment of Language Models for Personalized Psychotherapy

  Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna, Sriram Sankararaman, Matteo Malgaroli · Feb 17, 2026

  Automatic Metrics Medicine

  While AI systems show therapeutic promise, current alignment approaches optimize objectives independently, failing to balance patient preferences with clinical safety.

• Revisiting Northrop Frye's Four Myths Theory with Large Language Models

  Edirlei Soares de Lima, Marco A. Casanova, Antonio L. Furtado · Feb 17, 2026

  Automatic Metrics General

  Northrop Frye's theory of four fundamental narrative genres (comedy, romance, tragedy, satire) has profoundly influenced literary criticism, yet computational approaches to his framework have focused primarily on narrative patterns rather t

• Mechanistic Indicators of Steering Effectiveness in Large Language Models

  Mehdi Jafari, Hao Xue, Flora Salim · Feb 2, 2026

  Automatic Metrics Coding

  Despite its widespread use, the mechanistic factors that govern when steering succeeds or fails remain poorly understood, as prior work has relied primarily on black-box outputs or LLM-based judges.

• Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters

  Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye · Oct 29, 2025

  Automatic Metrics General

  Large language models (LLMs) are increasingly used as raters for evaluation tasks.

• Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

  Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab · Oct 27, 2025

  Human EvalAutomatic Metrics Coding

  We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural n

• Incentive-Aligned Multi-Source LLM Summaries

  Yanchen Jiang, Zhe Feng, Aranyak Mehta · Sep 29, 2025

  Automatic Metrics General

  Large language models (LLMs) are increasingly used in modern search and answer systems to synthesize multiple, sometimes conflicting, texts into a single response, yet current pipelines offer weak incentives for sources to be accurate and a

• A Scalable Framework for Evaluating Health Language Models

  Neil Mallinar, A. Ali Heydari, Xin Liu, Anthony Z. Faranesh, Brent Winslow · Mar 30, 2025

  Automatic Metrics Medicine

  As LLM-driven health applications are increasingly adopted, rigorous and efficient one-sided evaluation methodologies are crucial to ensure response quality across multiple dimensions, including accuracy, personalization and safety.

Other Metric Hubs