Agreement Metric Papers

Research Utility Snapshot

Top Papers Reporting This Metric

• GATES: Self-Distillation under Privileged Context with Consensus Gating

  Alex Stein, Furong Huang, Tom Goldstein · Feb 24, 2026

  Automatic Metrics Math

  Held-out in-domain accuracy under asymmetric evaluation improves from 46.0\% to 62.0\%, and average (maj@8) accuracy on public document-free math benchmarks improves from 20.2\% to 35.4\%.

• MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

  Zhenyu Wang, Xiaofen Xing, Yirong Chen, Xiangmin Xu · Feb 24, 2026

  Llm As Judge Coding

  Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions.

• PreScience: A Benchmark for Forecasting Scientific Contributions

  Anirudh Ajith, Amanpreet Singh, Jay DeYoung, Nadav Kunievsky, Austin C. Kozlowski · Feb 24, 2026

  Human EvalSimulation Env General

  We introduce PreScience -- a scientific forecasting benchmark that decomposes the research process into four interdependent generative tasks: collaborator prediction, prior work selection, contribution generation, and impact prediction.

• Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference

  Arindam Khaled · Feb 23, 2026

  Automatic Metrics Math

  In this work, we propose "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary.

• Can Large Language Models Replace Human Coders? Introducing ContentBench

  Michael Haman · Feb 23, 2026

  Automatic Metrics Coding

  This paper introduces ContentBench, a public benchmark suite that helps answer this replacement question by tracking how much agreement low-cost LLMs achieve and what they cost on the same interpretive coding tasks.

• Validating Political Position Predictions of Arguments

  Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn · Feb 20, 2026

  Human Eval General

  Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation.

• Context-Aware Mapping of 2D Drawing Annotations to 3D CAD Features Using LLM-Assisted Reasoning for Manufacturing Automation

  Muhammad Tayyab Khan, Lequn Chen, Wenhe Feng, Seung Ki Moon · Feb 20, 2026

  Automatic MetricsSimulation Env General

  When deterministic scoring cannot resolve an ambiguity, the system escalates to multimodal and constrained large-language-model reasoning, followed by a single human-in-the-loop (HITL) review step.

• Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems

  Zhangqi Duan, Arnav Kankaria, Dhruv Kartik, Andrew Lan · Feb 19, 2026

  Human Eval LawCoding

  Human evaluation further demonstrates substantial agreement between LLM and expert annotations.

• Supercharging Agenda Setting Research: The ParlaCAP Dataset of 28 European Parliaments and a Scalable Multilingual LLM-Based Classification

  Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić, Nikola Ljubešić · Feb 18, 2026

  Human Eval CodingMultilingual

  Agreement between the LLM and human annotators is comparable to inter-annotator agreement among humans, and the resulting model outperforms existing CAP classifiers trained on manually-annotated but out-of-domain data.

• Are LLMs Ready to Replace Bangla Annotators?

  Md. Najib Hasan, Touseef Hasan, Souvika Sarkar · Feb 18, 2026

  Human Eval General

  In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences.

• Multi-Objective Alignment of Language Models for Personalized Psychotherapy

  Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna, Sriram Sankararaman, Matteo Malgaroli · Feb 17, 2026

  Automatic Metrics Medicine

  While AI systems show therapeutic promise, current alignment approaches optimize objectives independently, failing to balance patient preferences with clinical safety.

• Revisiting Northrop Frye's Four Myths Theory with Large Language Models

  Edirlei Soares de Lima, Marco A. Casanova, Antonio L. Furtado · Feb 17, 2026

  Automatic Metrics General

  Northrop Frye's theory of four fundamental narrative genres (comedy, romance, tragedy, satire) has profoundly influenced literary criticism, yet computational approaches to his framework have focused primarily on narrative patterns rather t

• BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

  Md. Najib Hasan, Mst. Jannatun Ferdous Rain, Fyad Mohammed, Nazmul Siddique · Feb 16, 2026

  Human Eval Multilingual

  Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity.

• ADAB: Arabic Dataset for Automated Politeness Benchmarking -- A Large-Scale Resource for Computational Sociopragmatics

  Hend Al-Khalifa, Nadia Ghezaiel, Maria Bounnit, Hend Hamed Alhazmi, Noof Abdullah Alfear · Feb 14, 2026

  Human Eval General

  We benchmark 40 model configurations, including traditional machine learning, transformer-based models, and large language models.

• propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale

  Maximilian Idahl, Benedikt Droste, Björn Plüster, Jan Philipp Harries · Feb 12, 2026

  Human Eval Multilingual

  We introduce propella-1, a family of small multilingual LLMs (0.6B, 1.7B, 4B parameters) that annotate text documents across 18 properties organized into six categories: core content, classification, quality and value, audience and purpose,

• DCoPilot: Generative AI-Empowered Policy Adaptation for Dynamic Data Center Operations

  Minghao Li, Ruihang Wang, Rui Tan, Yonggang Wen · Feb 2, 2026

  Simulation Env General

  However, manually designing piecewise deep reinforcement learning (DRL) agents cannot keep pace with frequent dynamics shifts and service-level agreement (SLA) changes of an evolving DC.

• Mechanistic Indicators of Steering Effectiveness in Large Language Models

  Mehdi Jafari, Hao Xue, Flora Salim · Feb 2, 2026

  Automatic Metrics Coding

  Despite its widespread use, the mechanistic factors that govern when steering succeeds or fails remain poorly understood, as prior work has relied primarily on black-box outputs or LLM-based judges.

• Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

  Gaurav Negi, MA Waskow, John McCrae, Paul Buitelaar · Jan 23, 2026

  Human Eval General

  Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications.

• HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

  Laya Iyer, Kriti Aggarwal, Sanmi Koyejo, Gail Heyman, Desmond C. Ong · Jan 9, 2026

  Human EvalLlm As Judge General

  Despite rapid progress in language models, we still lack a clear way to understand how their abilities in these interpersonal domains compare to those of humans.

• Through the Judge's Eyes: Inferred Thinking Traces Improve Reliability of LLM Raters

  Xingjian Zhang, Tianhong Gao, Suliang Jin, Tianhao Wang, Teng Ye · Oct 29, 2025

  Automatic Metrics General

  Large language models (LLMs) are increasingly used as raters for evaluation tasks.

• Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

  Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab · Oct 27, 2025

  Human EvalAutomatic Metrics Coding

  We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural n

• Incentive-Aligned Multi-Source LLM Summaries

  Yanchen Jiang, Zhe Feng, Aranyak Mehta · Sep 29, 2025

  Automatic Metrics General

  Large language models (LLMs) are increasingly used in modern search and answer systems to synthesize multiple, sometimes conflicting, texts into a single response, yet current pipelines offer weak incentives for sources to be accurate and a

• A Scalable Framework for Evaluating Health Language Models

  Neil Mallinar, A. Ali Heydari, Xin Liu, Anthony Z. Faranesh, Brent Winslow · Mar 30, 2025

  Automatic Metrics Medicine

  As LLM-driven health applications are increasingly adopted, rigorous and efficient one-sided evaluation methodologies are crucial to ensure response quality across multiple dimensions, including accuracy, personalization and safety.

• Human-like Affective Cognition in Foundation Models

  Kanishk Gandhi, Zoe Lynch, Jan-Philipp Fränken, Kayla Patterson, Sharon Wambu · Sep 18, 2024

  Llm As Judge General

  Understanding emotions is fundamental to human interaction and experience.

Other Metric Hubs