Agreement + Human Eval Metric Papers

Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 12 papers for Agreement + Human Eval Metric Papers. Dominant protocol signals include human evaluation, automatic metrics, LLM-as-judge, with frequent benchmark focus on Retrieval and metric focus on agreement, cost. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.

Top Papers Reporting This Metric

• PreScience: A Benchmark for Forecasting Scientific Contributions

  Anirudh Ajith, Amanpreet Singh, Jay DeYoung, Nadav Kunievsky, Austin C. Kozlowski · Feb 24, 2026

  Human EvalSimulation Env General

  We introduce PreScience -- a scientific forecasting benchmark that decomposes the research process into four interdependent generative tasks: collaborator prediction, prior work selection, contribution generation, and impact prediction.

• Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language

  Toheeb Aduramomi Jimoh, Tabea De Wille, Nikola S. Nikolov · Feb 21, 2026

  Human Eval General

  One annotator pair achieved almost perfect agreement ($κ= 0.8743$; $93.8\%$ raw agreement), exceeding a number of reported benchmarks for English sarcasm research works.

• Validating Political Position Predictions of Arguments

  Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn · Feb 20, 2026

  Human Eval General

  Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation.

• Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems

  Zhangqi Duan, Arnav Kankaria, Dhruv Kartik, Andrew Lan · Feb 19, 2026

  Human Eval LawCoding

  Human evaluation further demonstrates substantial agreement between LLM and expert annotations.

• Supercharging Agenda Setting Research: The ParlaCAP Dataset of 28 European Parliaments and a Scalable Multilingual LLM-Based Classification

  Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić, Nikola Ljubešić · Feb 18, 2026

  Human Eval CodingMultilingual

  Agreement between the LLM and human annotators is comparable to inter-annotator agreement among humans, and the resulting model outperforms existing CAP classifiers trained on manually-annotated but out-of-domain data.

• Are LLMs Ready to Replace Bangla Annotators?

  Md. Najib Hasan, Touseef Hasan, Souvika Sarkar · Feb 18, 2026

  Human Eval General

  In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences.

• BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

  Md. Najib Hasan, Mst. Jannatun Ferdous Rain, Fyad Mohammed, Nazmul Siddique · Feb 16, 2026

  Human Eval Multilingual

  Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity.

• ADAB: Arabic Dataset for Automated Politeness Benchmarking -- A Large-Scale Resource for Computational Sociopragmatics

  Hend Al-Khalifa, Nadia Ghezaiel, Maria Bounnit, Hend Hamed Alhazmi, Noof Abdullah Alfear · Feb 14, 2026

  Human Eval General

  We benchmark 40 model configurations, including traditional machine learning, transformer-based models, and large language models.

• propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale

  Maximilian Idahl, Benedikt Droste, Björn Plüster, Jan Philipp Harries · Feb 12, 2026

  Human Eval Multilingual

  We introduce propella-1, a family of small multilingual LLMs (0.6B, 1.7B, 4B parameters) that annotate text documents across 18 properties organized into six categories: core content, classification, quality and value, audience and purpose,

• Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

  Gaurav Negi, MA Waskow, John McCrae, Paul Buitelaar · Jan 23, 2026

  Human Eval General

  Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications.

• HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

  Laya Iyer, Kriti Aggarwal, Sanmi Koyejo, Gail Heyman, Desmond C. Ong · Jan 9, 2026

  Human EvalLlm As Judge General

  Despite rapid progress in language models, we still lack a clear way to understand how their abilities in these interpersonal domains compare to those of humans.

• Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

  Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab · Oct 27, 2025

  Human EvalAutomatic Metrics Coding

  We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural n

Other Metric Hubs