Human Eval Papers

Why This Matters For Eval Research

• Common evaluation patterns here: Human Eval, Automatic Metrics.
• Benchmark signals emphasize: retrieval, AIME.
• Top reported metrics include: agreement, accuracy.

Research Utility Snapshot

Top Papers

• Distill and Align Decomposition for Enhanced Claim Verification

  Jabez Magomere, Elena Kochkina, Samuel Mensah, Simerjot Kaur, Fernando Acero · Feb 25, 2026 · Citations: 0

  Human EvalAutomatic Metrics

  Across six evaluation settings, our trained 8B decomposer improves downstream verification performance to (71.75%) macro-F1, outperforming prompt-based approaches ((+1.99), (+6.24)) and existing RL methods ((+5.84)).

• Improving Implicit Discourse Relation Recognition with Natural Language Explanations from LLMs

  Heng Wang, Changxing Wu · Feb 25, 2026 · Citations: 0

  Human Eval

  Experimental results on PDTB demonstrate that our approach significantly improves IDRR performance, while human evaluation further confirms that the generated explanations enhance model interpretability.

• MixSarc: A Bangla-English Code-Mixed Corpus for Implicit Meaning Identification

  Kazi Samin Yasar Alam, Md Tanbir Chowdhury, Tamim Ahmed, Ajwad Abrar, Md Rafid Haque · Feb 25, 2026 · Citations: 0

  Human EvalAutomatic Metrics

  We benchmark transformer-based models and evaluate zero-shot large language models under structured prompting.

• A Benchmark for Deep Information Synthesis

  Debjit Paul, Daniel Murphy, Milan Gritta, Ronald Cardenas, Victor Prokhorov · Feb 24, 2026 · Citations: 0

  Human EvalAutomatic Metrics Tool Use

  Large language model (LLM)-based agents are increasingly used to solve complex tasks involving tool use, such as web browsing, code execution, and data analysis.

• Pressure Reveals Character: Behavioural Alignment Evaluation at Depth

  Nora Petrova, John Burden · Feb 24, 2026 · Citations: 0

  Human Eval

  While alignment failures increasingly cause real-world harm, comprehensive evaluation frameworks with realistic multi-turn scenarios remain lacking.

• Balancing Multiple Objectives in Urban Traffic Control with Reinforcement Learning from AI Feedback

  Chenyang Zhao, Vinny Cahill, Ivana Dusparic · Feb 24, 2026 · Citations: 0

  Pairwise PreferenceRlaif Or Synthetic Feedback Human Eval

  Preference-based RL offers an appealing alternative by learning from human preferences over pairs of behavioural outcomes.

• CARE: An Explainable Computational Framework for Assessing Client-Perceived Therapeutic Alliance Using Large Language Models

  Anqi Li, Chenxiao Wang, Yu Lu, Renjun Xu, Lizhi Ma · Feb 24, 2026 · Citations: 0

  Human EvalAutomatic Metrics

  Experiments show that CARE outperforms leading LLMs and substantially reduces the gap between counselor evaluations and client-perceived alliance, achieving over 70% higher Pearson correlation with client ratings.

• PreScience: A Benchmark for Forecasting Scientific Contributions

  Anirudh Ajith, Amanpreet Singh, Jay DeYoung, Nadav Kunievsky, Austin C. Kozlowski · Feb 24, 2026 · Citations: 0

  Human EvalSimulation Env

  We introduce PreScience -- a scientific forecasting benchmark that decomposes the research process into four interdependent generative tasks: collaborator prediction, prior work selection, contribution generation, and impact prediction.

• AgenticSum: An Agentic Inference-Time Framework for Faithful Clinical Text Summarization

  Fahmida Liza Piya, Rahmatollah Beheshti · Feb 23, 2026 · Citations: 0

  Human Eval

  We present AgenticSum, an inference-time, agentic framework that separates context selection, generation, verification, and targeted correction to reduce hallucinated content.

• Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language

  Toheeb Aduramomi Jimoh, Tabea De Wille, Nikola S. Nikolov · Feb 21, 2026 · Citations: 0

  Pairwise Preference Human Eval

  One annotator pair achieved almost perfect agreement ($κ= 0.8743$; $93.8\%$ raw agreement), exceeding a number of reported benchmarks for English sarcasm research works.

• Think$^{2}$: Grounded Metacognitive Reasoning in Large Language Models

  Abraham Paul Elenjical, Vivek Hruday Kavuri, Vasudeva Varma · Feb 21, 2026 · Citations: 0

  Pairwise Preference Human Eval

  We introduce a psychologically grounded metacognitive framework that operationalizes Ann Brown's regulatory cycle (Planning, Monitoring, and Evaluation) as a structured prompting architecture, and study its integration within a lightweight

• Validating Political Position Predictions of Arguments

  Jordan Robinson, Angus R. Williams, Katie Atkinson, Anthony G. Cohn · Feb 20, 2026 · Citations: 0

  Pairwise Preference Human Eval

  Real-world knowledge representation often requires capturing subjective, continuous attributes -- such as political positions -- that conflict with pairwise validation, the widely accepted gold standard for human evaluation.

• Vichara: Appellate Judgment Prediction and Explanation for the Indian Judicial System

  Pavithra PM Nair, Preethu Rose Anish · Feb 20, 2026 · Citations: 0

  Human EvalAutomatic Metrics

  Vichara surpasses existing judgment prediction benchmarks on both datasets, with GPT-4o mini achieving the highest performance (F1: 81.5 on PredEx, 80.3 on ILDC_expert), followed by Llama-3.1-8B.

• Using LLMs for Knowledge Component-level Correctness Labeling in Open-ended Coding Problems

  Zhangqi Duan, Arnav Kankaria, Dhruv Kartik, Andrew Lan · Feb 19, 2026 · Citations: 0

  Human Eval

  Human evaluation further demonstrates substantial agreement between LLM and expert annotations.

• Claim Automation using Large Language Model

  Zhengda Mo, Zhiyu Quan, Eli O'Donohue, Kaiwen Zhong · Feb 18, 2026 · Citations: 0

  Human EvalAutomatic Metrics

  We assess this module using a multi-dimensional evaluation framework that combines automated semantic similarity metrics with human evaluation, enabling a rigorous examination of both practical utility and predictive accuracy.

• Supercharging Agenda Setting Research: The ParlaCAP Dataset of 28 European Parliaments and a Scalable Multilingual LLM-Based Classification

  Taja Kuzman Pungeršek, Peter Rupnik, Daniela Širinić, Nikola Ljubešić · Feb 18, 2026 · Citations: 0

  Human Eval

  Agreement between the LLM and human annotators is comparable to inter-annotator agreement among humans, and the resulting model outperforms existing CAP classifiers trained on manually-annotated but out-of-domain data.

• Are LLMs Ready to Replace Bangla Annotators?

  Md. Najib Hasan, Touseef Hasan, Souvika Sarkar · Feb 18, 2026 · Citations: 0

  Human Eval

  In this work, we study the behavior of LLMs as zero-shot annotators for Bangla hate speech, a task where even human agreement is challenging, and annotator bias can have serious downstream consequences.

• Discovering Implicit Large Language Model Alignment Objectives

  Edward Chen, Sanmi Koyejo, Carlos Guestrin · Feb 17, 2026 · Citations: 0

  Rubric Rating Human Eval

  To address these limitations, we introduce Obj-Disco, a framework that automatically decomposes an alignment reward signal into a sparse, weighted combination of human-interpretable natural language objectives.

• FrameRef: A Framing Dataset and Simulation Testbed for Modeling Bounded Rational Information Health

  Victor De Lima, Jiqun Liu, Grace Hui Yang · Feb 17, 2026 · Citations: 0

  Human EvalSimulation Env Long Horizon

  Within this framework, we construct framing-sensitive agent personas by fine-tuning language models with framing-conditioned loss attenuation, inducing targeted biases while preserving overall task competence.

• Multi-Agent Comedy Club: Investigating Community Discussion Effects on LLM Humor Generation

  Shiwei Hong, Lingyao Li, Ethan Z. Rong, Chenxinran Shen, Zhicong Lu · Feb 16, 2026 · Citations: 0

  Pairwise PreferenceRubric Rating Human Eval Multi Agent

  Prior work has explored multi-turn interaction and feedback for LLM writing, but evaluations still largely center on prompts and localized feedback, leaving persistent public reception in online communities underexamined.

• BETA-Labeling for Multilingual Dataset Construction in Low-Resource IR

  Md. Najib Hasan, Mst. Jannatun Ferdous Rain, Fyad Mohammed, Nazmul Siddique · Feb 16, 2026 · Citations: 0

  Human Eval

  Manual annotation is expensive and difficult to scale, while using large language models (LLMs) as automated annotators introduces concerns about label reliability, bias, and evaluation validity.

• Annotation-Efficient Vision-Language Model Adaptation to the Polish Language Using the LLaVA Framework

  Grzegorz Statkiewicz, Alicja Dobrzeniecka, Karolina Seweryn, Aleksandra Krasnodębska, Karolina Piosek · Feb 15, 2026 · Citations: 0

  Human Eval

  Despite relying almost entirely on automatic translation and minimal manual intervention to the training data, our approach yields strong results: we observe a +9.5% improvement over LLaVA-1.6-Vicuna-13B on a Polish-adapted MMBench, along w

• ADAB: Arabic Dataset for Automated Politeness Benchmarking -- A Large-Scale Resource for Computational Sociopragmatics

  Hend Al-Khalifa, Nadia Ghezaiel, Maria Bounnit, Hend Hamed Alhazmi, Noof Abdullah Alfear · Feb 14, 2026 · Citations: 0

  Human Eval

  We benchmark 40 model configurations, including traditional machine learning, transformer-based models, and large language models.

• propella-1: Multi-Property Document Annotation for LLM Data Curation at Scale

  Maximilian Idahl, Benedikt Droste, Björn Plüster, Jan Philipp Harries · Feb 12, 2026 · Citations: 0

  Human Eval

  We introduce propella-1, a family of small multilingual LLMs (0.6B, 1.7B, 4B parameters) that annotate text documents across 18 properties organized into six categories: core content, classification, quality and value, audience and purpose,

• The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage

  Preni Golazizian, Elnaz Rahmati, Jackson Trager, Zhivar Sourati, Nona Ghazizadeh · Feb 10, 2026 · Citations: 0

  Pairwise PreferenceRubric Rating Human Eval

  To this end, we (i) develop a domain-specific evaluation rubric grounded in procedural justice theory, LAPD training materials, and extensive fieldwork; (ii) introduce a rubric-driven preference data construction framework for perspective-c

• Large Language Models as Automatic Annotators and Annotation Adjudicators for Fine-Grained Opinion Analysis

  Gaurav Negi, MA Waskow, John McCrae, Paul Buitelaar · Jan 23, 2026 · Citations: 0

  Human Eval

  Although this level of detail is sound, it requires considerable human effort and substantial cost to annotate opinions in datasets for training models, especially across diverse domains and real-world applications.

• RebuttalAgent: Strategic Persuasion in Academic Rebuttal via Theory of Mind

  Zhitao He, Zongwei Lyu, Yi R Fung · Jan 22, 2026 · Citations: 0

  Pairwise PreferenceCritique Edit Human Eval

  In this paper, we introduce RebuttalAgent, the first framework to ground academic rebuttal in Theory of Mind (ToM), operationalized through a ToM-Strategy-Response (TSR) framework that models reviewer mental state, formulates persuasion str

• HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

  Laya Iyer, Kriti Aggarwal, Sanmi Koyejo, Gail Heyman, Desmond C. Ong · Jan 9, 2026 · Citations: 0

  Pairwise PreferenceRubric Rating Human EvalLlm As Judge

  Despite rapid progress in language models, we still lack a clear way to understand how their abilities in these interpersonal domains compare to those of humans.

• Beyond Understanding: Evaluating the Pragmatic Gap in LLMs' Cultural Processing of Figurative Language

  Mena Attia, Aashiq Muhamed, Mai Alkhamissi, Thamar Solorio, Mona Diab · Oct 27, 2025 · Citations: 0

  Human EvalAutomatic Metrics

  We present a comprehensive evaluation of the ability of large language models (LLMs) to process culturally grounded language, specifically to understand and pragmatically use figurative expressions that encode local knowledge and cultural n

• Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

  Ran Xu, Jingjing Chen, Jiayu Ye, Yu Wu, Jun Yan · Oct 27, 2025 · Citations: 0

  Pairwise Preference Human Eval

  Large Language Models (LLMs) are widely used as judges to evaluate response quality, providing a scalable alternative to human evaluation.

• Estonian Native Large Language Model Benchmark

  Helena Grete Lillepalu, Tanel Alumäe · Oct 24, 2025 · Citations: 0

  Human Eval

  The availability of LLM benchmarks for the Estonian language is limited, and a comprehensive evaluation comparing the performance of different LLMs on Estonian tasks has yet to be conducted.

• Peeking inside the Black-Box: Reinforcement Learning for Explainable and Accurate Relation Extraction

  Xinyu Guo, Zhengliang Shi, Minglai Yang, Mahdi Rahimi, Mihai Surdeanu · Oct 7, 2025 · Citations: 0

  Human EvalAutomatic Metrics

  Finally, human evaluation shows that our best model generates relational keywords closely aligned with gold labels, increasing human explanation quality ratings by 54% (relative).

• PoeTone: A Framework for Constrained Generation of Structured Chinese Songci with LLMs

  Zhan Qu, Shuzhou Yuan, Michael Färber · Aug 4, 2025 · Citations: 0

  Human Eval

  We first develop a comprehensive, multi-faceted evaluation framework that includes: (i) a formal conformity score, (ii) automated quality assessment using LLMs, (iii) human evaluation, and (iv) classification-based probing tasks.

• EuroGEST: Investigating gender stereotypes in multilingual language models

  Jacqueline Rowe, Mateusz Klimaszewski, Liane Guillou, Shannon Vallor, Alexandra Birch · Jun 4, 2025 · Citations: 0

  Human EvalAutomatic Metrics

  Large language models increasingly support multiple languages, yet most benchmarks for gender bias remain English-centric.

• Measuring the Measurers: Quality Evaluation of Hallucination Benchmarks for Large Vision-Language Models

  Bei Yan, Jie Zhang, Zheng Yuan, Shiguang Shan, Xilin Chen · Jun 24, 2024 · Citations: 0

  Human Eval

  While previous works have proposed various benchmarks to evaluate this issue, the quality of these evaluations remains unverified.

• Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questions

  Hanjie Chen, Zhouxiang Fang, Yash Singla, Mark Dredze · Feb 28, 2024 · Citations: 0

  Human Eval

  Experiments demonstrate that our datasets are harder than previous benchmarks.

Related Hubs

• General Papers Hub
• Agreement Metric Papers Metric
• Automatic Metrics Papers Hub
• Coding Papers Hub
• Pairwise Preference + Human Eval Papers Hub
• Pairwise Preference Papers Hub