HFEPX Metric Hub
Updated from current HFEPX corpus (2026-09-02). This page tracks 60 papers for Relevance.
Updated from current HFEPX corpus (2026-09-02). This page tracks 60 papers for Relevance. Use it to compare how relevance is measured across human feedback and evaluation studies.
Useful for shortlisting papers and comparing how this metric is used across different eval setups. Quality band: High .
Metric Coverage
100.0%
60 sampled papers include metric names.
Benchmark Anchoring
28.3%
Papers with explicit dataset/benchmark anchors for fair comparison.
Quality Controls
10.0%
6 papers report calibration/adjudication/IAA controls.
Recommended next step: Use the top metric-reliable papers first, then compare benchmark context in the matrix before drawing conclusions.
Main limitation: Benchmark coverage is good enough for side-by-side comparison, but you should still verify definitions in the source papers.
Ranked for metric reporting completeness and comparability.
Aug 26, 2026 · Citations: 0 · Score: 9.5
Metrics: Recall, Cost · Eval: Simulation Env
Aug 21, 2026 · Citations: 0 · Score: 9.5
Metrics: Accuracy, F1 · Eval: Automatic Metrics
Jun 28, 2026 · Citations: 0 · Score: 8.5
Metrics: Relevance · Eval: Automatic Metrics
Jul 2, 2026 · Citations: 0 · Score: 8.5
Metrics: Accuracy, Spearman · Eval: Automatic Metrics
Jul 2, 2026 · Citations: 0 · Score: 8.5
Metrics: Precision, Relevance · Eval: Automatic Metrics
Aug 31, 2026 · Citations: 0 · Score: 8.0
Metrics: Accuracy, Relevance · Eval: Automatic Metrics
Compare metric, benchmark, and evaluation context side by side.
Gap: Human feedback
Human feedback is present in 12 of 60 papers.
Gap: Quality controls
Quality controls is present in 6 of 60 papers.
Moderate: Benchmarks
Benchmarks is present in 17 of 60 papers.
Strong: Metrics
Metrics is present in 60 of 60 papers.
Gap: Known rater population
Known rater population is present in 5 of 60 papers.
Moderate: Known annotation unit
Known annotation unit is present in 19 of 60 papers.
Top Metrics
Evaluation Modes
Top Benchmarks
Agentic Mix
Gaoming Zhang, Angqing Jiang, Jianchun Song, Kena Qi, Dayao Chen · Aug 31, 2026 · Citations: 0
To address these challenges, we propose Cross-component Hierarchical semantic Alignment for Personalized generative retrieval (CHAP), a novel personalized GR framework from a hierarchical perspective.
Xiaoyang Chen, Jie Liu, Haijin Liang, Haibo Shi, Jin Ma · Aug 31, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Yang Zhao, Chengxiao Dai, Mengying Kou, Yue Xiu, Dusit Niyato · Jan 29, 2026 · Citations: 0
We present SHARD-MEMO, an agentic memory system built on scope-before-routing: metadata predicates first identify the admissible shards, and a learned router then selects a small number of them for shard-local approximate nearest neighbor…
Kazuki Nakayashiki · Aug 26, 2026 · Citations: 0
An agent that inherits a consolidated memory may inherit a constraint that was true when written and has since been withdrawn by a newer authoritative record.
Zhiyuan Li, Linyuan Gao, Xuechun Ding, Hongwei Chen, Yuan Wu · Aug 26, 2026 · Citations: 0
Reusable skill libraries allow large language model (LLM) agents to reuse procedural knowledge across tasks, but they also turn memory access into a challenging retrieval problem.
Jinpu Jiang, Xuan Wu, Wenhao Song, Bo Yang, You Zhou · Aug 26, 2026 · Citations: 0
To address this limitation, we propose ReliableRAG, which, to the best of our knowledge, is the first reliability-driven framework that mitigates deceptive misinformation in multi-hop QA through fine-grained evaluation of individual…
Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson · Aug 21, 2026 · Citations: 0
We propose an Evaluation Agent, middleware that combines Natural Language Inference (NLI) factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index T = 0.4 F + 0.35 C + 0.25 (1 - P ) with a…
Sergiy Lunyakin · Aug 21, 2026 · Citations: 0
An evaluation on FinanceBench (145 questions) compares KDAF against zero-context inference, BM25, concept-weighted lexical retrieval, and ungrounded graph traversal.
Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, Jiawei Zhou · Aug 20, 2026 · Citations: 0
Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience.
Ernest Beta, Odysseas S. Chlapanis, Dimitrios Galanis, Ion Androutsopoulos · Aug 19, 2026 · Citations: 0
We introduce GreekBarRetrieval, a public retrieval benchmark derived from, and complementing GreekBarBench, which did not include retrieval.
Ting-Wei Li, Sirui Chen, Jiaru Zou, Yingbing Huang, Tianxin Wei · Apr 28, 2026 · Citations: 0
Such adaptation often requires iteratively improving the model toward a targeted task, yet collecting high-quality human-labeled data to support this process is costly and difficult to scale.
Aarik Gulaya · May 27, 2026 · Citations: 0
We evaluate the Specification on a prototype benchmark of held-out behavioral predictions scored by a calibrated 5-judge LLM panel.
Enhan Li, Junhao He, Hongyang Du · Aug 13, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Zhili Shen, Craig Macdonald · Aug 13, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Le Zhang, Ke Sun · Aug 12, 2026 · Citations: 0
We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent.
Zhipeng Song, Yizhi Zhou, Xiangyu Kong, Jiulong Jiao, Xuezhou Ye · May 6, 2026 · Citations: 0
CAR converts these confidence changes into coarse precedence constraints and returns the feasible ranking with minimum Kendall distance from the baseline, preserving existing pairwise preferences unless generator-side evidence supports…
Yilin Liu, Rui Meng, Wangze Ni, Jianxin Yan, Heng Cao · Aug 12, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Yuhang Wu, Xiangqing Shen, Fanfan Wang, Cangqi Zhou, Zhen Wu · Apr 2, 2026 · Citations: 0
However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process.
Yi Ren · Jun 28, 2026 · Citations: 0
Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline…
Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng · Jul 2, 2026 · Citations: 0
We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks.
Javier Irigoyen, Roberto Daza, Francisco Jurado, Julian Fierrez, Ruben Tolosana · Jul 2, 2026 · Citations: 0
For each question, the dataset includes an explanation written by a human teacher alongside 11 explanations generated by LLM-simulated teacher profiles associated with distinct pedagogical risks.
ZhiShu Jiang, Haibo Liu, Xin Shen, Guanqiang QI, Chenxi Miao · May 28, 2026 · Citations: 0
Long-term conversational agents are expected to remember past interactions, but memory is useful only when the right evidence is recalled for the right user.
Alexander W. Lee, Benjamin Han, Shayak Sen, Sam Yeom, Ugur Cetintemel · Apr 28, 2026 · Citations: 0
On a benchmark of production-inspired workloads over restaurant review and customer support datasets, Evergreen's optimized configurations occupy the entire cost-quality Pareto frontier.
Ananto Nayan Bala · Jul 1, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Anzhe Xie, Weihang Su, Yujia Zhou, Yiqun Liu, Qingyao Ai · Jun 15, 2026 · Citations: 0
Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline.
Romain Karpinsky, Julien Mozziconacci, Mickaël Delcey · Jun 29, 2026 · Citations: 0
However, systematic benchmark comparisons across these methods remain scarce.
Leonardo S. Goodall, Andrea I. Luppi, Pedro A. M. Mediano · Jun 29, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova, Ivo Petrov, Dimitar I. Dimitrov · Jun 29, 2026 · Citations: 0
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources.
Hafez Abdelghaffar, Ahmed Alansary, Ali Hamdi · Jun 4, 2026 · Citations: 0
Our methodology involves fine-tuning a pre-trained LLM on a benchmark QA dataset to improve its contextual comprehension and answer extraction capabilities.
Ahmed Alansary, Molham Mohamed, Ali Hamdi · Jun 3, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Ahmed Alansary, Ali Hamdi · Jun 3, 2026 · Citations: 0
The generated summaries are then evaluated using automatic evaluation metrics that capture both lexical similarity and semantic relevance.
Yuxin Wang, Paul Thomas, Zhiwei Yu, Yuan Gao, Saeed Hassanpour · Jun 24, 2026 · Citations: 0
Specifically, how they shape an agent's responses under varying conversational contexts and whether they lead to substantively different response behaviors.
Kirill Dubovikov, Omar El Mansouri, Hachem Madmoun, Yanda Li, Sandeep Kumar · Jun 23, 2026 · Citations: 0
Reranker adaptation improves the public Earth Science benchmark by 44% relative and a six-task reasoning-intensive panel by 23%.
Yijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng · Jun 23, 2026 · Citations: 0
Experiments show that AVOC achieves state-of-the-art performance on long-form audio-video benchmarks, surpassing the second-best model by 4.9 and 5.5 points in average accuracy on OmniVideoBench and LVOmniBench, respectively.
Shaswata Mitra, Subash Neupane, Martin Duclos, Sudip Mittal, Aritran Piplai · Aug 26, 2025 · Citations: 0
To address these challenges, we introduce FALCON, an agentic framework for CTI-grounded rule retrieval, generation, and validation.
Mike Riess · May 26, 2026 · Citations: 0
The documents have been sourced from the public self-help pages of four Nordic telecommunications operators and subsequently filtered for person-identifiable information and relevance through a combined LLM and human annotation pipeline.
Darrien McKenzie, Nicklas Hansen, Xiaolong Wang · Jun 18, 2026 · Citations: 0
Empirically, we find that different sampling strategies induce non-trivial tradeoffs between productivity (learning signal), diversity (coverage of the task manifold), and utility (evaluation relevance).
Zhiyuan Peng, Xuyang Wu, Huaixiao Tou, Yi Fang, Yu Gong · Mar 31, 2026 · Citations: 0
LLM-based shopping agents increasingly rely on long purchase histories and multi-turn interactions for personalization, yet naively appending raw history to prompts is often ineffective due to noise, length, and relevance mismatch.
Thibault Bañeras-Roux, Shashi Kumar, Driss Khalil, Sergio Burdisso, Petr Motlicek · Apr 23, 2026 · Citations: 0
Embedding-based semantic metrics are better correlated with human perception, but decoder-based Large Language Models (LLMs) remain underexplored for this task.
Fabian Retkowski, Maike Züfle, Thai Binh Nguyen, Jan Niehues, Alexander Waibel · Feb 9, 2026 · Citations: 0
Despite its relevance, research remains limited and text-based, leaving key questions unresolved about leveraging audio information, handling ASR errors, and transcript-free evaluation.
Jiawei He, Mengyu Shi, Jiawei Liu, Dong Sun, Chunrong Fang · May 22, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Sovandara Chhoun, Pichdara Po, Sereiwathna Ros, Wan-Sup Cho, Saksonita Khoeurn · May 21, 2026 · Citations: 0
For evaluation, we perform 5-fold cross-validation over 18 question-answer pairs.
Sereiwathna Ros, Phannet Pov, Ratanaktepi Chhor, Kimleang Ly, Wan-Sup Cho · May 21, 2026 · Citations: 0
We conduct a two-phase comparative evaluation.
Saksham Sahai Srivastava · May 17, 2026 · Citations: 0
Long-horizon LLM agents rely on persistent memory to support interactions across sessions, yet existing memory systems often retrieve context using semantic similarity or broad history inclusion, treating retrieved memories as uniformly…
Manish Nagaraj, Sakshi Choudhary, Utkarsh Saxena, Deepak Ravikumar, Kaushik Roy · Oct 8, 2025 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Qinhao Chen, Linyang He, Nima Mesgarani · Apr 18, 2026 · Citations: 0
PIE connects Pruning, automatic Interpretation, and interpretation Evaluation, establishing a comprehensive benchmarking environment to systematically measure behavioral fidelity and downstream interpretability under pruning.
Sayed Mohammad Vakilzadeh Hatefi, Maximilian Dreyer, Reduan Achtibat, Patrick Kahardipraja, Thomas Wiegand · Jun 16, 2025 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Ken M. Nakanishi · Apr 1, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Chunyu Li, Jingyi Kang, Ding Chen, Mengyuan Zhang, Jiajun Shen · May 7, 2026 · Citations: 0
In agent memory systems, the reranking model serves as the critical bridge connecting user queries with long-term memory.
Tianyang Han, Hengyu Shi, Junjie Hu, Xu Yang, Zhiling Wang · May 5, 2026 · Citations: 0
Extensive experiments on code and math benchmarks show that this executor-grounded reasoning reward improves the two-stage planner-executor system over execution-only training, suggesting that reasoning supervision should evaluate not only…
Dominik Dahlem, Diego Maniloff, Mac Misiura · May 6, 2026 · Citations: 0
The resulting two-axis diagnostic (φ for capacity, G for direction) yields a falsifiable polarity prediction: bottleneck- and diffuse-dominated benchmarks should exhibit opposite polarity.
Wei Jiang, Wei Wang · Apr 23, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Yilun Zhao, Jinbiao Wei, Tingyu Song, Siyue Zhang, Chen Zhao · May 5, 2026 · Citations: 0
This capability is increasingly important for agentic search systems, where retrievers must provide complementary evidence across iterative search and synthesis.
Vincent Degenhart, Dewi Timman, Arjen P. de Vries, Faegheh Hasibi, Mohanna Hoveyda · May 5, 2026 · Citations: 0
We conduct a reproducibility study to benchmark major retrieval families and reasoning-targeted methods on QUEST and QUEST+Variants, and introduce LIMIT+, a controlled benchmark where relevance depends on arbitrary attribute predicates and…
Negar Arabzadeh, Andrew Drozdov, Michael Bendersky, Matei Zaharia · Apr 24, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Sourav Saha, Mandar Mitra, Aditya Dutta · Jan 13, 2026 · Citations: 0
A good deal of recent research has focused on how Large Language Models (LLMs) may be used as judges in place of humans to evaluate the quality of the output produced by various text / image processing systems.
Bin Wu, Arastun Mammadli, Xiaoyu Zhang, Emine Yilmaz · Apr 24, 2026 · Citations: 0
The rapid growth of AI agent ecosystems is transforming how complex tasks are delegated and executed, creating a new challenge of identifying suitable agents for a given task.
Meghyn Bienvenu, Diego Figueira, Pierre Lafourcade · Apr 24, 2026 · Citations: 0
Relevance has already been shown to be harder than query evaluation: namely, Σ^p_2-complete for CQs, even over a binary signature.
Weixu Zhang, Fanghua Ye, Qiang Gao, Jian Li, Haolun Wu · Apr 24, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
Filippo Ruffini, Camillo Maria Caruso, Claudia Tacconi, Lorenzo Nibid, Francesca Miccolis · Jan 15, 2026 · Citations: 0
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
