- Glance: The Generator Is Its Own Reranker
Adilet Metinov, Chingiz Arziev · Oct 29, 2026 · Citations: 0
Automatic Metrics General
These initial results demonstrate generator-based reranking with one resident backbone; evaluation currently covers two retrieval datasets, one QA dataset and one backbone family.
- It Takes Two to Probe, Thousands to Retrieve: Putting Vision--Language Probes to the Retrieval Test
Ivana Beňová, Michal Gregor, Marián Šimko · Oct 29, 2026 · Citations: 0
Automatic Metrics General
We measure how these two evaluation methods relate by using two probing benchmarks, and SugarCrepe, for retrieval tasks.
- Contextual Embeddings Beyond the Gold Chunk
Sedigheh Eslami, Markus Krimmel, William Barber, Antoine Chaffin, Maximilian Schall · Oct 28, 2026 · Citations: 0
Automatic Metrics General
On the evaluation side, we introduce two benchmarks that label both the chunks that answer a query and the evidence needed to verify the answer.
- SpectRAG: Weighted Chunk Graphs for Token-Budgeted Context Selection in Multi-Hop Retrieval-Augmented Generation
Zafer Sahinoglu, Abdulazeez Mohammed, Buse Yaren Kazangirler, Cahit Berkay Kazangirler · Oct 27, 2026 · Citations: 0
General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- Temporal Evidence-Parameter Co-Alignment for LLM Reasoning over Temporal Knowledge Graphs
Jiapu Wang · Oct 20, 2026 · Citations: 0
Automatic Metrics General
Experimental results on two TKG benchmarks demonstrate that TEPCA achieves state-of-the-art performance.
- PsychoSafe: Eliciting Psychologically-Informed Refusals in Large Language Models
Gianluca Barmina, Federico Torrielli, Sven Harms, Jacob Nielsen, Flavia Filesi · Jun 8, 2026 · Citations: 0
Automatic Metrics General
On a balanced validation set of 500 prompts, evaluated with an LLM judge and validated through human ratings, PsychoSafe prompting improves overall refusal quality by 28.1% over a generic baseline, with particularly strong gains in external…
- RIFT: KV Cache Compression for Draft Models with Complementary Compressibility Training
Dor Tsur, Maor Ashkenazi, Maximilian Golub, Peter Hicks, Yaniv Galron · Oct 22, 2026 · Citations: 0
Automatic Metrics General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- Where and Why Search-Augmented LLMs Fail When Reasoning over Rapidly Evolving Knowledge
Tianyi Hu, Bhavyajeet Singh, Niket Tandon, Akhil Arora · Oct 18, 2026 · Citations: 0
Automatic Metrics General
Existing evaluations treat information seeking as a black box, revealing little about where and why LLMs fail.
- MemBubble: Benchmarking How Agent Memory Shapes LLM Response Diversity
Jiajun Tan, Fei Sun, Xinyu Ma, Shixuan Zhang, Du Su · Oct 18, 2026 · Citations: 0
Automatic Metrics General
Memory serves as the core component of LLM-based agents, enabling personalized responses via extracting and using user preferences from past interactions.
- DeGMatch: Target-Aware Reasoning Data Selection via Deconfounded Gradient Matching
Changho Lee, Janghoon Han, Rajdeep Mukherjee, Yeda Song, Lening Nick Cui · Oct 18, 2026 · Citations: 0
MathCoding
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- RAG Strategies for Natural Language-Based SQL Query and REST API Call Generation
Tim Schlippe, Simon Martin, Michael Marketsmüller · Feb 6, 2026 · Citations: 0
Automatic Metrics General
We evaluate three training-free RAG configurations - Standard RAG, Relevance Filtering RAG, and Iterative Retrieval RAG - for SQL query generation, REST API call generation, and a combined task requiring routing between both operation…
- Safety Kernel: Mechanistic Alignment for Cross-Lingual Defense in LLMs
Chengyu Xie, Fei Shen, Shuyi Miao, Yue Feng, Jian Yu · Oct 17, 2026 · Citations: 0
Multilingual
The safety capabilities of large language models (LLMs) are highly uneven across different languages, often allowing cross-lingual attacks to bypass safety guardrails.
- Break It Down, Pass It On: Cross-Task Skill Transfer in LLM Agents
Yiyang Feng, Biddut Sarker Bijoy, Niranjan Balasubramanian, Jiawei Zhou · Aug 20, 2026 · Citations: 0
Automatic Metrics Coding
Large language model (LLM) agents can induce skills from completed tasks and reuse them later to grow more capable with experience.
- Measuring AI "Slop" in Text
Chantal Shaib, Tuhin Chakrabarty, Diego Garcia-Olano, Byron C. Wallace · Sep 23, 2025 · Citations: 0
Automatic Metrics General
Our framework can be used to examine AI-generated text in both detection and binary preference tasks, potentially offering new insights into the linguistic and stylistic factors that contribute to quality judgments.
- Lychee-Reranker: Fast but Not Late Interaction for Compressed Document Reranking
Xinping Zhao, Jiaxin Xu, Ziqi Dai, Xin Zhang, Huiyao Chen · Jun 22, 2026 · Citations: 0
Multilingual
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- Leveraging a four-quadrant approach for evaluating Redpine Science
Filip Dorm, Leonora Vesterbacka · Oct 6, 2026 · Citations: 0
Automatic Metrics General
On ScholarQABench SciFact, the public answer-quality benchmark reported here, an agent with Redpine Science answers 94.4% of claims correctly against 87.6% with no retrieval.
- Does Steering Break Your Model? A Multi-Dimensional Evaluation Suite for LLM Steering Methods
Haotian Yang, Huikang Jiang, Yucheng Wu, Wen-Jie Jiang, Chenpeng Wang · Oct 6, 2026 · Citations: 0
Automatic Metrics General
Existing evaluations cover these dimensions only in fragments.
- Disentangling Models from Personas in Heterogeneous LLM Simulations
Dani Roytburg, Daphne Ippolito · Oct 6, 2026 · Citations: 0
Simulation Env General
Multi-agent simulations with large language models (LLMs) often operate networks of agents with a single base model.
- CoLMbo-SV: A Grounded Language Model for Explainable Speaker Verification
Massa Baali, Sarthak Bisht, Ziyue Qiu, Joseph Konan, Rita Singh · Sep 27, 2026 · Citations: 0
Automatic Metrics General
We also develop an evaluation framework that separates what acoustic information a speaker representation encodes, what influences the verification score, and what the generated report discusses.
- Building Interpretable Feature Representations for Resume-Vacancy Matching by Distilling Production LLM Signals
Ilya Chekin, Vyacheslav Malyugin, Vladimir Chirkov, Mikhail Yurushkin · Oct 2, 2026 · Citations: 0
Human Eval General
This is operational, non-blinded agreement rather than an independent human evaluation.
- Screening Is Enough
Ken M. Nakanishi · Apr 1, 2026 · Citations: 0
General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- ContextRender: From Execution Dependencies to Agent Context
Savini Kashmira, Jayanaka L. Dantanarayana, Lingjia Tang, Jason Mars · Sep 29, 2026 · Citations: 0
Automatic Metrics General
LLM agents performing long-horizon tasks accumulate tool results that later steps may need.
- Bridging Semantic Gaps in RAG through Generated Context Knowledge Fusion
Xinkai Du, Chao Lv, Yalin Sun, Quanjie Han, Lei Yao · Sep 29, 2026 · Citations: 0
Automatic Metrics General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- EpiKV: Epiphany-Aware KV Cache Eviction Without the Attention Matrix
Steven Kolawole, Virginia Smith · Jun 25, 2026 · Citations: 0
Automatic Metrics General
On multiple benchmarks, EpiKV matches or outperforms the strongest attention-based eviction baselines while running directly in vLLM with unmodified attention kernels.
- CoEM: Empowering Long-Context Reasoning with Commit-on-Evidence Memory
Jingguang Li, Yebo Wu, Zuyi Guo, Kailang Ma, Xianjie Dai · Sep 29, 2026 · Citations: 0
Automatic Metrics Coding
In this paper, we introduce Commit-on-Evidence Memory (CoEM), which learns when to convert source evidence into compact memory facts.
- Preference Shapes Relevance: Cross-component Hierarchical Semantic Alignment for Personalized Generative Retrieval
Gaoming Zhang, Angqing Jiang, Jianchun Song, Kena Qi, Dayao Chen · Aug 31, 2026 · Citations: 0
Coding
To address these challenges, we propose Cross-component Hierarchical semantic Alignment for Personalized generative retrieval (CHAP), a novel personalized GR framework from a hierarchical perspective.
- Beyond Polarization: The Generative Constraint of Chain-of-Thought in Pointwise Reranking
Xiaoyang Chen, Jie Liu, Haijin Liang, Haibo Shi, Jin Ma · Aug 31, 2026 · Citations: 0
Automatic Metrics General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- ShardMemo: Scope-Before-Routing for Agentic Memory Retrieval
Yang Zhao, Chengxiao Dai, Mengying Kou, Yue Xiu, Dusit Niyato · Jan 29, 2026 · Citations: 0
Automatic Metrics Law
We present SHARD-MEMO, an agentic memory system built on scope-before-routing: metadata predicates first identify the admissible shards, and a learned router then selects a small number of them for shard-local approximate nearest neighbor…
- When Stale Constraints Go Unchecked: Budgeted Verification Failures in Inherited Agent Memory
Kazuki Nakayashiki · Aug 26, 2026 · Citations: 0
General
An agent that inherits a consolidated memory may inherit a constraint that was true when written and has since been withdrawn by a newer authoritative record.
- CaSKG: Counterfactual-Causal Skill Graphs for Scalable Agent Skill Retrieval
Zhiyuan Li, Linyuan Gao, Xuechun Ding, Hongwei Chen, Yuan Wu · Aug 26, 2026 · Citations: 0
Simulation Env Coding
Reusable skill libraries allow large language model (LLM) agents to reuse procedural knowledge across tasks, but they also turn memory access into a challenging retrieval problem.
- ReliableRAG: Combating Misinformation in Retrieval-Augmented Generation via Reliability-Guided Reasoning Chains
Jinpu Jiang, Xuan Wu, Wenhao Song, Bo Yang, You Zhou · Aug 26, 2026 · Citations: 0
Automatic Metrics General
To address this limitation, we propose ReliableRAG, which, to the best of our knowledge, is the first reliability-driven framework that mitigates deceptive misinformation in multi-hop QA through fine-grained evaluation of individual…
- Trustworthy RAG: An Evaluation Agent for Detecting Misinformation and Knowledge Poisoning in Generative AI Systems
Balkrishna Giri, Md Toufique Hasan, Jussi Rasku, Muhammad Waseem, Pekka Abrahamsson · Aug 21, 2026 · Citations: 0
Automatic Metrics Coding
We propose an Evaluation Agent, middleware that combines Natural Language Inference (NLI) factual verification, a five-signal poison detector with relevance-weighted aggregation, and a Trust Index T = 0.4 F + 0.35 C + 0.25 (1 - P ) with a…
- Auditable by Construction: An Ontology-Driven Framework for Trustworthy LLM Analytics in Enterprise Finance
Sergiy Lunyakin · Aug 21, 2026 · Citations: 0
Automatic Metrics General
An evaluation on FinanceBench (145 questions) compares KDAF against zero-context inference, BM25, concept-weighted lexical retrieval, and ungrounded graph traversal.
- GreekBarRetrieval: A Benchmark for Greek Statutory Retrieval
Ernest Beta, Odysseas S. Chlapanis, Dimitrios Galanis, Ion Androutsopoulos · Aug 19, 2026 · Citations: 0
Automatic Metrics LawMultilingual
We introduce GreekBarRetrieval, a public retrieval benchmark derived from, and complementing GreekBarBench, which did not include retrieval.
- EvoSelect: Data-Efficient LLM Evolution for Targeted Task Adaptation
Ting-Wei Li, Sirui Chen, Jiaru Zou, Yingbing Huang, Tianxin Wei · Apr 28, 2026 · Citations: 0
Automatic Metrics General
Such adaptation often requires iteratively improving the model toward a targeted task, yet collecting high-quality human-labeled data to support this process is costly and difficult to scale.
- Beyond Recall: Behavioral Specification as an Interpretive Layer for AI Personalization
Aarik Gulaya · May 27, 2026 · Citations: 0
Automatic Metrics General
We evaluate the Specification on a prototype benchmark of held-out behavioral predictions scored by a calibrated 5-judge LLM panel.
- CROP: Task Relevance via Counterfactuals for Selective On-Policy Distillation
Enhan Li, Junhao He, Hongyang Du · Aug 13, 2026 · Citations: 0
General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- GEM: A Generative Embedding Model Bridging Reasoning and Retrieval
Zhili Shen, Craig Macdonald · Aug 13, 2026 · Citations: 0
Automatic Metrics Coding
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- EgoCITE: Context-Augmented Indexing and Time-Aware Retrieval for Long-Horizon Egocentric Memory
Le Zhang, Ke Sun · Aug 12, 2026 · Citations: 0
Automatic Metrics General
We demonstrate two bottlenecks in existing systems: indices built from context-poor captions are unreliable for agentic search, while retrieval ignores a question's temporal intent.
- CAR: Query-Guided Confidence-Aware Reranking for Retrieval-Augmented Generation
Zhipeng Song, Yizhi Zhou, Xiangyu Kong, Jiulong Jiao, Xuezhou Ye · May 6, 2026 · Citations: 0
Automatic Metrics General
CAR converts these confidence changes into coarse precedence constraints and returns the feasible ranking with minimum Kendall distance from the baseline, preserving existing pairwise preferences unless generator-side evidence supports…
- QV-PIC: Query-Aware Visual Position-Independent Caching for Efficient RAG Serving
Yilin Liu, Rui Meng, Wangze Ni, Jianxin Yan, Heng Cao · Aug 12, 2026 · Citations: 0
Automatic Metrics General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
Yuhang Wu, Xiangqing Shen, Fanfan Wang, Cangqi Zhou, Zhen Wu · Apr 2, 2026 · Citations: 0
Automatic Metrics General
However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process.
- mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
Yi Ren · Jun 28, 2026 · Citations: 0
Automatic Metrics Medicine
Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline…
- PACE: A Proxy for Agentic Capability Evaluation
Yueqi Song, Lintang Sutawika, Jiarui Liu, Lindia Tjuatja, Jiayi Geng · Jul 2, 2026 · Citations: 0
Automatic Metrics Coding
We introduce PACE, a framework that constructs proxy benchmarks by selecting instances from existing non-agentic evaluations whose aggregate scores most reliably predict model performances on agentic benchmarks.
- AIriskEval-edu: New Dataset for Risk Assessment in AI-mediated K-12 Educational Explanations
Javier Irigoyen, Roberto Daza, Francisco Jurado, Julian Fierrez, Ruben Tolosana · Jul 2, 2026 · Citations: 0
Automatic Metrics General
For each question, the dataset includes an explanation written by a human teacher alongside 11 explanations generated by LLM-simulated teacher profiles associated with distinct pedagogical risks.
- Learning User-Aware Recall: Personalized Retrieval in Long-Term Conversational Memory
ZhiShu Jiang, Haibo Liu, Xin Shen, Guanqiang QI, Chenxi Miao · May 28, 2026 · Citations: 0
General
Long-term conversational agents are expected to remember past interactions, but memory is useful only when the right evidence is recalled for the right user.
- Evergreen: Efficient Claim Verification for Semantic Aggregates
Alexander W. Lee, Benjamin Han, Shayak Sen, Sam Yeom, Ugur Cetintemel · Apr 28, 2026 · Citations: 0
Automatic Metrics General
On a benchmark of production-inspired workloads over restaurant review and customer support datasets, Evergreen's optimized configurations occupy the entire cost-quality Pareto frontier.
- What Survives Into Context: A Diagnostic for Budget-Constrained Multi-Hop RAG and When Submodular Evidence Packing Improves It
Ananto Nayan Bala · Jul 1, 2026 · Citations: 0
Automatic Metrics General
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- Benchmarking LLM Agents on Meta-Analysis Articles from Nature Portfolio
Anzhe Xie, Weihang Su, Yujia Zhou, Yiqun Liu, Qingyao Ai · Jun 15, 2026 · Citations: 0
Automatic Metrics General
Its structured, verifiable workflow makes it an ideal substrate for evaluating systematic scientific reasoning, yet existing benchmarks lack ground truth across the full retrieval-screening-synthesis pipeline.
- DNA Language Models: An Assessment of Pre-Training for Fine-Tuning Tasks
Romain Karpinsky, Julien Mozziconacci, Mickaël Delcey · Jun 29, 2026 · Citations: 0
Automatic Metrics General
However, systematic benchmark comparisons across these methods remain scarce.
- Information Dynamics of Language Communication
Leonardo S. Goodall, Andrea I. Luppi, Pedro A. M. Mediano · Jun 29, 2026 · Citations: 0
Medicine
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova, Ivo Petrov, Dimitar I. Dimitrov · Jun 29, 2026 · Citations: 0
Automatic Metrics Math
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources.
- Improving Answer Extraction in Context-based Question Answering Systems Using LLMs
Hafez Abdelghaffar, Ahmed Alansary, Ali Hamdi · Jun 4, 2026 · Citations: 0
Automatic Metrics General
Our methodology involves fine-tuning a pre-trained LLM on a benchmark QA dataset to improve its contextual comprehension and answer extraction capabilities.
- Severity-Aware Curriculum Learning with Multi-Model Response Selection for Medical Text Generation
Ahmed Alansary, Molham Mohamed, Ali Hamdi · Jun 3, 2026 · Citations: 0
Automatic Metrics Medicine
Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.
- MASF: A Multi-Model Adaptive Selection Framework for Abstractive Text summarization
Ahmed Alansary, Ali Hamdi · Jun 3, 2026 · Citations: 0
Automatic Metrics General
The generated summaries are then evaluated using automatic evaluation metrics that capture both lexical similarity and semantic relevance.
- Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents
Yuxin Wang, Paul Thomas, Zhiwei Yu, Yuan Gao, Saeed Hassanpour · Jun 24, 2026 · Citations: 0
Automatic Metrics General
Specifically, how they shape an agent's responses under varying conversational contexts and whether they lead to substantively different response behaviors.
- PETRA: Transforming Web Text for Petroleum-Engineering Domain Adaptation
Kirill Dubovikov, Omar El Mansouri, Hachem Madmoun, Yanda Li, Sandeep Kumar · Jun 23, 2026 · Citations: 0
Automatic Metrics General
Reranker adaptation improves the public Earth Science benchmark by 44% relative and a six-task reasoning-intensive panel by 23%.
- AVOC: Enhancing Hour-Level Audio-Video Understanding in Omni-Modal LLMs via Retrieval-Inspired Token Compression
Yijing Chen, Wenhui Tan, Xiaoyi Yu, Yuyue Wang, Xin Cheng · Jun 23, 2026 · Citations: 0
Automatic Metrics General
Experiments show that AVOC achieves state-of-the-art performance on long-form audio-video benchmarks, surpassing the second-best model by 4.9 and 5.5 points in average accuracy on OmniVideoBench and LVOmniBench, respectively.
- FALCON: Transforming Cyber Threat Intelligence into Deployable IDS Rules with Self-Reflection
Shaswata Mitra, Subash Neupane, Martin Duclos, Sudip Mittal, Aritran Piplai · Aug 26, 2025 · Citations: 0
Automatic Metrics General
To address these challenges, we introduce FALCON, an agentic framework for CTI-grounded rule retrieval, generation, and validation.
- Telenor Nordics Customer Service self-help corpus
Mike Riess · May 26, 2026 · Citations: 0
Automatic Metrics Multilingual
The documents have been sourced from the public self-help pages of four Nordic telecommunications operators and subsequently filtered for person-identifiable information and relevance through a combined LLM and human annotation pipeline.