- Personalized RewardBench: Evaluating Reward Models with Human Aligned Personalization
Qiyao Ma, Dechen Gao, Rui Cai, Boqi Zhao, Hanchu Zhou · Apr 8, 2026 · Citations: 0
Human EvalAutomatic Metrics General
Pluralistic alignment has emerged as a critical frontier in the development of Large Language Models (LLMs), with reward models (RMs) serving as a central mechanism for capturing diverse human values.
- PubMed Reasoner: Dynamic Reasoning-based Retrieval for Evidence-Grounded Biomedical Question Answering
Yiqing Zhang, Xiaozhong Liu, Fabricio Murai · Mar 28, 2026 · Citations: 0
Llm As JudgeAutomatic Metrics Medicine
In this context, we introduce PubMed Reasoner, a biomedical QA agent composed of three stages: self-critic query refinement evaluates MeSH terms for coverage, alignment, and redundancy to enhance PubMed queries based on partial (metadata)…
- mamabench and mamaretrieval: Benchmarks for Evaluating Medical Retrieval-Augmented Generation in Maternal, Neonatal, and Reproductive Health
Yi Ren · Jun 28, 2026 · Citations: 0
Automatic Metrics Medicine
Medical question-answering benchmarks rarely cover the maternal, neonatal, child, and reproductive-health questions a nurse-midwife asks, and, to our knowledge, no public chunk-level relevance benchmark exists for maternal-health guideline…
- Correct Is Not Enough: Training Reasoning Planners with Executor-Grounded Rewards
Tianyang Han, Hengyu Shi, Junjie Hu, Xu Yang, Zhiling Wang · May 5, 2026 · Citations: 0
Automatic Metrics MathLaw
Extensive experiments on code and math benchmarks show that this executor-grounded reasoning reward improves the two-stage planner-executor system over execution-only training, suggesting that reasoning supervision should evaluate not only…
- SABER-Math: Automated Benchmark for Information Retrieval Evaluation in Mathematics
Nikolay Georgiev, Maria Drencheva, Kseniia Ibragimova, Ivo Petrov, Dimitar I. Dimitrov · Jun 29, 2026 · Citations: 0
Automatic Metrics Math
As agentic AI systems tackle more complex mathematical tasks, they increasingly rely on information retrieval (IR) to search problem databases, theorem libraries, and educational resources.
- CRIMSON: A Clinically-Grounded LLM-Based Metric for Generative Radiology Report Evaluation
Mohammed Baharoon, Thibault Heintz, Siavash Raissi, Mahmoud Alabbad, Mona Alhammad · Mar 6, 2026 · Citations: 0
Automatic Metrics Medicine
We introduce CRIMSON, a clinically grounded evaluation framework for chest X-ray report generation that assesses reports based on diagnostic correctness, contextual relevance, and patient safety.
- TaoSR1: The Thinking Model for E-commerce Relevance Search
Chenhe Dong, Shaowei Yao, Pengkun Jiao, Jianhui Yang, Yiming Jin · Aug 17, 2025 · Citations: 0
Human Eval General
Our framework, TaoSR1, involves three stages: (1) Supervised Fine-Tuning (SFT) with CoT to instill reasoning; (2) Offline sampling with a pass@N strategy and Direct Preference Optimization (DPO) to improve generation quality; and (3)…
- MemReranker: Reasoning-Aware Reranking for Agent Memory Retrieval
Chunyu Li, Jingyi Kang, Ding Chen, Mengyuan Zhang, Jiajun Shen · May 7, 2026 · Citations: 0
Automatic Metrics General
In agent memory systems, the reranking model serves as the critical bridge connecting user queries with long-term memory.
- Optimizing RAG Rerankers with LLM Feedback via Reinforcement Learning
Yuhang Wu, Xiangqing Shen, Fanfan Wang, Cangqi Zhou, Zhen Wu · Apr 2, 2026 · Citations: 0
Automatic Metrics General
However, current reranking models are typically optimized on static human annotated relevance labels in isolation, decoupled from the downstream generation process.
- Preference learning in shades of gray: Interpretable and bias-aware reward modeling for human preferences
Simona-Vasilica Oprea, Adela Bâra · Apr 1, 2026 · Citations: 0
Automatic Metrics General
Using the Anthropic HHRLHF dataset, we evaluate ten diverse large language models LLMs under a standard pairwise preference setting, where baseline performance remains below 0.74 ROC AUC, highlighting the difficulty of the task.
- MemRerank: Preference Memory for Personalized Product Reranking
Zhiyuan Peng, Xuyang Wu, Huaixiao Tou, Yi Fang, Yu Gong · Mar 31, 2026 · Citations: 0
Automatic Metrics General
LLM-based shopping agents increasingly rely on long purchase histories and multi-turn interactions for personalization, yet naively appending raw history to prompts is often ineffective due to noise, length, and relevance mismatch.
- Decision-Level Ordinal Modeling for Multimodal Essay Scoring with Large Language Models
Han Zhang, Jiamin Su, Li liu · Mar 16, 2026 · Citations: 0
Automatic Metrics General
Experiments on the multimodal EssayJudge dataset show that DLOM improves over a generation-based SFT baseline across scoring traits, and DLOM-GF yields further gains when modality relevance is heterogeneous.
- PEEM: Prompt Engineering Evaluation Metrics for Interpretable Joint Evaluation of Prompts and Responses
Minki Hong, Eunsoo Lee, Sohyun Park, Jihie Kim · Mar 11, 2026 · Citations: 0
Automatic Metrics Medicine
We propose PEEM (Prompt Engineering Evaluation Metrics), a unified framework for joint and interpretable evaluation of both prompts and responses.
- Query-focused and Memory-aware Reranker for Long Context Processing
Yuqing Li, Jiangnan Li, Mo Yu, Guoxuan Ding, Zheng Lin · Feb 12, 2026 · Citations: 0
Automatic Metrics General
It further establishes a new state-of-the-art on the LoCoMo benchmark that assesses the capabilities of dialogue understanding and memory usage.
- Evaluating Vision-Language and Large Language Models for Automated Student Assessment in Indonesian Classrooms
Nurul Aisyah, Muhammad Dehan Al Kautsar, Arif Hidayat, Raqib Chowdhury, Fajri Koto · Jun 5, 2025 · Citations: 0
Automatic Metrics Math
Assessment tasks include grading and generating personalized Indonesian feedback guided by rubric-based evaluation.
- EvoScientist: Towards Multi-Agent Evolving AI Scientists for End-to-End Scientific Discovery
Yougang Lyu, Xi Zhang, Xinhao Yi, Yuyue Zhao, Shuyu Guo · Mar 9, 2026 · Citations: 0
Human Eval Coding
To address this, we introduce EvoScientist, an evolving multi-agent AI scientist framework that continuously improves research strategies through persistent memory and self-evolution.
- Memory Makes the Difference: Evaluating How Different Memory Roles Shape Conversational Agents
Yuxin Wang, Paul Thomas, Zhiwei Yu, Yuan Gao, Saeed Hassanpour · Jun 24, 2026 · Citations: 0
Automatic Metrics General
Specifically, how they shape an agent's responses under varying conversational contexts and whether they lead to substantively different response behaviors.
- OneSearch-V2: The Latent Reasoning Enhanced Self-distillation Generative Search Framework
Ben Chen, Siyuan Wang, Yufei Ma, Zihan Liang, Xuxin Zhang · Mar 25, 2026 · Citations: 0
Automatic Metrics General
However, its inadequate understanding of complex queries, inefficient exploitation of latent user intents, and overfitting to narrow historical preferences have limited its further performance improvement.
- LocalSUG: Geography-Aware LLM for Query Suggestion in Local-Life Services
Jinwen Chen, Shuai Gong, Shiwen Zhang, Zheng Zhang, Yachao Zhao · Mar 5, 2026 · Citations: 0
Automatic Metrics General
While LLMs offer strong semantic generalization, deploying them in local-life services introduces three key challenges: lack of geographic grounding, exposure bias in preference optimization, and online inference latency.
- PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark
Ziyang Zeng, Dun Zhang, Yu Yan, Xu Sun, Cuiqiaoshu Pan · Jan 13, 2026 · Citations: 0
Automatic Metrics Medicine
To address these limitations, we introduce PosIR (Position-Aware Information Retrieval), the first standardized benchmark designed to systematically diagnose position bias in diverse retrieval scenarios.
- CRANE: Causal Relevance Analysis of Language-Specific Neurons in Multilingual Large Language Models
Yifan Le, Yunliang Li · Jan 8, 2026 · Citations: 0
Automatic Metrics Multilingual
Prior work has identified language-related neurons mainly through activation-based heuristics, which conflate language preference with functional importance.
- Multi-Agent Environments for Vehicle Routing Problems
Ricardo Gama, Ricardo Cunha, Daniel Fuertes, Carlos R. del-Blanco, Hugo L. Fernandes · Nov 21, 2024 · Citations: 0
Simulation Env Coding
Here, we propose MAEnvs4VRP library, a unified framework for multi-agent vehicle routing environments that supports classical, dynamic, stochastic, and multi-task problem variants within a single modular design.
- When Metrics Disagree: Automatic Similarity vs. LLM-as-a-Judge for Clinical Dialogue Evaluation
Bian Sun, Zhenjian Wang, Orvill de la Torre, Zirui Wang · Feb 27, 2026 · Citations: 0
Llm As JudgeAutomatic Metrics Medicine
Due to the resource-intensive nature of large-scale human validation, the model's performance was evaluated through a dual-track framework: Track A utilized traditional lexical similarity metrics (e.g., BLEU, ROUGE), while Track B employed…
- Q$^2$: Quantization-Aware Gradient Balancing and Attention Alignment for Low-Bit Quantization
Zhaoyang Wang, Dong Wang · Nov 8, 2025 · Citations: 0
Automatic Metrics Medicine
Quantization-aware training (QAT) has achieved remarkable success in low-bit ($\leq$4-bit) quantization for classification networks.
- A Multi-Agent Framework for Medical AI: Leveraging Fine-Tuned GPT, LLaMA, and DeepSeek R1 for Evidence-Based and Bias-Aware Clinical Query Processing
Naeimeh Nourmohammadi, Md Meem Hossain, The Anh Han, Safina Showkat Ara, Zia Ush Shamszaman · Feb 15, 2026 · Citations: 0
Automatic Metrics Medicine
We propose a multi-agent medical QA framework that combines complementary LLMs with evidence retrieval, uncertainty estimation, and bias checks to improve answer reliability.
- Causal Intervention-Based Memory Selection for Long-Horizon LLM Agents
Saksham Sahai Srivastava · May 17, 2026 · Citations: 0
Automatic Metrics Coding
Long-horizon LLM agents rely on persistent memory to support interactions across sessions, yet existing memory systems often retrieve context using semantic similarity or broad history inclusion, treating retrieved memories as uniformly…
- Novel Memory Forgetting Techniques for Autonomous AI Agents: Balancing Relevance and Efficiency
Payal Fofadiya, Sunil Tiwari · Apr 2, 2026 · Citations: 0
Automatic Metrics General
Long-horizon conversational agents require persistent memory for coherent reasoning, yet uncontrolled accumulation causes temporal decay and false memory propagation.
- Chow-Liu Ordering for Long-Context Reasoning in Chain-of-Agents
Naman Gupta, Vaibhav Singh, Arun Iyer, Kirankumar Shiragur, Pratham Grover · Mar 10, 2026 · Citations: 0
Automatic Metrics General
Sequential multi-agent reasoning frameworks such as Chain-of-Agents (CoA) handle long-context queries by decomposing inputs into chunks and processing them sequentially using LLM-based worker agents that read from and update a bounded…
- LieCraft: A Multi-Agent Framework for Evaluating Deceptive Capabilities in Language Models
Matthew Lyle Olson, Neale Ratzlaff, Musashi Hinck, Tri Nguyen, Vasudev Lal · Mar 6, 2026 · Citations: 0
Automatic Metrics General
Large Language Models (LLMs) exhibit impressive general-purpose capabilities but also introduce serious safety risks, particularly the potential for deception as models acquire increased agency and human oversight diminishes.
- From Medical Records to Diagnostic Dialogues: A Clinical-Grounded Approach and Dataset for Psychiatric Comorbidity
Tianxi Wan, Jiaming Luo, Siyuan Chen, Kunyao Lan, Jianhua Chen · Oct 29, 2025 · Citations: 0
Automatic Metrics Medicine
To address this, we develop a novel approach integrating synthetic patient electronic medical record (EMR) construction and multi-agent diagnostic dialogue generation.
- CodeRefine: A Pipeline for Enhancing LLM-Generated Code Implementations of Research Papers
Ekaterina Trofimova, Emil Sataev, Abhijit Singh Jowhari · Aug 23, 2024 · Citations: 0
Automatic Metrics Coding
Evaluations on diverse scientific papers demonstrate CodeRefine's ability to improve code implementation from the paper, potentially accelerating the adoption of cutting-edge algorithms in real-world applications.
- DysLexLens: A Low-Resource LLM Framework for Analysing Dyslexic Learners Insights from Online Forums
Dana Rezazadegan, Atie Kia, Phongpadid Nandavong, Dominique Carlon, Jeremy Nguyen · Jun 26, 2026 · Citations: 0
- Information-Aware KV Cache Compression for Long Reasoning
Jushi Kai, Zhuiri Xiao, Alexandra Birch, Zhouhan Lin · Jun 25, 2026 · Citations: 0
- ConvMemory v3: A Validity Context Layer for Conversational Memory via Target-Conditioned Relation Verification
Taiheng Pan · Jun 25, 2026 · Citations: 0
- Compact Geometric Representations of Hierarchies
Prashant Gokhale, Piotr Indyk, Yuhao Liu, Sandeep Silwal, Tony Chang Wang · Jun 16, 2026 · Citations: 0
- SproutRAG: Attention-Guided Tree Search with Progressive Embeddings for Long-Document RAG
Amirhossein Abaskohi, Issam H. Laradji, Peter West, Giuseppe Carenini · Jun 16, 2026 · Citations: 0
- RCEM: Robust Conversational Search EMbedder in Distributional Shift
Kilho Son, Paul Hsu, Cha Zhang, Dinei Florencio · Jun 1, 2026 · Citations: 0
- Learning Whom to Trust: Market-Feedback Adaptive Retrieval for Frozen LLMs in Event-Driven Financial RAG
Zijie Zhao, Roy E. Welsch · May 29, 2026 · Citations: 0
- EviLink: Multi-Path Schema Linking with Uncertainty-Guided Evidence Acquisition for Large-Scale Text-to-SQL
Huawei Zheng, Sen Yang, Zhaorui Yang, Yuhui Zhang, Haozhe Feng · May 28, 2026 · Citations: 0
- Source-Grounded Semantic Reinforcement Learning for Low-Resource Target-Language Generation
Zeli Su, Ziyin Zhang, Zewei Pan, Zhou Liu, Dingcheng Huang · May 28, 2026 · Citations: 0
- Relevance as a Vulnerability: How Web Retrieval Degrades Safety Alignment in LLM Agents
Aditya Nawal, Manit Baser, Mohan Gurusamy · May 28, 2026 · Citations: 0
- Efficient Benchmarking Is Just Feature Selection and Multiple Regression
Sam Bowyer, Acyr Locatelli, Kris Cao · May 25, 2026 · Citations: 0
- FlyRoute: Self-Evolving Agent Profiling via Data Flywheel for Adaptive Task Routing
Rongjun Li, Ziyu Zhou, Yihang Wu · May 21, 2026 · Citations: 0
- GradeLegal: Automated Grading for German Legal Cases
Abdullah Al Zubaer, Lorenz Wendlinger, Simon Alexander Nonn, Michael Granitzer, Jelena Mitrovic · May 20, 2026 · Citations: 0
- BELIEF: Structured Evidence Modeling and Uncertainty-Aware Fusion for Biomedical Question Answering
Chang Zong, Hao Ning, Siliang Tang, Jie Huang, Jian Wan · May 17, 2026 · Citations: 0
- Learning Faster with Better Tokens: Parameter-Efficient Vocabulary Adaptation for Specialized Text Summarization
Gunjan Balde, Soumyadeep Roy, Mainack Mondal, Niloy Ganguly · May 17, 2026 · Citations: 0
- Qwen Goes Brrr: Off-the-Shelf RAG for Ukrainian Multi-Domain Document Understanding
Anton Bazdyrev, Ivan Bashtovyi, Ivan Havlytskyi, Oleksandr Kharytonov, Artur Khodakovskyi · May 11, 2026 · Citations: 0
- Cited but Not Verified: Parsing and Evaluating Source Attribution in LLM Deep Research Agents
Hailey Onweller, Elias Lumer, Austin Huber, Pia Ramchandani, Vamse Kumar Subbiah · May 7, 2026 · Citations: 0
- VideoRouter: Query-Adaptive Dual Routing for Efficient Long-Video Understanding
Kuanwei Lin, Wenhao Zhang, Ge Li · May 7, 2026 · Citations: 0
- The Cost of Context: Mitigating Textual Bias in Multimodal Retrieval-Augmented Generation
Hoin Jung, Xiaoqian Wang · May 7, 2026 · Citations: 0
- From Unstructured to Structured: LLM-Guided Attribute Graphs for Entity Search and Ranking
Yilun Zhu, Nikhita Vedula, Shervin Malmasi · Apr 30, 2026 · Citations: 0
- Learning When to Remember: Risk-Sensitive Contextual Bandits for Abstention-Aware Memory Retrieval in LLM-Based Coding Agents
Mehmet Iscan · Apr 30, 2026 · Citations: 0
- Hypencoder Revisited: Reproducibility and Analysis of Non-Linear Scoring for First-Stage Retrieval
Arne Eichholtz, Yongkang Li, Jutte Vijverberg, Tobias Groot, Mohammad Aliannejadi · Apr 29, 2026 · Citations: 0
- From Prompt Risk to Response Risk: Paired Analysis of Safety Behavior of Large Language Model
Mengya Hu, Qiong Wei, Sandeep Atluri · Apr 28, 2026 · Citations: 0
- Learning to Route Queries to Heads for Attention-based Re-ranking with Large Language Models
Yuxing Tian, Fengran Mo, Zhiqi Huang, Weixu Zhang, Jian-Yun Nie · Apr 27, 2026 · Citations: 0
- MEG-RAG: Quantifying Multi-modal Evidence Grounding for Evidence Selection in RAG
Xihang Wang, Zihan Wang, Chengkai Huang, Quan Z. Sheng, Lina Yao · Apr 27, 2026 · Citations: 0
- Learning Evidence of Depression Symptoms via Prompt Induction
Eliseo Bao, Anxo Perez, David Otero, Javier Parapar · Apr 27, 2026 · Citations: 0
- JudgeSense: A Benchmark for Prompt Sensitivity in LLM-as-a-Judge Systems
Rohith Reddy Bellibatlu · Apr 26, 2026 · Citations: 0
- Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA
Teng Chen, Sheng Xu, Feixiang Guo, Xiaoyu Wang, Qingqing Gu · Apr 25, 2026 · Citations: 0
- From Similarity to Structure: Training-free LLM Context Compression with Hybrid Graph Priors
Yitian Zhou, Chaoning Zhang, Jiaquan Zhang, Zhenzhen Huang, Jinyu Guo · Apr 25, 2026 · Citations: 0