Kaiyue Yang, Yuyan Bu, Jingwei Yi, Yuchi Wang, Biyu Zhou, Juntao Dai · Jun 18, 2026 · Citations: 0
Pairwise PreferenceGeneral
As LLM agents increasingly select tools autonomously, their choices among tools with different privileges become safety-relevant.
We introduce ToolPrivBench to evaluate whether agents choose higher-privilege tools despite sufficient lower-privilege alternatives, measuring both initial selection and escalation after transient tool failures.
To Eun Kim, Xuhong He, Dishank Jain, Ambuj Agrawal, Negar Arabzadeh, Fernando Diaz · Jun 18, 2026 · Citations: 0
Simulation EnvGeneral
The decentralized deployment of LLM agents with diverse capabilities across diverse tasks motivates infrastructure for knowledge sharing across heterogeneous agent populations.
We propose Multi-Agent Transactive Memory (MATM), a framework for population-level storage and retrieval of agent-generated trajectories, where producer agents contribute trajectories to a shared repository and consumer agents retrieve them…
Dang Nguyen, Wanqing Hao, Yanai Elazar, Chenhao Tan · Jun 18, 2026 · Citations: 0
Pairwise PreferenceAutomatic MetricsGeneral
A new class of agentic review systems are emerging as a remedy to the pressure placed on peer review systems by AI-assisted research, but it is unclear how they should be evaluated.
Together, by evaluating full review systems backed by state-of-the-art models on real research papers, we show that while AI reviews still have room for improvement, they can already track human quality judgments well, catch important…
Pranav Bhandari, Nicolas Fay, Amitava Datta, Usman Naseem, Mehwish Nasim · Jun 18, 2026 · Citations: 0
Pairwise PreferenceGeneral
Aligning language models with human preferences often requires optimising multiple behavioural objectives.
A practical approach is to apply these objectives sequentially using preference optimisation methods such as Direct Preference Optimisation (DPO), but it remains unclear whether later training uniformly degrades preferences learned earlier…
Yuhang Zhou, Lizhu Zhang, Yifan Wu, Mingyi Wang, Bo Peng, Jiayi Liu · Jun 17, 2026 · Citations: 0
Simulation EnvGeneral
On-policy distillation (OPD) improves student models by training them on trajectories induced by their own policy, making it a promising approach for mitigating exposure bias in agent training.
Experiments on agent tasks show that SAGE-OPD consistently improves over baselines, achieving up to a 13.3% relative improvement in ALFWorld unseen success rate over standard OPD.
Recent position papers argue that the classical aleatoric/epistemic uncertainty framework is insufficient for interactive large language model (LLM) agents and call for underspecification-aware, decomposed, and communicable uncertainty…
To evaluate it, we introduce two clarification-augmented benchmarks (WebShop-Clarification and ALFWorld-Clarification) in which 50% of tasks are deliberately underspecified, and systematically compare the proposed decomposition against…
Justin D. Norman, Michael U. Rivera, D. Alex Hughes · Jun 17, 2026 · Citations: 0
Pairwise PreferenceRubric RatingLlm As JudgeAutomatic MetricsGeneral
We present the largest systematic evaluation of LLM-as-a-Judge to date: 21 judges from nine providers across MT-Bench, JudgeBench, and RewardBench, evaluated under three protocols (agreement, consistency, bias audit) over 118 runs and…
Four findings emerge, consistent across the full cohort, including the April 2026 frontier: kappa deflation between exact match and Cohen's kappa is universal (33--41 pp on MT-Bench), judge rankings shift by up to 14 positions across…
Jaward Sesay, Yue Yu, Siwei Dong, Guangyao Chen, Börje F. Karlsson · Jun 15, 2026 · Citations: 0
Rubric RatingSimulation EnvGeneral
To this end, we propose LectūraAgents - a multi-agent framework that enables personalized learning through end-to-end adaptive embodied teaching.
We evaluate LectūraAgents on diverse courses at high school, undergraduate, and graduate levels using sample-specific rubric-based analysis; with generated lecture materials and teaching actions assessed and validated by expert educators.
Uwe König, Hamza Kazmi, Ruizhe Li, Maheep Chaudhary · Jun 9, 2026 · Citations: 0
Red TeamGeneral
Evaluation on 100 JailbreakBench prompts with GPT-4.1, serving as the evaluator, indicates that transfer is robust but exhibits distinct scaling behaviors.
Hao Chen, Ziyu Han, Yukun Yan, Qingfu Zhu, Maosong Sun, Wanxiang Che · Jun 7, 2026 · Citations: 0
Rubric RatingGeneral
As Large Language Models (LLMs) advance toward open-ended autonomous agents, the mechanisms used to evaluate and guide their behavior must evolve accordingly.
This work introduces the rubric as a unifying framework capturing this evolution, characterizing rubrics as a dynamic response to successive LLM paradigm shifts that recurs across otherwise independent efforts in evaluation, reinforcement…
The results reveal effective and robust stance transitions in both text-only and multimodal strategies across different polarization-preference mechanisms.
Our study contributes an evaluation framework for understanding the context sensitivity of LLM-based stance simulation (https://github.com/STARResearchLab/StanceShift).
Magnus Jørgenvåg, David Kaczér, Lasse Ruttert, Marvin Gülhan, Lucie Flek, Florian Mai · May 29, 2026 · Citations: 0
Pairwise PreferenceGeneral
Second, we show that EM from RL can be induced by reward signals that could plausibly arise naturally, such as unpopular aesthetic preferences or poor rhetorical appeals.
Third, we evaluate in-training mitigations developed for SFT-induced EM and find that they broadly transfer, with preventive steering with persona vectors, interleaving safety data and inoculation prompting all performing well.
Across two public LLM leaderboards, many displayed pairwise rankings do not meet a conventional paired-test resolution target under the actual paired evaluation design: 11 of 40 Open LLM Leaderboard v1 pairwise comparisons and 4 of 9…
We frame paired LLM evaluation as a hypothesis-testing problem, invert level-alpha, power-(1-beta) tests, and report a per-pair resolution ratio q = N/N* as the primary diagnostic.
Qiuyue Wang, Mingsheng Li, Jian Guan, Jinhui Ye, Sicheng Xie, Yitao Liu · May 28, 2026 · Citations: 0
DemonstrationsSimulation EnvGeneral
Qwen-VLA is trained with a large-scale joint pretraining recipe over diverse data sources, including robotics manipulation trajectories, human egocentric demonstrations, synthetic simulation data, vision-and-language navigation data,…
Experiments on manipulation, navigation, and trajectory-centric benchmarks show consistent multi-task performance and out-of-distribution generalization under variations in scene layout, background, lighting, object configuration, and robot…
