Metric Hub
Updated from current HFEPX corpus (Feb 27, 2026). 15 papers are grouped in this metric page. Common evaluation modes: Automatic Metrics, Simulation Env. Most common rater population: Domain Experts. Frequently cited benchmark: APPS. Common metric signal: success rate. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 26, 2026.
Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 15 papers for Success Rate + General Metric Papers. Dominant protocol signals include automatic metrics, simulation environments, LLM-as-judge, with frequent benchmark focus on APPS, Re-Bench and metric focus on success rate, jailbreak success rate. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.
40% of papers report explicit human-feedback signals, led by red-team protocols.
Evidence: MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs , What Matters For Safety Alignment? , Reasoning Up the Instruction Ladder for Controllable Language Models , When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment
automatic metrics appears in 73.3% of papers in this hub.
Evidence: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , Uncovering Context Reliance in Unstructured Knowledge Editing
APPS is a recurring benchmark anchor for cross-paper comparisons in this page.
Evidence: The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution , Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
Evidence: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Rater context is mostly domain experts, and annotation is commonly mixed annotation units; use this to scope replication staffing.
Evidence: EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis , Measuring AI Ability to Complete Long Software Tasks , Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning
Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
Evidence: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Papers with explicit human feedback
Coverage is usable but incomplete (40% vs 45% target).
Papers reporting quality controls
Coverage is a replication risk (0% vs 30% target).
Papers naming benchmarks/datasets
Coverage is a replication risk (20% vs 35% target).
Papers naming evaluation metrics
Coverage is strong (100% vs 35% target).
Papers with known rater population
Coverage is a replication risk (13.3% vs 35% target).
Papers with known annotation unit
Coverage is a replication risk (0% vs 35% target).
1. Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies
Start here for detailed protocol reporting, including rater and quality-control evidence.
2. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning
Start here for detailed protocol reporting, including rater and quality-control evidence.
3. LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Start here for detailed protocol reporting, including rater and quality-control evidence.
4. EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis
Include an LLM-as-judge paper to assess judge design and agreement assumptions.
Adds automatic metrics for broader coverage within this hub.
6. AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMs
Adds simulation environments for broader coverage within this hub.
7. Uncovering Context Reliance in Unstructured Knowledge Editing
Adds automatic metrics for broader coverage within this hub.
8. MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs
Adds automatic metrics with red-team protocols for broader coverage within this hub.
llm_as_judge vs automatic_metrics
both=0, left_only=1, right_only=11
0 papers use both Llm As Judge and Automatic Metrics.
automatic_metrics vs simulation_env
both=0, left_only=11, right_only=4
0 papers use both Automatic Metrics and Simulation Env.
simulation_env vs llm_as_judge
both=1, left_only=3, right_only=0
1 papers use both Simulation Env and Llm As Judge.
Benchmark Brief
APPS
Coverage: 1 papers (6.7%)
1 papers (6.7%) mention APPS.
Benchmark Brief
Re-Bench
Coverage: 1 papers (6.7%)
1 papers (6.7%) mention Re-Bench.
Examples: Measuring AI Ability to Complete Long Software Tasks
Benchmark Brief
Retrieval
Coverage: 1 papers (6.7%)
1 papers (6.7%) mention Retrieval.
Metric Brief
success rate
Coverage: 15 papers (100%)
15 papers (100%) mention success rate.
Examples: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Metric Brief
jailbreak success rate
Coverage: 7 papers (46.7%)
7 papers (46.7%) mention jailbreak success rate.
Examples: Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , AdapTools: Adaptive Tool-based Indirect Prompt Injection Attacks on Agentic LLMs , MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs
Metric Brief
cost
Coverage: 3 papers (20%)
3 papers (20%) mention cost.
Examples: Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning , EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis
Most Cited In This Hub
Fast path to methods with the strongest citation traction in this scope.
Papers: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Most Recent
Fast path to latest protocol changes and newly published evaluation setups.
Papers: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Best Protocol Detail
Papers with explicit rater/unit metadata and quality-control signals for reproducibility.
Papers: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies
Shinnosuke Nozue, Yuto Nakano, Yotaro Watanabe, Meguru Takasaki, Shoji Moriya · Feb 26, 2026
Current approaches to developing persuasive dialogue agents often rely on a limited set of predefined persuasive strategies that fail to capture the complexity of real-world interactions.
Tomoya Kawabe, Rin Takano · Feb 25, 2026
We present a hierarchical multi-agent LLM-based planner with prompt optimization: an upper layer decomposes tasks and assigns them to lower-layer agents, which generate PDDL problems solved by a classical planner.
Yue Yang, Shuo Cheng, Yu Fang, Homanga Bharadhwaj, Mingyu Ding · Feb 25, 2026
We introduce a 21-task simulation benchmark consisting of two challenging suites: LIBERO-Long++ and Ultra-Long.
Inderjeet Singh, Vikas Pahuja, Aishvariya Priya Rathina Sabapathy, Chiara Picardi, Amit Giloni · Feb 24, 2026
Current stateless defences for multimodal agentic RAG fail to detect adversarial strategies that distribute malicious semantics across retrieval, planning, and generation components.
Che Wang, Jiaming Zhang, Ziqi Zhang, Zijie Wang, Yinghui Wang · Feb 24, 2026
The integration of external data services (e.g., Model Context Protocol, MCP) has made large language model-based agents increasingly powerful for complex task execution.
Zisheng Zhou, Mengqi Zhang, Shiguang Wu, Xiaotian Ye, Chi Zhang · Feb 22, 2026
Evaluations show that COIN reduces Context Reliance by 45.2% and outperforms strong baselines by 23.6% in editing success rate, highlighting the vital role of mitigating Context Reliance for robust editing.
Chun Yan Ryan Kan, Tommy Tran, Vedant Yadav, Ava Cai, Kevin Zhu · Feb 21, 2026
Defending LLMs against adversarial jailbreak attacks remains an open challenge.
Jyotin Goel, Souvik Maji, Pratik Mazumder · Feb 19, 2026
Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates.
Xing Li, Hui-Ling Zhen, Lihao Yin, Xianzhi Yu, Zhenhua Dong · Jan 7, 2026
This paper presents a comprehensive empirical study on the safety alignment capabilities.
Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman, Sachin Kumar · Oct 30, 2025
Our finetuned models achieve consistent improvements on instruction following and instruction hierarchy benchmarks, achieving roughly a 20% improvement on the IHEval conflict setup.
Junlong Li, Wenshuo Zhao, Jian Zhao, Weihao Zeng, Haoze Wu · Oct 29, 2025
Real-world language agents must handle complex, multi-step workflows across diverse Apps.
Mohammad Hossein Samaei, Faryad Darabi Sahneh, Lee W. Cohnstaedt, Caterina Scoglio · Sep 24, 2025
We introduce EpidemIQs, a novel multi-agent LLM framework that integrates user inputs and autonomously conducts literature review, analytical derivation, network modeling, mechanistic modeling, stochastic simulations, data visualization and
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi · Aug 5, 2025
Autonomous agents that operate computers via Graphical User Interfaces (GUIs) often struggle with efficiency and reliability on complex, long-horizon tasks.
Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, Marzyeh Ghassemi · Jun 9, 2025
In this work, we seek to understand whether style patterns compromise LLM safety, how superficial style alignment increases model vulnerability, and how best to mitigate these risks during alignment.
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia · Mar 18, 2025
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.
