Metric Hub
Updated from current HFEPX corpus (Feb 27, 2026). 12 papers are grouped in this metric page. Common evaluation modes: Automatic Metrics. Most common rater population: Domain Experts. Frequently cited benchmark: AIME. Common metric signal: success rate. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 26, 2026.
Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 12 papers for Success Rate + Automatic Metrics Metric Papers. Dominant protocol signals include automatic metrics, with frequent benchmark focus on AIME, Re-Bench and metric focus on success rate, jailbreak success rate. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.
41.7% of papers report explicit human-feedback signals, led by red-team protocols.
Evidence: MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs , What Matters For Safety Alignment? , Reasoning Up the Instruction Ladder for Controllable Language Models , When Style Breaks Safety: Defending LLMs Against Superficial Style Alignment
automatic metrics appears in 100% of papers in this hub.
Evidence: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , Uncovering Context Reliance in Unstructured Knowledge Editing
AIME is a recurring benchmark anchor for cross-paper comparisons in this page.
Evidence: Evolutionary System Prompt Learning for Reinforcement Learning in LLMs , Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
Evidence: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , Uncovering Context Reliance in Unstructured Knowledge Editing
Rater context is mostly domain experts, and annotation is commonly mixed annotation units; use this to scope replication staffing.
Evidence: Measuring AI Ability to Complete Long Software Tasks , Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Stratify by benchmark (AIME vs Re-Bench) before comparing methods.
Evidence: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , Uncovering Context Reliance in Unstructured Knowledge Editing
Direct snippets from paper abstracts to ground protocol and benchmark interpretation.
Human-eval abstract signal: Current approaches to developing persuasive dialogue agents often rely on a limited set of predefined persuasive strategies that fail to capture the complexity of real-world interactions.
Human-eval abstract signal: Multi-robot task planning requires decomposing natural-language instructions into executable actions for heterogeneous robot teams.
AIME benchmark signal: We validated our proposed framework through experiments on two distinct datasets: the Persuasion for Good dataset, which represents a specific in-domain scenario, and the DailyPersuasion dataset, which encompasses a wide range of scenarios.
success rate metric signal: When plans fail, the system applies TextGrad-inspired textual-gradient updates to optimize each agent's prompt and thereby improve planning accuracy.
Protocol abstract signal: Current stateless defences for multimodal agentic RAG fail to detect adversarial strategies that distribute malicious semantics across retrieval, planning, and generation components.
Protocol abstract signal: Editing Large language models (LLMs) with real-world, unstructured knowledge is essential for correcting and updating their internal parametric knowledge.
Protocol abstract signal: Defending LLMs against adversarial jailbreak attacks remains an open challenge.
Protocol abstract signal: To estimate safety risk at training time, we explore two distinct approaches: a judge-based Safety Critic that assigns high-level harm scores to training batches, and an activation-based risk predictor built with a lightweight classifier...
Papers with explicit human feedback
Coverage is usable but incomplete (41.7% vs 45% target).
Papers reporting quality controls
Coverage is a replication risk (0% vs 30% target).
Papers naming benchmarks/datasets
Coverage is usable but incomplete (25% vs 35% target).
Papers naming evaluation metrics
Coverage is strong (100% vs 35% target).
Papers with known rater population
Coverage is a replication risk (8.3% vs 35% target).
Papers with known annotation unit
Coverage is a replication risk (0% vs 35% target).
1. Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies
Start here for detailed protocol reporting, including rater and quality-control evidence.
2. Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning
Start here for detailed protocol reporting, including rater and quality-control evidence.
Start here for detailed protocol reporting, including rater and quality-control evidence.
4. Uncovering Context Reliance in Unstructured Knowledge Editing
Adds automatic metrics for broader coverage within this hub.
5. MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs
Adds automatic metrics with red-team protocols for broader coverage within this hub.
6. Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning
Adds automatic metrics for broader coverage within this hub.
7. Evolutionary System Prompt Learning for Reinforcement Learning in LLMs
Adds automatic metrics for broader coverage within this hub.
8. What Matters For Safety Alignment?
Adds automatic metrics with red-team protocols for broader coverage within this hub.
Benchmark Brief
AIME
Coverage: 1 papers (8.3%)
1 papers (8.3%) mention AIME.
Examples: Evolutionary System Prompt Learning for Reinforcement Learning in LLMs
Benchmark Brief
Re-Bench
Coverage: 1 papers (8.3%)
1 papers (8.3%) mention Re-Bench.
Examples: Measuring AI Ability to Complete Long Software Tasks
Benchmark Brief
Retrieval
Coverage: 1 papers (8.3%)
1 papers (8.3%) mention Retrieval.
Metric Brief
success rate
Coverage: 12 papers (100%)
12 papers (100%) mention success rate.
Examples: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Metric Brief
jailbreak success rate
Coverage: 6 papers (50%)
6 papers (50%) mention jailbreak success rate.
Examples: Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , MANATEE: Inference-Time Lightweight Diffusion Based Safety Defense for LLMs , Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning
Metric Brief
cost
Coverage: 2 papers (16.7%)
2 papers (16.7%) mention cost.
Examples: Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG , Learning to Stay Safe: Adaptive Regularization Against Safety Degradation during Fine-Tuning
Most Cited In This Hub
Fast path to methods with the strongest citation traction in this scope.
Papers: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Most Recent
Fast path to latest protocol changes and newly published evaluation setups.
Papers: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Best Protocol Detail
Papers with explicit rater/unit metadata and quality-control signals for reproducibility.
Papers: Enhancing Persuasive Dialogue Agents by Synthesizing Cross-Disciplinary Communication Strategies , Hierarchical LLM-Based Multi-Agent Framework with Prompt Optimization for Multi-Robot Task Planning , Adversarial Intent is a Latent Variable: Stateful Trust Inference for Securing Multimodal Agentic RAG
Shinnosuke Nozue, Yuto Nakano, Yotaro Watanabe, Meguru Takasaki, Shoji Moriya · Feb 26, 2026
Current approaches to developing persuasive dialogue agents often rely on a limited set of predefined persuasive strategies that fail to capture the complexity of real-world interactions.
Tomoya Kawabe, Rin Takano · Feb 25, 2026
We present a hierarchical multi-agent LLM-based planner with prompt optimization: an upper layer decomposes tasks and assigns them to lower-layer agents, which generate PDDL problems solved by a classical planner.
Inderjeet Singh, Vikas Pahuja, Aishvariya Priya Rathina Sabapathy, Chiara Picardi, Amit Giloni · Feb 24, 2026
Current stateless defences for multimodal agentic RAG fail to detect adversarial strategies that distribute malicious semantics across retrieval, planning, and generation components.
Zisheng Zhou, Mengqi Zhang, Shiguang Wu, Xiaotian Ye, Chi Zhang · Feb 22, 2026
Evaluations show that COIN reduces Context Reliance by 45.2% and outperforms strong baselines by 23.6% in editing success rate, highlighting the vital role of mitigating Context Reliance for robust editing.
Chun Yan Ryan Kan, Tommy Tran, Vedant Yadav, Ava Cai, Kevin Zhu · Feb 21, 2026
Defending LLMs against adversarial jailbreak attacks remains an open challenge.
Jyotin Goel, Souvik Maji, Pratik Mazumder · Feb 19, 2026
Instruction-following language models are trained to be helpful and safe, yet their safety behavior can deteriorate under benign fine-tuning and worsen under adversarial updates.
Lunjun Zhang, Ryan Chen, Bradly C. Stadie · Feb 16, 2026
Building agentic systems that can autonomously self-improve from experience is a longstanding goal of AI.
Xing Li, Hui-Ling Zhen, Lihao Yin, Xianzhi Yu, Zhenhua Dong · Jan 7, 2026
This paper presents a comprehensive empirical study on the safety alignment capabilities.
Zishuo Zheng, Vidhisha Balachandran, Chan Young Park, Faeze Brahman, Sachin Kumar · Oct 30, 2025
Our finetuned models achieve consistent improvements on instruction following and instruction hierarchy benchmarks, achieving roughly a 20% improvement on the IHEval conflict setup.
Linxin Song, Yutong Dai, Viraj Prabhu, Jieyu Zhang, Taiwei Shi · Aug 5, 2025
Autonomous agents that operate computers via Graphical User Interfaces (GUIs) often struggle with efficiency and reliability on complex, long-horizon tasks.
Yuxin Xiao, Sana Tonekaboni, Walter Gerych, Vinith Suriyakumar, Marzyeh Ghassemi · Jun 9, 2025
In this work, we seek to understand whether style patterns compromise LLM safety, how superficial style alignment increases model vulnerability, and how best to mitigate these risks during alignment.
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia · Mar 18, 2025
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.
