BFCL or MATH-500 Benchmark Papers

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: High .

• 54 papers explicitly name benchmark datasets in the sampled set.
• 36 papers report at least one metric term in metadata extraction.
• Start with the ranked shortlist below before reading all papers.

Primary action: Start with the top 2 benchmark-matched papers, then compare evaluation modes in the protocol matrix.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

• Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning

  May 30, 2026 · Citations: 0 · Score: 7.5

  Eval: Simulation Env · Metrics: Precision

• Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning

  Jun 24, 2026 · Citations: 0 · Score: 7.5

  Eval: Automatic Metrics · Metrics: Accuracy

• What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

  Oct 17, 2026 · Citations: 0 · Score: 7.0

  Eval: Automatic Metrics · Metrics: Pass@1

• Base Models Can Reason By Taking a Cue From Training Data

  Oct 5, 2026 · Citations: 0 · Score: 7.0

  Eval: Automatic Metrics · Metrics: Accuracy

• Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Models

  Apr 22, 2026 · Citations: 0 · Score: 7.0

  Eval: Automatic Metrics · Metrics: Jailbreak success rate

• Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning

  Apr 1, 2026 · Citations: 0 · Score: 7.0

  Eval: Automatic Metrics · Metrics: Error rate

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Top Papers On This Benchmark

• Toollery: Scaling LLM Agents to Thousands of Skills and Tools

  Xiangxi Tian, Ran Guan · Sep 2, 2026 · Citations: 0

  Automatic Metrics

  As LLM agents are exposed to hundreds to tens of thousands of skills, tools, and API functions, full-library prompting becomes costly, slow, and less reliable: each added candidate increases prompt tokens and latency, while longer candidate…

• What RL Brings to Reasoning: A Few Decisions the Base Model Would Not Make

  Siyu Zhang, Yuquan Lu, Yoshua Y. Li, Yifan Ding, Kun Zeng · Oct 17, 2026 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Evaluating the Robustness of Proof Autoformalization in Lean 4

  Zhengtao Gui, Sheng Yang, Zhouxing Shi · Jun 12, 2026 · Citations: 0

  We build a benchmark with both perturbations on miniF2F and MATH-500, and automatically measure how stable a proof autoformalization's correctness is under global perturbations and how faithfully its output reflects local perturbations.

• BRIDGE: Bridging Reasoning In Distillation Gap Elimination via Structure-Aware Masking

  Bowen Yu, Sheng Zhang, Binhao Wang, Yi Wen, Jingtong Gao · Feb 5, 2026 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• ELF-REG: Scaling Continuous Diffusion Language Models to Reasoning Tasks

  Zeyu Michael Li, William Xingxu Chen, Bingshuo Qian, Jiayin Liu, Xiang Cheng · Sep 24, 2026 · Citations: 0

  Automatic Metrics

  ELF-REG-L achieves 55.96% pass@1 on GSM8K at 64 network function evaluations (NFE), and 13.39% on MATH-500 and 22.56% on HumanEval at 128 NFE.

• Base Models Can Reason By Taking a Cue From Training Data

  Sophie L. Wang, Amil Dravid, Rulin Shao, Kevin Farhat, Sewon Min · Oct 5, 2026 · Citations: 0

  Automatic Metrics

  Finally, we extend our study of token cues with a case study in language model safety, finding that different cues elicit distinct refusal and compliance behaviors that correspond to different types of training data.

• HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents

  Woongyeong Yeo, Yumin Choi, Taekyung Ki, Sung Ju Hwang · May 18, 2026 · Citations: 0

  Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected.

• Reasoning with Continuous Latent Diffusion

  Xiang Cheng · Sep 28, 2026 · Citations: 0

  Automatic Metrics

  Our supervised models outperform reported results from recent continuous-diffusion baselines at comparable backbone scales on mathematical reasoning and HumanEval code generation.

• AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

  Rishabh Agrawal, Hejie Cui, Shasha Li, Shanchan Wu, Sercan Ö. Arık · Sep 29, 2026 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• LOCKS: Page-Local Compact Key Summaries for Efficient Long-Context Decoding

  Junsung Hwang · Jul 27, 2026 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning

  Chishui Chen, Jiaye Lin, Te Sun, Yi Yang, Junxi Wang · May 30, 2026 · Citations: 0

  Pairwise Preference Simulation Env

  Agent skills are callable procedural modules that provide reusable knowledge and execution policies for complex agentic tasks.

• SPADE: Self-Play in Adaptive Synthetic Executable Environments

  Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao · Aug 19, 2026 · Citations: 0

  For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales.

• Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Models

  Yannis Belkhiter, Giulio Zizzo, Sergio Maffeis, Seshu Tirupathi, John D. Kelleher · Apr 22, 2026 · Citations: 0

  Pairwise PreferenceRed Team Automatic Metrics

  The growth of agentic AI has drawn significant attention to function calling Large Language Models (LLMs), which are designed to extend the capabilities of AI-powered system by invoking external functions.

• Tool Verification for Test-Time Reinforcement Learning

  Ruotong Liao, Nikolai Röhrich, Xiaohan Wang, Yuhui Zhang, Yasaman Samadzadeh · Mar 2, 2026 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Moxia: A Trust-First Neuro-Symbolic Execution Architecture for Self-Explaining Mathematical Reasoning

  Alessio Bruno · May 30, 2026 · Citations: 0

  Automatic Metrics

  The rule-only path answers the 20,000-record lm-eval arithmetic benchmark at 100%, 1 ms per record.

• Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents

  Dylan Bouchard, Mohit Singh Chauhan · Aug 12, 2026 · Citations: 0

  Automatic Metrics

  For LLM agents, however, the unit of observation is an interactive trajectory, where the model can ask clarifying questions, call tools, update state, and make intermediate decisions whose errors propagate to the final outcome.

• Shorthand for Thought: Compressing LLM Reasoning via Entropy-Guided Supertokens

  Zhenyu Zhao, Sander Land, Daniel M. Bikel, Waseem Alshikh · Apr 29, 2026 · Citations: 0

  Automatic Metrics

  Across three model families and five mathematical reasoning benchmarks, our approach shortens reasoning traces by 8.1% on average; under a TOST equivalence analysis at a +/- 2pp margin, accuracy is equivalent or inconclusive on 13/15 model…

• Learning How to Use Tools, Not Just When: Pattern-Aware Tool-Integrated Reasoning

  Ningning Xu, Yuxuan Jiang, Shubhashis Roy Dipta, Hengyuan Zhang · Sep 27, 2025 · Citations: 0

  Pairwise Preference Automatic Metrics

  We propose a two-stage framework that first builds code competence from both patterns and then aligns pattern selection with teacher preferences.

• JetSpec: Breaking the Scaling Ceiling of Speculative Decoding with Parallel Tree Drafting

  Lanxiang Hu, Zhaoxiang Feng, Yulun Wu, Haoran Yuan, Yujie Zhao · Jun 16, 2026 · Citations: 0

  Automatic Metrics

  Across math, coding, and chat benchmarks on dense and MoE Qwen3 models, JetSpec consistently outperforms bidirectional-head and tree-based SD baselines.

• Cliff Tokens: Identifying Single-Token Failure Triggers in LLM Mathematical Reasoning

  Jaeyong Ko, Pilsung Kang, Yukyung Lee · Jun 24, 2026 · Citations: 0

  Pairwise Preference Automatic Metrics

  Across seven models and three mathematical reasoning benchmarks (GSM1K, MATH500, AIME 2025), cliff tokens act as failure triggers; deleting the first cliff token and resampling recovers pass@64 to 1.0, while keeping it limits recovery to…

• ConPress: Learning Efficient Reasoning from Multi-Question Contextual Pressure

  Jie Deng, Shining Liang, Jun Li, Hongzhi Li, Yutao Xie · Feb 1, 2026 · Citations: 0

  Automatic Metrics

  This phenomenon arises from multi-question contextual pressure during generation and consistently manifests across models and benchmarks.

• Blockwise Policy-Drift Gating for On-Policy Distillation

  Liwen Zheng, Haiyun Jiang · Jun 23, 2026 · Citations: 0

  Automatic Metrics

  In a six-variant Qwen3 math reasoning benchmark with a uniform 200-step training budget for all trained variants, we use pass@8 as the primary problem-level solve-rate metric.

• SIGMA: Search-Augmented On-Demand Knowledge Integration for Agentic Mathematical Reasoning

  Ali Asgarov, Umid Suleymanov, Aadyant Khatri · Oct 31, 2025 · Citations: 0

  Automatic Metrics

  We introduce SIGMA (Search-Augmented On-Demand Knowledge Integration for AGentic Mathematical reAsoning), a unified framework that orchestrates specialized agents to independently reason, perform targeted searches, and synthesize findings…

• Reasoning with Sampling: Cutting at Decision Points

  Felix Zhou, Anay Mehrotra, Quanquan C. Liu · May 28, 2026 · Citations: 0

  Across MATH500, HumanEval, GPQA Diamond, and AIME26, our method consistently improves over baselines and RL-trained models.

• Modeling Hierarchical Thinking in Large Reasoning Models

  G M Shahariar, Erfan Shayegani, Ali Nazari, Nael Abu-Ghazaleh · Oct 25, 2025 · Citations: 0

  Experiments across four benchmarks (AIME25, MATH-500, GSM8k, and GPQA Diamond) using three state-of-the-art open reasoning models demonstrate that Q-Value steering policy achieves significant performance gains with "surgical" efficiency,…

• Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs

  Andrea Sassella, Andrea Chizzola, Tommaso Bianchi, Luca Alessandrelli, Mark James Carman · May 8, 2026 · Citations: 0

  This report benchmarks the performance of ENGINEERING Ingegneria Informatica S.p.A.'s EngGPT2MoE-16B-A3B LLM, a 16B parameter Mixture of Experts (MoE) model with 3B active parameters.

• MUR: Momentum Uncertainty guided Reasoning for Large Language Models

  Hang Yan, Fangzhi Xu, Rongman Xu, Yifei Li, Jian Zhang · Jul 20, 2025 · Citations: 0

  Automatic Metrics

  MUR is comprehensively evaluated against various TTS methods across four challenging benchmarks (MATH-500, AIME24, AIME25, and GPQA-diamond) using different sizes of recent Qwen3 models (1.7B, 4B, and 8B).

• Tool Calling is Linearly Readable and Steerable in Language Models

  Zekun Wu, Ze Wang, Seonglae Cho, Yufei Yang, Adriano Koshiyama · May 8, 2026 · Citations: 0

  Automatic Metrics

  When a tool-calling agent picks the wrong tool, the failure is invisible until execution: the email gets sent, the meeting gets missed.

• The Bitter Lesson of Diffusion Language Models for Agentic Workflows: A Comprehensive Reality Check

  Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, Dacheng Tao · Jan 19, 2026 · Citations: 0

  Automatic Metrics

  In this work, we present a comprehensive evaluation of dLLMs (e.g., LLaDA, Dream) across two distinct agentic paradigms: Embodied Agents (requiring long-horizon planning) and Tool-Calling Agents (requiring precise formatting).

• Toward Automated Robustness Evaluation of Mathematical Reasoning

  Yutao Hou, Zeguan Xiao, Fei Yu, Yihan Jiang, Ma Shuguang · Jun 5, 2025 · Citations: 0

  Existing robustness evaluations predominantly rely on hand-crafted templates or a limited set of perturbation rules.

• SkillX: Automatically Constructing Skill Knowledge Bases for Agents

  Chenxi Wang, Zhuoyun Yu, Xin Xie, Wuguannan Yao, Runnan Fang · Apr 6, 2026 · Citations: 0

  Automatic Metrics

  Learning from experience is critical for building capable large language model (LLM) agents, yet prevailing self-evolving paradigms remain inefficient: agents learn in isolation, repeatedly rediscover similar behaviors from limited…

• DeepPrune: Parallel Scaling without Inter-trace Redundancy

  Shangqing Tu, Yaxuan Li, Yushi Bai, Lei Hou, Juanzi Li · Oct 9, 2025 · Citations: 0

  Llm As JudgeAutomatic Metrics

  Our method features a specialized judge model trained with out-of-distribution data (AIME 2022, AIME 2023, and MATH-500) using oversampling techniques to accurately predict answer equivalence from partial reasoning traces, achieving 0.7072…

• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions

  Junhao Su, Yuanliang Wan, Junwei Yang, Hengyu Shi, Tianyang Han · Sep 23, 2025 · Citations: 0

  Automatic Metrics

  The agent produces a short yet precise reflection: it diagnoses the failure using evidence from the previous step and then proposes a correct, executable follow-up call.

• Towards Hierarchical Multi-Step Reward Models for Enhanced Reasoning in Large Language Models

  Teng Wang, Zhangyi Jiang, Zhenqi He, Shenyang Tong, Wenhan Yang · Mar 16, 2025 · Citations: 0

  Automatic Metrics

  Empirical results on the PRM800K dataset show that HRM, together with HNC, provides more stable and reliable evaluations than PRM.

• Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents

  Xuan Qi · Apr 2, 2026 · Citations: 0

  Automatic Metrics

  Chain-of-thought (CoT) reasoning is widely assumed to improve agent performance, but the relationship between reasoning length and accuracy in structured tool-use settings remains poorly understood.

• Reliable Control-Point Selection for Steering Reasoning in Large Language Models

  Haomin Zhuang, Hojun Yoo, Xiaonan Luo, Kehan Guo, Xiangliang Zhang · Apr 2, 2026 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Online Reasoning Calibration: Test-Time Training Enables Generalizable Conformal LLM Reasoning

  Cai Zhou, Zekai Wang, Menghua Wu, Qianyu Julie Zhu, Flora C. Shi · Apr 1, 2026 · Citations: 0

  Automatic Metrics

  Under zero-shot out-of-domain settings, it improves MATH-500 savings from 24.8% of the static calibration baseline to 67.0% while maintaining a low empirical error rate, and the same trend holds across model families and downstream…

• S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models

  Jack Young · Apr 1, 2026 · Citations: 0

  Automatic Metrics

  Using roughly 48 execution-verified HumanEval training solutions, tuning a single initial state matrix per recurrent layer, with zero inference overhead, outperforms LoRA by +10.8 pp (p < 0.001) on HumanEval.

• Locally Confident, Globally Stuck: The Quality-Exploration Dilemma in Diffusion Language Models

  Liancheng Fang, Aiwei Liu, Henry Peng Zou, Yankai Chen, Enze Ma · Apr 1, 2026 · Citations: 0

  Experiments across a range of reasoning benchmarks including MATH500, AIME24/25, HumanEval, and MBPP show that our approach yields better exploration-quality tradeoff than both random and low-confidence remasking.

• TAPS: Task Aware Proposal Distributions for Speculative Sampling

  Mohamad Zbib, Mohamad Bazzi, Ammar Mohanna, Hasan Abed Al Kader Hammoud, Bernard Ghanem · Mar 27, 2026 · Citations: 0

  Measured by acceptance length, task-specific training yields clear specialization: MathInstruct-trained drafts are strongest on reasoning benchmarks, while ShareGPT-trained drafts are strongest on MT-Bench.

• When to Ensemble: Identifying Token-Level Points for Stable and Fast LLM Ensembling

  Heecheol Yun, Kwangmin Ki, Junghyun Lee, Eunho Yang · Oct 17, 2025 · Citations: 0

  Automatic Metrics

  Our experiments on diverse benchmarks, including MATH500 and BBH, demonstrate that SAFE outperforms existing methods in both accuracy and efficiency, with gains achieved even when ensembling fewer than 1% of tokens.

• Try, Check and Retry: A Divide-and-Conquer Framework for Boosting Long-context Tool-Calling Performance of LLMs

  Kunfeng Chen, Qihuang Zhong, Juhua Liu, Bo Du, Dacheng Tao · Mar 12, 2026 · Citations: 0

  Tool-DC (TF) brings up to +25.10% average gains against the baseline on BFCL and ACEBench benchmarks, while Tool-DC (TB) enables Qwen2.5-7B to achieve comparable or even better performance than proprietary LLMs, e.g., OpenAI o3 and…

• KV Cache Transform Coding for Compact Storage in LLM Inference

  Konrad Staniszewski, Adrian Łańcucki · Nov 3, 2025 · Citations: 0

  Automatic Metrics

  We test KVTC with Llama 3, Mistral NeMo, and R1-Qwen 2.5 models across benchmarks including AIME25, GSM8K, LiveCodeBench, LongBench, MATH-500, MMLU, Qasper and RULER.

• DistillGuard: Evaluating Defenses Against LLM Knowledge Distillation

  Bo Jiang · Mar 8, 2026 · Citations: 0

  We introduce a taxonomy of three defense categories -- output perturbation, data poisoning, and information throttling -- and evaluate nine defense configurations using a standardized pipeline with Qwen3-14B as teacher and…

• MathSmith: Towards Extremely Hard Mathematical Reasoning by Forging Synthetic Problems with a Reinforced Policy

  Shaoxiong Zhan, Yanlin Lai, Ziyu Lu, Dahua Lin, Ziqing Yang · Aug 7, 2025 · Citations: 0

  Existing synthesis methods largely rely on transforming human-written templates, limiting both diversity and scalability.

• CyclicReflex: Improving Reasoning Models via Cyclical Reflection Token Scheduling

  Chongyu Fan, Yihua Zhang, Jinghan Jia, Alfred Hero, Sijia Liu · Jun 4, 2025 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Not All Errors Are Created Equal: ASCoT Addresses Late-Stage Fragility in Efficient LLM Reasoning

  Dongxu Zhang, Yujun Wu, Yiding Sun, Jinnan Yang, Ning Yang · Aug 7, 2025 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Cache What Lasts: Token Retention for Memory-Bounded KV Cache in LLMs

  Ngoc Bui, Shubham Sharma, Simran Lamba, Saumitra Mishra, Rex Ying · Dec 3, 2025 · Citations: 0

  Automatic Metrics

  Across mathematical reasoning (GSM8K, MATH-500, AIME24), procedural generation (LongProc), conversational long-memory benchmarks (LongMemEval), and long-context understanding (LongBenchV2 and SCBench), TRIM-KV consistently outperforms…

• InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models

  Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang · Mar 9, 2025 · Citations: 0

  Experiments across multiple model architectures demonstrate that our approach reduces computational costs while improving performance, with Qwen2.5-Math-7B showing 3-11% improvements across MATH500, AIME24, and GPQA_diamond benchmarks.

• Spurious Rewards: Rethinking Training Signals in RLVR

  Rulin Shao, Shuyue Stella Li, Rui Xin, Scott Geng, Yiping Wang · Jun 12, 2025 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Overconfident Errors Need Stronger Correction: Asymmetric Confidence Penalties for Reinforcement Learning

  Yuanda Xu, Hejian Sang, Zhengze Zhou, Ran He, Zhipeng Wang · Feb 24, 2026 · Citations: 0

  Automatic Metrics

  Evaluated on MATH-500 and AIME 2025, ACE composes seamlessly with existing methods and consistently improves the full Pass@k spectrum across all three model families and benchmarks.

• What If We Allocate Test-Time Compute Adaptively?

  Ahsan Bilal, Ahmed Mohsin, Muhammad Umer, Ali Subhan, Hassan Rizwan · Feb 1, 2026 · Citations: 0

  For each problem, the agent runs multiple inference iterations.

• $\texttt{SPECS}$: Faster Test-Time Scaling through Speculative Drafts

  Mert Cemri, Nived Rajaraman, Rishabh Tiwari, Xiaoxuan Liu, Kurt Keutzer · Jun 15, 2025 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Parameter-Efficient Fine-Tuning of LLMs with Mixture of Space Experts

  Buze Zhang, Jinkai Tao, Zilang Zeng, Neil He, Ali Maatouk · Feb 16, 2026 · Citations: 0

  Our experiments across diverse benchmarks demonstrate that MoSLoRA consistently outperforms strong baselines, achieving up to 5.6% improvement on MATH500 and 15.9% on MAWPS.

Related Benchmark Hubs