BFCL or GPQA Benchmark Papers

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: High .

• 40 papers explicitly name benchmark datasets in the sampled set.
• 29 papers report at least one metric term in metadata extraction.
• Start with the ranked shortlist below before reading all papers.

Primary action: Start with the top 2 benchmark-matched papers, then compare evaluation modes in the protocol matrix.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

• When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs

  Aug 11, 2026 · Citations: 0 · Score: 9.5

  Eval: Automatic Metrics · Metrics: Accuracy

• Decomposing Wrong-Consensus Agreement in LLM Self-Consistency

  Aug 19, 2026 · Citations: 0 · Score: 8.0

  Eval: Automatic Metrics · Metrics: Accuracy

• Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning

  May 30, 2026 · Citations: 0 · Score: 7.5

  Eval: Simulation Env · Metrics: Precision

• CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

  Jun 23, 2026 · Citations: 0 · Score: 7.5

  Eval: Automatic Metrics · Metrics: Accuracy

• Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Models

  Apr 22, 2026 · Citations: 0 · Score: 7.0

  Eval: Automatic Metrics · Metrics: Jailbreak success rate

• Toollery: Scaling LLM Agents to Thousands of Skills and Tools

  Sep 2, 2026 · Citations: 0 · Score: 6.5

  Eval: Automatic Metrics · Metrics: Accuracy

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Top Papers On This Benchmark

• When Self-Consistency Backfires: Majority Vote Hurts the Majority of Hard Science Problems for Small LLMs

  Utkarsh Bahuguna · Aug 11, 2026 · Citations: 0

  Demonstrations Automatic Metrics

  On the full GPQA Diamond benchmark (198 graduate-level science questions), majority voting reduces per-problem accuracy on a majority of problems for two instruction-tuned models from different families: 56.6% of problems for Qwen2.5-7B and…

• Toollery: Scaling LLM Agents to Thousands of Skills and Tools

  Xiangxi Tian, Ran Guan · Sep 2, 2026 · Citations: 0

  Automatic Metrics

  As LLM agents are exposed to hundreds to tens of thousands of skills, tools, and API functions, full-library prompting becomes costly, slow, and less reliable: each added candidate increases prompt tokens and latency, while longer candidate…

• HINT-SD: Targeted Hindsight Self-Distillation for Long-Horizon Agents

  Woongyeong Yeo, Yumin Choi, Taekyung Ki, Sung Ju Hwang · May 18, 2026 · Citations: 0

  Training long-horizon LLM agents with reinforcement learning is challenging because sparse outcome rewards reveal whether a task succeeds, but not which intermediate actions caused the outcome or how they should be corrected.

• AdviSD: Learning to Advise Frontier LLMs via Targeted Multi-Turn Self-Distillation

  Rishabh Agrawal, Hejie Cui, Shasha Li, Shanchan Wu, Sercan Ö. Arık · Sep 29, 2026 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Time-Anchored Diffusion Language Models: Latent-Space Caching for Fast Generation

  Joel Anto Paul, Litu Rout, Aditya Akella, Sanjay Shakkottai · Sep 29, 2026 · Citations: 0

  Automatic Metrics

  Applied to DiffusionGemma-26B, TADM:Post-train improves throughput by approximately 49% to 79% on several math, code, and STEM benchmarks (GSM8K, AIME26, GPQA-Diamond, LiveCodeBench-v6, HumanEval, MMLU-Pro).

• Skill or Skip? Learning Selective Skill Invocation in Agentic Tasks via Dual-Granularity Preference Learning

  Chishui Chen, Jiaye Lin, Te Sun, Yi Yang, Junxi Wang · May 30, 2026 · Citations: 0

  Pairwise Preference Simulation Env

  Agent skills are callable procedural modules that provide reusable knowledge and execution policies for complex agentic tasks.

• Decomposing Wrong-Consensus Agreement in LLM Self-Consistency

  Lizhuo Zhang, Mengmeng Tang, Chenfeng Long, Xiaoyong Tang, Xiang Luo · Aug 19, 2026 · Citations: 0

  Pairwise Preference Automatic Metrics

  The mechanical reference is leak-free: each case's preference and accuracy are estimated from its other runs only.

• SPADE: Self-Play in Adaptive Synthetic Executable Environments

  Bo Liu, Simon Yu, Yiding Jiang, Ao Qu, Andrew Zhao · Aug 19, 2026 · Citations: 0

  For language agents, existing training environment pools (hand-curated, statically synthesized, or frozen-verifier) keep the goal distribution fixed as the learner scales.

• Breaking MCP with Function Hijacking Attacks: Novel Threats for Function Calling and Agentic Models

  Yannis Belkhiter, Giulio Zizzo, Sergio Maffeis, Seshu Tirupathi, John D. Kelleher · Apr 22, 2026 · Citations: 0

  Pairwise PreferenceRed Team Automatic Metrics

  The growth of agentic AI has drawn significant attention to function calling Large Language Models (LLMs), which are designed to extend the capabilities of AI-powered system by invoking external functions.

• The Metanym Game: An LLM Benchmark Without Ground Truth That Rises With the Models It Measures

  David Nordfors · Jun 19, 2026 · Citations: 0

  In our benchmark, LLMs compete in generating sets of analogous statements and rate each other's sets on their own understandings of factual correctness, beauty, intelligence, distinctness, length, and structural diversity.

• Beyond Single-Turn Confidence: Trajectory-Adapted Uncertainty Quantification for LLM Agents

  Dylan Bouchard, Mohit Singh Chauhan · Aug 12, 2026 · Citations: 0

  Automatic Metrics

  For LLM agents, however, the unit of observation is an interactive trajectory, where the model can ask clarifying questions, call tools, update state, and make intermediate decisions whose errors propagate to the final outcome.

• Epistemic Goggles: A Pretrained Module that Induces an Epistemic Frame via Gradient Editing

  Joshua Penman · Jul 2, 2026 · Citations: 0

  In our evaluations, models trained on documents prefixed and suffixed with such annotations correctly identify the relevant claims as fictional only about 9% of the time.

• Theoria: Rewrite-Acceptability Verification over Informal Reasoning States

  Michael Saldivar, Ben Slivinski · Jul 1, 2026 · Citations: 0

  Formal proof assistants offer certainty but cannot reach most of the problem distribution; scalar LLM judges offer coverage but produce opaque scores that cannot be audited after the fact and are subject to the same coherence issues as any…

• IsoSci: A Benchmark of Isomorphic Cross-Domain Science Problems for Evaluating Reasoning versus Knowledge Retrieval in LLMs

  Samir Abdaljalil, Erchin Serpedin, Hasan Kurban · Jul 1, 2026 · Citations: 0

  Automatic Metrics

  We introduce ISOSCI, a benchmark of isomorphic cross-domain science problem pairs that separates reasoning ability from domain knowledge retrieval in LLM evaluation.

• CALIBER: Calibrating Confidence Before and After Reasoning in Language Models

  Conor Finlay, Joshua Kurien, Saurabh Dash, Marzieh Fadaee, Beyza Ermis · Jun 23, 2026 · Citations: 0

  Automatic Metrics

  Ablations further show that this position-target alignment is most beneficial under distribution shift where it consistently reduces calibration error across all out-of-distribution benchmarks.

• SIGMA: Search-Augmented On-Demand Knowledge Integration for Agentic Mathematical Reasoning

  Ali Asgarov, Umid Suleymanov, Aadyant Khatri · Oct 31, 2025 · Citations: 0

  Automatic Metrics

  We introduce SIGMA (Search-Augmented On-Demand Knowledge Integration for AGentic Mathematical reAsoning), a unified framework that orchestrates specialized agents to independently reason, perform targeted searches, and synthesize findings…

• Reasoning with Sampling: Cutting at Decision Points

  Felix Zhou, Anay Mehrotra, Quanquan C. Liu · May 28, 2026 · Citations: 0

  Across MATH500, HumanEval, GPQA Diamond, and AIME26, our method consistently improves over baselines and RL-trained models.

• Reasoning Theater: Disentangling Model Beliefs from Chain-of-Thought

  Siddharth Boppana, Annabel Ma, Max Loeffler, Raphael Sarfati, Eric Bigelow · Mar 5, 2026 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Modeling Hierarchical Thinking in Large Reasoning Models

  G M Shahariar, Erfan Shayegani, Ali Nazari, Nael Abu-Ghazaleh · Oct 25, 2025 · Citations: 0

  Experiments across four benchmarks (AIME25, MATH-500, GSM8k, and GPQA Diamond) using three state-of-the-art open reasoning models demonstrate that Q-Value steering policy achieves significant performance gains with "surgical" efficiency,…

• Accelerated Test-Time Scaling with Model-Free Speculative Sampling

  Woomin Song, Saket Dingliwal, Sai Muralidhar Jayanthi, Bhavana Ganesh, Jinwoo Shin · Jun 5, 2025 · Citations: 0

  Automatic Metrics

  Extensive evaluations across multiple models and reasoning tasks (AIME-2024, GPQA-Diamond, and LiveCodeBench) demonstrate that STAND reduces inference latency by 60-65% compared to standard autoregressive decoding while maintaining…

• Benchmarking EngGPT2-16B-A3B against Comparable Italian and International Open-source LLMs

  Andrea Sassella, Andrea Chizzola, Tommaso Bianchi, Luca Alessandrelli, Mark James Carman · May 8, 2026 · Citations: 0

  This report benchmarks the performance of ENGINEERING Ingegneria Informatica S.p.A.'s EngGPT2MoE-16B-A3B LLM, a 16B parameter Mixture of Experts (MoE) model with 3B active parameters.

• MUR: Momentum Uncertainty guided Reasoning for Large Language Models

  Hang Yan, Fangzhi Xu, Rongman Xu, Yifei Li, Jian Zhang · Jul 20, 2025 · Citations: 0

  Automatic Metrics

  MUR is comprehensively evaluated against various TTS methods across four challenging benchmarks (MATH-500, AIME24, AIME25, and GPQA-diamond) using different sizes of recent Qwen3 models (1.7B, 4B, and 8B).

• Tool Calling is Linearly Readable and Steerable in Language Models

  Zekun Wu, Ze Wang, Seonglae Cho, Yufei Yang, Adriano Koshiyama · May 8, 2026 · Citations: 0

  Automatic Metrics

  When a tool-calling agent picks the wrong tool, the failure is invisible until execution: the email gets sent, the meeting gets missed.

• Diffusion-Inspired Masked Fine-Tuning for Knowledge Injection in Autoregressive LLMs

  Xu Pan, Ely Hahami, Jingxuan Fan, Ziqian Xie, Haim Sompolinsky · Oct 10, 2025 · Citations: 0

  Automatic Metrics

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• When to Think, When to Speak: Learning Disclosure Policies for LLM Reasoning

  Jiaqi Wei, Xuehang Guo, Pengfei Yu, Xiang Zhang, Wanli Ouyang · May 5, 2026 · Citations: 0

  Automatic Metrics

  Across two Qwen3 architectures/scales (MoE Qwen3-30B-A3B, dense Qwen3-4B) and both in-domain (AIME25) and out-of-domain (GPQA-Diamond) benchmarks, SxS improves accuracy--content-latency Pareto trade-offs under token-level proxies such as…

• The Bitter Lesson of Diffusion Language Models for Agentic Workflows: A Comprehensive Reality Check

  Qingyu Lu, Liang Ding, Kanjian Zhang, Jinxia Zhang, Dacheng Tao · Jan 19, 2026 · Citations: 0

  Automatic Metrics

  In this work, we present a comprehensive evaluation of dLLMs (e.g., LLaDA, Dream) across two distinct agentic paradigms: Embodied Agents (requiring long-horizon planning) and Tool-Calling Agents (requiring precise formatting).

• SkillX: Automatically Constructing Skill Knowledge Bases for Agents

  Chenxi Wang, Zhuoyun Yu, Xin Xie, Wuguannan Yao, Runnan Fang · Apr 6, 2026 · Citations: 0

  Automatic Metrics

  Learning from experience is critical for building capable large language model (LLM) agents, yet prevailing self-evolving paradigms remain inefficient: agents learn in isolation, repeatedly rediscover similar behaviors from limited…

• DeepPrune: Parallel Scaling without Inter-trace Redundancy

  Shangqing Tu, Yaxuan Li, Yushi Bai, Lei Hou, Juanzi Li · Oct 9, 2025 · Citations: 0

  Llm As JudgeAutomatic Metrics

  Our method features a specialized judge model trained with out-of-distribution data (AIME 2022, AIME 2023, and MATH 500) using oversampling techniques to accurately predict answer equivalence from partial reasoning traces, achieving 0.7072…

• Failure Makes the Agent Stronger: Enhancing Accuracy through Structured Reflection for Reliable Tool Interactions

  Junhao Su, Yuanliang Wan, Junwei Yang, Hengyu Shi, Tianyang Han · Sep 23, 2025 · Citations: 0

  Automatic Metrics

  The agent produces a short yet precise reflection: it diagnoses the failure using evidence from the previous step and then proposes a correct, executable follow-up call.

• Brief Is Better: Non-Monotonic Chain-of-Thought Budget Effects in Function-Calling Language Agents

  Xuan Qi · Apr 2, 2026 · Citations: 0

  Automatic Metrics

  Chain-of-thought (CoT) reasoning is widely assumed to improve agent performance, but the relationship between reasoning length and accuracy in structured tool-use settings remains poorly understood.

• Schema for In-Context Learning

  Pan Chen, Shaohong Chen, Mark Wang, Shi Xuan Leong, Priscilla Fung · Oct 14, 2025 · Citations: 0

  Demonstrations

  Inspired by cognitive science, specifically schema theory, which holds that humans interpret new information by activating pre-existing mental frameworks (schemas) to structure understanding, we introduce Schema-Activated In-Context…

• Why Models Know But Don't Say: Chain-of-Thought Faithfulness Divergence Between Thinking Tokens and Answers in Open-Weight Reasoning Models

  Richard J. Young · Mar 27, 2026 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes

  Deepon Halder, Raj Dabre · Mar 15, 2026 · Citations: 0

  Automatic Metrics

  Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating…

• Try, Check and Retry: A Divide-and-Conquer Framework for Boosting Long-context Tool-Calling Performance of LLMs

  Kunfeng Chen, Qihuang Zhong, Juhua Liu, Bo Du, Dacheng Tao · Mar 12, 2026 · Citations: 0

  Tool-DC (TF) brings up to +25.10% average gains against the baseline on BFCL and ACEBench benchmarks, while Tool-DC (TB) enables Qwen2.5-7B to achieve comparable or even better performance than proprietary LLMs, e.g., OpenAI o3 and…

• CyclicReflex: Improving Reasoning Models via Cyclical Reflection Token Scheduling

  Chongyu Fan, Yihua Zhang, Jinghan Jia, Alfred Hero, Sijia Liu · Jun 4, 2025 · Citations: 0

  Abstract shows limited direct human-feedback or evaluation-protocol detail; use as adjacent methodological context.

• D-COT: Disciplined Chain-of-Thought Learning for Efficient Reasoning in Small Language Models

  Shunsuke Ubukata · Feb 25, 2026 · Citations: 0

  Automatic Metrics

  In this study, we propose Disciplined Chain-of-Thought (D-CoT), a novel framework that enforces a structured reasoning process using control tags -- such as <TEMP_LOW> for fact-checking and <TEMP_HIGH> for multi-perspective exploration --…

• InftyThink: Breaking the Length Limits of Long-Context Reasoning in Large Language Models

  Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Mengdi Zhang · Mar 9, 2025 · Citations: 0

  Experiments across multiple model architectures demonstrate that our approach reduces computational costs while improving performance, with Qwen2.5-Math-7B showing 3-11% improvements across MATH500, AIME24, and GPQA_diamond benchmarks.

• Slm-mux: Orchestrating small language models for reasoning

  Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie · Oct 6, 2025 · Citations: 0

  Automatic Metrics

  Additional experiments show that the core principle of SLM-MUX extends to open-ended generation tasks (e.g., HumanEval) and benefits other model classes, including frontier LLMs and domain-specific fine-tuned SLMs.

• HEART: Emotionally-Driven Test-Time Scaling of Language Models

  Gabriela Pinto, Palash Goyal, Mihir Parmar, Yiwen Song, Souradip Chakraborty · Sep 26, 2025 · Citations: 0

  Automatic Metrics

  We introduce HEART, a framework that uses emotional cues to guide the model's focus, much like how feelings contribute to human decision-making.

• Recursive Concept Evolution for Compositional Reasoning in Large Language Models

  Sarim Chaudhry · Feb 17, 2026 · Citations: 0

  Automatic Metrics

  Large language models achieve strong performance on many complex reasoning tasks, yet their accuracy degrades sharply on benchmarks that require compositional reasoning, including ARC-AGI-2, GPQA, MATH, BBH, and HLE.

Related Benchmark Hubs