Benchmark Hub
Updated from current HFEPX corpus (Feb 27, 2026). 12 papers are grouped in this benchmark page. Common evaluation modes: Automatic Metrics, Simulation Env. Frequent quality control: Calibration. Frequently cited benchmark: GSM8K. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 26, 2026.
Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 12 papers for GSM8K + Automatic Metrics Benchmark Papers. Dominant protocol signals include automatic metrics, simulation environments, with frequent benchmark focus on GSM8K, HumanEval+ and metric focus on accuracy, cost. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.
automatic metrics appears in 100% of papers in this hub.
Evidence: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference , SPQ: An Ensemble Technique for Large Language Model Compression
GSM8K is a recurring benchmark anchor for cross-paper comparisons in this page.
Evidence: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference , SPQ: An Ensemble Technique for Large Language Model Compression
Most common quality-control signal is rater calibration (16.7% of papers).
Evidence: Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference , InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , SPQ: An Ensemble Technique for Large Language Model Compression
Stratify by benchmark (GSM8K vs HumanEval+) before comparing methods.
Evidence: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference , SPQ: An Ensemble Technique for Large Language Model Compression
Track metric sensitivity by reporting both accuracy and cost.
Evidence: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference , SPQ: An Ensemble Technique for Large Language Model Compression
Papers with explicit human feedback
Coverage is a replication risk (0% vs 45% target).
Papers reporting quality controls
Coverage is a replication risk (16.7% vs 30% target).
Papers naming benchmarks/datasets
Coverage is strong (100% vs 35% target).
Papers naming evaluation metrics
Coverage is strong (75% vs 35% target).
Papers with known rater population
Coverage is a replication risk (0% vs 35% target).
Papers with known annotation unit
Coverage is a replication risk (0% vs 35% target).
1. InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models
Start here for detailed protocol reporting, including rater and quality-control evidence.
Start here for detailed protocol reporting, including rater and quality-control evidence.
3. Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Start here for detailed protocol reporting, including rater and quality-control evidence.
4. SPQ: An Ensemble Technique for Large Language Model Compression
Adds automatic metrics for broader coverage within this hub.
5. TFL: Targeted Bit-Flip Attack on Large Language Model
Adds automatic metrics for broader coverage within this hub.
6. Weight space Detection of Backdoors in LoRA Adapters
Adds automatic metrics for broader coverage within this hub.
7. Scaling Beyond Masked Diffusion Language Models
Adds automatic metrics for broader coverage within this hub.
8. Search or Accelerate: Confidence-Switched Position Beam Search for Diffusion Language Models
Adds automatic metrics for broader coverage within this hub.
automatic_metrics vs simulation_env
both=1, left_only=11, right_only=0
1 papers use both Automatic Metrics and Simulation Env.
Benchmark Brief
GSM8K
Coverage: 12 papers (100%)
12 papers (100%) mention GSM8K.
Examples: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Benchmark Brief
HumanEval+
Coverage: 3 papers (25%)
3 papers (25%) mention HumanEval+.
Examples: Weight space Detection of Backdoors in LoRA Adapters , Search or Accelerate: Confidence-Switched Position Beam Search for Diffusion Language Models , Slm-mux: Orchestrating small language models for reasoning
Benchmark Brief
MMLU
Coverage: 2 papers (16.7%)
2 papers (16.7%) mention MMLU.
Examples: Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Diffusion Language Models Know the Answer Before Decoding
Metric Brief
accuracy
Coverage: 7 papers (58.3%)
7 papers (58.3%) mention accuracy.
Examples: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Metric Brief
cost
Coverage: 2 papers (16.7%)
2 papers (16.7%) mention cost.
Examples: Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference , Diffusion Language Models Know the Answer Before Decoding
Metric Brief
latency
Coverage: 2 papers (16.7%)
2 papers (16.7%) mention latency.
Examples: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Most Cited In This Hub
Fast path to methods with the strongest citation traction in this scope.
Papers: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Most Recent
Fast path to latest protocol changes and newly published evaluation setups.
Papers: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Best Protocol Detail
Papers with explicit rater/unit metadata and quality-control signals for reproducibility.
Papers: InnerQ: Hardware-aware Tuning-free Quantization of KV Cache for Large Language Models , Black-Box Reliability Certification for AI Agents via Self-Consistency Sampling and Conformal Calibration , Pyramid MoA: A Probabilistic Framework for Cost-Optimized Anytime Inference
Sayed Mohammadreza Tayaranian Hosseini, Amir Ardakani, Warren J. Gross · Feb 26, 2026
Our evaluation experiments on Llama models shows that InnerQ maintains a few-shot GSM8K performance comparable to non-quantized KV caches and surpasses prior KV cache quantization methods.
Charafeddine Mouzouni · Feb 24, 2026
We validate across five benchmarks, five models from three families, and both synthetic and real data.
Arindam Khaled · Feb 23, 2026
In this work, we propose "Pyramid MoA", a hierarchical Mixture-of-Agents architecture that uses a lightweight Router to dynamically escalate queries only when necessary.
Jiamin Yao, Eren Gultepe · Feb 20, 2026
Applied to LLaMA-2-7B, SPQ achieves up to 75% memory reduction while maintaining or improving perplexity (e.g., WikiText-2 5.47 to 4.91) and preserving accuracy on downstream benchmarks such as C4, TruthfulQA, and GSM8K.
Jingkai Guo, Chaitali Chakrabarti, Deliang Fan · Feb 19, 2026
Large language models (LLMs) are increasingly deployed in safety and security critical applications, raising concerns about their robustness to model parameter fault injection attacks.
David Puertolas Merenciano, Ekaterina Vasyagina, Raghav Dixit, Kevin Zhu, Ruizhe Li · Feb 16, 2026
We evaluate the method on 500 LoRA adapters -- 400 clean, and 100 poisoned for Llama-3.2-3B on instruction and reasoning datasets: Alpaca, Dolly, GSM8K, ARC-Challenge, SQuADv2, NaturalQuestions, HumanEval, and GLUE dataset.
Subham Sekhar Sahoo, Jean-Marie Lemercier, Zhihan Yang, Justin Deschenaux, Jingyu Liu · Feb 16, 2026
Among discrete diffusion approaches, Masked diffusion currently dominates, largely driven by strong perplexity on language modeling benchmarks.
Mingyu Cao, Alvaro H. C. Correia, Christos Louizos, Shiwei Liu, Lu Yin · Feb 11, 2026
Across mathematical reasoning and code generation benchmarks (GSM8K, MBPP, HumanEval) on Dream-7B and LLaDA-8B, SOAR improves generation quality while maintaining competitive inference speed, offering a practical way to balance quality and
Chenyu Wang, Zishen Wan, Hao Kang, Emma Chen, Zhiqiang Xie · Oct 6, 2025
Additional experiments show that the core principle of SLM-MUX extends to open-ended generation tasks (e.g., HumanEval) and benefits other model classes, including frontier LLMs and domain-specific fine-tuned SLMs.
Pengxiang Li, Yefan Zhou, Dilxat Muhtar, Lu Yin, Shilin Yan · Aug 27, 2025
Empirical evaluations of LLaDA-8B and Dream-7B across multiple tasks show that Prophet reduces the number of decoding steps by up to 3.4x while preserving high generation quality.
Dongxu Zhang, Ning Yang, Yiding Sun, Jihua Zhu, Jinnan Yang · Aug 7, 2025
While Chain-of-Thought (CoT) prompting empowers Large Language Models (LLMs), ensuring reasoning reliability remains an open challenge.
Yuchen Yan, Yongliang Shen, Yang Liu, Jin Jiang, Xin Xu · Feb 17, 2025
Through comprehensive experiments on multiple mathematical reasoning datasets, including MathInstruct, MetaMathQA and etc., we demonstrate that models trained on MathFimer-expanded data consistently outperform their counterparts trained on
