HFEPX Benchmark Hub

Arena/Judge Suite Benchmark Papers

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this benchmark page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 10 papers are grouped in this benchmark page. Common evaluation modes: Automatic Metrics. Common annotation unit: Pairwise. Frequent quality control: Calibration. Frequently cited benchmark: LMSYS Chatbot Arena. Common metric signal: error rate. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 13, 2026.

Papers: 10 Last published: Feb 13, 2026 Global RSS

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: Developing .

High-Signal Coverage

100.0%

10 / 10 sampled papers are not low-signal flagged.

Replication-Ready Set

Papers with explicit benchmark + metric + eval mode fields.

Quality Controls

10.0%

1 papers report calibration/adjudication/IAA controls.

6 papers explicitly name benchmark datasets in the sampled set.
1 papers report at least one metric term in metadata extraction.
Start with the ranked shortlist below before reading all papers.

Primary action: Use this page to map benchmark mentions first; wait for stronger metric/QC coverage before strict comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

100% of papers report explicit human-feedback signals, led by pairwise preferences.
automatic metrics appears in 10% of papers in this hub.
LMSYS Chatbot Arena is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Notes (Expanded)

Protocol Takeaways

Most common quality-control signal is rater calibration (10% of papers).
Rater context is mostly unspecified rater pools, and annotation is commonly pairwise annotation; use this to scope replication staffing.
Stratify by benchmark (LMSYS Chatbot Arena vs AlpacaEval) before comparing methods.

Benchmark Interpretation

LMSYS Chatbot Arena appears in 83.3% of hub papers (5/10); use this cohort for benchmark-matched comparisons.
AlpacaEval appears in 33.3% of hub papers (2/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

error rate is reported in 16.7% of hub papers (1/10); compare with a secondary metric before ranking methods.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Feb 13, 2026 · Citations: 0 · Score: 10.0

Eval: Automatic Metrics · Metrics: Error rate
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty
Oct 7, 2025 · Citations: 0 · Score: 5.5

Eval: Not reported · Metrics: Not Reported
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
Sep 27, 2025 · Citations: 0 · Score: 5.5

Eval: Not reported · Metrics: Not Reported
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
Jul 11, 2025 · Citations: 0 · Score: 5.0

Eval: Not reported · Metrics: Not Reported
Search Arena: Analyzing Search-Augmented LLMs
Jun 5, 2025 · Citations: 0 · Score: 5.0

Eval: Not reported · Metrics: Not Reported
Less is More: Improving LLM Alignment via Preference Data Selection
Feb 20, 2025 · Citations: 0 · Score: 4.4

Eval: Not reported · Metrics: Not Reported

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Paper	Eval Modes	Human Feedback	Metrics	Quality Controls
SCOPE: Selective Conformal Optimized Pairwise LLM Judging Feb 13, 2026	Automatic Metrics	Pairwise Preference	Error rate	Calibration
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty Oct 7, 2025	Not reported	Pairwise Preference	Not reported	Not reported
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization Sep 27, 2025	Not reported	Pairwise Preference	Not reported	Not reported
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench Jul 11, 2025	Not reported	Pairwise Preference	Not reported	Not reported
Search Arena: Analyzing Search-Augmented LLMs Jun 5, 2025	Not reported	Pairwise Preference	Not reported	Not reported
Less is More: Improving LLM Alignment via Preference Data Selection Feb 20, 2025	Not reported	Pairwise Preference	Not reported	Not reported
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation Feb 26, 2026	Not reported	Not reported	Not reported	Not reported
Elo-Evolve: A Co-evolutionary Framework for Language Model Alignment Feb 14, 2026	Not reported	Not reported	Not reported	Not reported
Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding Sep 26, 2025	Not reported	Not reported	Not reported	Not reported
LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning Jun 23, 2025	Not reported	Not reported	Not reported	Not reported

Researcher Workflow (Detailed)

Checklist

Strong: Papers with explicit human feedback

Coverage is strong (100% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (16.7% vs 30% target).
Strong: Papers naming benchmarks/datasets

Coverage is strong (100% vs 35% target).
Gap: Papers naming evaluation metrics

Coverage is a replication risk (16.7% vs 35% target).
Gap: Papers with known rater population

Coverage is a replication risk (0% vs 35% target).
Moderate: Papers with known annotation unit

Coverage is usable but incomplete (33.3% vs 35% target).

Strengths

Strong human-feedback signal (100% of papers).

Known Gaps

Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
Rater population is under-specified (0% coverage).
Metric coverage is thin (16.7% of papers mention reported metrics).

Suggested Next Analyses

Stratify by benchmark (LMSYS Chatbot Arena vs AlpacaEval) before comparing methods.
Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

Benchmark Slice: LMSYS Chatbot Arena Metric Slice: error rate Recent High-Signal Papers

Known Limitations

Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
Rater population is under-specified (0% coverage).
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot (Detailed)

Evaluation Modes

Automatic Metrics (1)

Human Feedback Mix

Pairwise Preference (6)

Top Benchmarks

LMSYS Chatbot Arena (5)
AlpacaEval (2)
AlpacaEval 2.0 (2)
Arena Hard (2)

Top Metrics

Error rate (1)

Top Papers On This Benchmark

SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Sher Badshah, Ali Emami, Hassan Sajjad · Feb 13, 2026 · Citations: 0

Pairwise Preference Automatic Metrics

Large language models (LLMs) are increasingly used as judges to replace costly human preference labels in pairwise evaluation.
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty
Yao Xiao, Jung-jae Kim, Roy Ka-wei Lee, Lidong Bing · Oct 7, 2025 · Citations: 0

Pairwise Preference

Self-play preference optimization has emerged as a prominent paradigm for aligning large language models (LLMs).
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
Junming Yang, Ning Xu, Biao Liu, Shiqi Qiao, Xin Geng · Sep 27, 2025 · Citations: 0

Pairwise Preference

To bridge this gap, we propose Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically couples data generation with model training.
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
David Schlangen, Sherzod Hakimov, Chalamalasetti Kranti, Jonathan Jordan, Philipp Sadler · Jul 11, 2025 · Citations: 0

Pairwise Preference

There are currently two main paradigms for evaluating large language models (LLMs), reference-based evaluation and preference-based evaluation.
Search Arena: Analyzing Search-Augmented LLMs
Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan · Jun 5, 2025 · Citations: 0

Pairwise Preference

In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs.
Less is More: Improving LLM Alignment via Preference Data Selection
Xun Deng, Han Zhong, Rui Ai, Fuli Feng, Zheng Wang · Feb 20, 2025 · Citations: 0

Pairwise Preference

Direct Preference Optimization (DPO) has emerged as a promising approach for aligning large language models with human preferences.
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Jiahao Zhao, Feng Jiang, Shaowei Qin, Zhonghui Zhang, Junhao Liu · Feb 26, 2026 · Citations: 0
Elo-Evolve: A Co-evolutionary Framework for Language Model Alignment
Jing Zhao, Ting Zhen, Junwei Bao, Hongfei Jiang, Yang Song · Feb 14, 2026 · Citations: 0
Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding
Shijing Hu, Jingyang Li, Zhihui Lu, Pan Zhou · Sep 26, 2025 · Citations: 0
LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, Juanzi Li · Jun 23, 2025 · Citations: 0

Related Benchmark Hubs

LMSYS Chatbot Arena Benchmark Papers LMSYS Chatbot Arena In CS.CL Papers LMSYS Chatbot Arena + Pairwise Preference Benchmark Papers CS.CL Human Feedback And Eval Papers LMSYS Chatbot Arena In CS.AI Papers MT-Bench Benchmark Papers WebArena Ecosystem Benchmark Papers (15) Reasoning & Math Suite Benchmark Papers (16) SWE-Bench Ecosystem Benchmark Papers (10) WebArena Ecosystem Benchmark Papers In CS.CL (10) Reasoning & Math Suite Benchmark Papers In CS.CL (15) Reasoning & Math Suite Benchmark Papers In CS.AI (13)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote