HFEPX Benchmark Hub

LMSYS Chatbot Arena Benchmark Papers

Updated from current HFEPX corpus (Mar 8, 2026). 7 papers are grouped in this benchmark page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 7 papers are grouped in this benchmark page. Common evaluation modes: Automatic Metrics. Common annotation unit: Pairwise. Frequent quality control: Calibration. Frequently cited benchmark: LMSYS Chatbot Arena. Common metric signal: coherence. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 13, 2026.

Papers: 7 Last published: Feb 13, 2026 Global RSS

Researcher Quick Triage

Use this page for benchmark-matched method comparisons and eval protocol selection. Quality band: Developing .

High-Signal Coverage

100.0%

7 / 7 sampled papers are not low-signal flagged.

Replication-Ready Set

Papers with explicit benchmark + metric + eval mode fields.

Quality Controls

14.3%

1 papers report calibration/adjudication/IAA controls.

5 papers explicitly name benchmark datasets in the sampled set.
1 papers report at least one metric term in metadata extraction.
Start with the ranked shortlist below before reading all papers.

Primary action: Use this page to map benchmark mentions first; wait for stronger metric/QC coverage before strict comparisons.

Why This Matters (Expanded)

Why This Matters For Eval Research

71.4% of papers report explicit human-feedback signals, led by pairwise preferences.
automatic metrics appears in 14.3% of papers in this hub.
LMSYS Chatbot Arena is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Notes (Expanded)

Protocol Takeaways

Most common quality-control signal is rater calibration (14.3% of papers).
Rater context is mostly unspecified rater pools, and annotation is commonly pairwise annotation; use this to scope replication staffing.
Stratify by benchmark (LMSYS Chatbot Arena vs AlpacaEval) before comparing methods.

Benchmark Interpretation

LMSYS Chatbot Arena appears in 100% of hub papers (7/7); use this cohort for benchmark-matched comparisons.
AlpacaEval appears in 28.6% of hub papers (2/7); use this cohort for benchmark-matched comparisons.

Metric Interpretation

coherence is reported in 14.3% of hub papers (1/7); compare with a secondary metric before ranking methods.
error rate is reported in 14.3% of hub papers (1/7); compare with a secondary metric before ranking methods.

Start Here (Benchmark-Matched First 6)

Ranked by protocol completeness so you can quickly find papers suitable for comparison studies.

SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Feb 13, 2026 · Citations: 0 · Score: 10.0

Eval: Automatic Metrics · Metrics: Error rate
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty
Oct 7, 2025 · Citations: 0 · Score: 5.5

Eval: Not reported · Metrics: Not Reported
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
Sep 27, 2025 · Citations: 0 · Score: 5.5

Eval: Not reported · Metrics: Not Reported
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
Jul 11, 2025 · Citations: 0 · Score: 5.0

Eval: Not reported · Metrics: Not Reported
Search Arena: Analyzing Search-Augmented LLMs
Jun 5, 2025 · Citations: 0 · Score: 5.0

Eval: Not reported · Metrics: Not Reported
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Feb 26, 2026 · Citations: 0 · Score: 0.0

Eval: Not reported · Metrics: Not Reported

Protocol Matrix (Top 10)

Compare protocol ingredients quickly before deep-reading full papers.

Paper	Eval Modes	Human Feedback	Metrics	Quality Controls
SCOPE: Selective Conformal Optimized Pairwise LLM Judging Feb 13, 2026	Automatic Metrics	Pairwise Preference	Error rate	Calibration
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty Oct 7, 2025	Not reported	Pairwise Preference	Not reported	Not reported
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization Sep 27, 2025	Not reported	Pairwise Preference	Not reported	Not reported
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench Jul 11, 2025	Not reported	Pairwise Preference	Not reported	Not reported
Search Arena: Analyzing Search-Augmented LLMs Jun 5, 2025	Not reported	Pairwise Preference	Not reported	Not reported
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation Feb 26, 2026	Not reported	Not reported	Not reported	Not reported
LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning Jun 23, 2025	Not reported	Not reported	Not reported	Not reported

Researcher Workflow (Detailed)

Checklist

Strong: Papers with explicit human feedback

Coverage is strong (71.4% vs 45% target).
Gap: Papers reporting quality controls

Coverage is a replication risk (14.3% vs 30% target).
Strong: Papers naming benchmarks/datasets

Coverage is strong (100% vs 35% target).
Moderate: Papers naming evaluation metrics

Coverage is usable but incomplete (28.6% vs 35% target).
Gap: Papers with known rater population

Coverage is a replication risk (0% vs 35% target).
Moderate: Papers with known annotation unit

Coverage is usable but incomplete (28.6% vs 35% target).

Strengths

Strong human-feedback signal (71.4% of papers).

Known Gaps

Only 14.3% of papers report quality controls; prioritize calibration/adjudication evidence.
Rater population is under-specified (0% coverage).

Suggested Next Analyses

Stratify by benchmark (LMSYS Chatbot Arena vs AlpacaEval) before comparing methods.
Track metric sensitivity by reporting both coherence and error rate.
Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

Benchmark Slice: LMSYS Chatbot Arena Metric Slice: coherence Recent High-Signal Papers

Known Limitations

Only 14.3% of papers report quality controls; prioritize calibration/adjudication evidence.
Rater population is under-specified (0% coverage).
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot (Detailed)

Evaluation Modes

Automatic Metrics (1)

Human Feedback Mix

Pairwise Preference (5)

Top Benchmarks

LMSYS Chatbot Arena (7)
AlpacaEval (2)
Arena Hard (2)
MT Bench (2)

Top Metrics

Coherence (1)
Error rate (1)

Top Papers On This Benchmark

SCOPE: Selective Conformal Optimized Pairwise LLM Judging
Sher Badshah, Ali Emami, Hassan Sajjad · Feb 13, 2026 · Citations: 0

Pairwise Preference Automatic Metrics

Large language models (LLMs) are increasingly used as judges to replace costly human preference labels in pairwise evaluation.
Revisiting Self-Play Preference Optimization: On the Role of Prompt Difficulty
Yao Xiao, Jung-jae Kim, Roy Ka-wei Lee, Lidong Bing · Oct 7, 2025 · Citations: 0

Pairwise Preference

Self-play preference optimization has emerged as a prominent paradigm for aligning large language models (LLMs).
Alignment through Meta-Weighted Online Sampling: Bridging the Gap between Data Generation and Preference Optimization
Junming Yang, Ning Xu, Biao Liu, Shiqi Qiao, Xin Geng · Sep 27, 2025 · Citations: 0

Pairwise Preference

To bridge this gap, we propose Meta-Weighted Adaptive Preference Optimization (MetaAPO), a novel framework that dynamically couples data generation with model training.
A Third Paradigm for LLM Evaluation: Dialogue Game-Based Evaluation using clembench
David Schlangen, Sherzod Hakimov, Chalamalasetti Kranti, Jonathan Jordan, Philipp Sadler · Jul 11, 2025 · Citations: 0

Pairwise Preference

There are currently two main paradigms for evaluating large language models (LLMs), reference-based evaluation and preference-based evaluation.
Search Arena: Analyzing Search-Augmented LLMs
Mihran Miroyan, Tsung-Han Wu, Logan King, Tianle Li, Jiayi Pan · Jun 5, 2025 · Citations: 0

Pairwise Preference

In this work, we introduce Search Arena, a crowd-sourced, large-scale, human-preference dataset of over 24,000 paired multi-turn user interactions with search-augmented LLMs.
SC-Arena: A Natural Language Benchmark for Single-Cell Reasoning with Knowledge-Augmented Evaluation
Jiahao Zhao, Feng Jiang, Shaowei Qin, Zhonghui Zhang, Junhao Liu · Feb 26, 2026 · Citations: 0
LongWriter-Zero: Mastering Ultra-Long Text Generation via Reinforcement Learning
Yuhao Wu, Yushi Bai, Zhiqiang Hu, Roy Ka-Wei Lee, Juanzi Li · Jun 23, 2025 · Citations: 0

Related Benchmark Hubs

Arena/Judge Suite Benchmark Papers LMSYS Chatbot Arena In CS.CL Papers LMSYS Chatbot Arena + Pairwise Preference Benchmark Papers LMSYS Chatbot Arena In CS.AI Papers CS.CL Human Feedback And Eval Papers LMSYS Chatbot Arena + General Benchmark Papers WebArena Ecosystem Benchmark Papers (15) Reasoning & Math Suite Benchmark Papers (16) SWE-Bench Ecosystem Benchmark Papers (10) WebArena Ecosystem Benchmark Papers In CS.CL (10) Reasoning & Math Suite Benchmark Papers In CS.CL (15) Arena/Judge Suite Benchmark Papers (10) Reasoning & Math Suite Benchmark Papers In CS.AI (13)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote