Skip to content
← Back to explorer

HFEPX Hub

CS.CL Papers (Last 90 Days)

Updated from current HFEPX corpus (Mar 1, 2026). 705 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 1, 2026). 705 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Simulation Env. Most common rater population: Domain Experts. Common annotation unit: Trajectory. Frequent quality control: Calibration. Frequently cited benchmark: BrowseComp. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 15, 2026.

Papers: 705 Last published: Feb 15, 2026 Global RSS
Cs.CLLast 90d

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Medium .

Analysis blocks below are computed from the currently loaded sample (60 of 705 total papers in this hub).

High-Signal Coverage

100.0%

60 / 60 sampled papers are not low-signal flagged.

Replication-Ready Set

11

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

1

Papers containing both `human_eval` and `llm_as_judge`.

  • 11 papers are replication-ready (benchmark + metric + explicit evaluation mode).
  • 1 papers support judge-vs-human agreement analysis.
  • 12 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Start with the top 2 papers in “Start Here”, then validate assumptions in the protocol matrix.

Why This Matters (Expanded)

Why This Matters For Eval Research

  • 14.2% of papers report explicit human-feedback signals, led by pairwise preferences.
  • automatic metrics appears in 40.9% of papers in this hub.
  • BrowseComp is a recurring benchmark anchor for cross-paper comparisons in this page.
Protocol Notes (Expanded)

Protocol Takeaways

  • 1 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
  • Most common quality-control signal is rater calibration (2.6% of papers).
  • Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.

Benchmark Interpretation

  • BrowseComp appears in 0.4% of hub papers (3/705); use this cohort for benchmark-matched comparisons.
  • LiveCodeBench appears in 0.4% of hub papers (3/705); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • accuracy is reported in 6% of hub papers (42/705); compare with a secondary metric before ranking methods.
  • cost is reported in 2.6% of hub papers (18/705); compare with a secondary metric before ranking methods.
Researcher Checklist (Expanded)

Researcher Checklist

  • Gap: Papers with explicit human feedback

    Coverage is a replication risk (14.2% vs 45% target).

  • Gap: Papers reporting quality controls

    Coverage is a replication risk (4.8% vs 30% target).

  • Gap: Papers naming benchmarks/datasets

    Coverage is a replication risk (4.5% vs 35% target).

  • Gap: Papers naming evaluation metrics

    Coverage is a replication risk (11.8% vs 35% target).

  • Gap: Papers with known rater population

    Coverage is a replication risk (9.9% vs 35% target).

  • Gap: Papers with known annotation unit

    Coverage is a replication risk (12.2% vs 35% target).

Strengths

  • Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

  • Only 4.8% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (9.9% coverage).
  • Annotation unit is under-specified (12.2% coverage).

Suggested Next Analyses

  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
  • Stratify by benchmark (BrowseComp vs LiveCodeBench) before comparing methods.
  • Track metric sensitivity by reporting both accuracy and cost.
Recommended Queries (Expanded)

Recommended Queries

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper HF Signal Eval Modes Benchmarks Metrics QC
HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

Feb 15, 2026

Yes Automatic Metrics HLE Accuracy Adjudication
SCOPE: Selective Conformal Optimized Pairwise LLM Judging

Feb 13, 2026

Yes Automatic Metrics MT Bench , LMSYS Chatbot Arena Error rate Calibration
CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

Dec 26, 2025

Yes Automatic Metrics DROP , BIRD Accuracy Gold Questions
AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

Feb 15, 2026

Yes Simulation Env Ad Bench Pass@1 , Pass@3 Not Reported
Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling

Feb 18, 2026

Yes Not Reported LiveCodeBench Not Reported Calibration
Step 3.5 Flash: Open Frontier-Level Intelligence with 11B Active Parameters

Feb 11, 2026

Yes Not Reported LiveCodeBench , BrowseComp Latency , Cost Not Reported
Document Reconstruction Unlocks Scalable Long-Context RLVR

Feb 9, 2026

Yes Automatic Metrics LongBench Coherence Not Reported
MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

Feb 18, 2026

Yes Automatic Metrics Memoryarena Recall Not Reported
Duel-Evolve: Reward-Free Test-Time Scaling via LLM Self-Preferences

Feb 25, 2026

Yes Automatic Metrics LiveCodeBench , Mathbench Accuracy Not Reported
Can Large Language Models Replace Human Coders? Introducing ContentBench

Feb 23, 2026

Yes Automatic Metrics ContentBench Agreement , Cost Not Reported
Validating Political Position Predictions of Arguments

Feb 20, 2026

Yes Human Eval Not Reported Agreement Gold Questions , Inter Annotator Agreement Reported
An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

Feb 23, 2026

Yes Automatic Metrics Not Reported F1 , Precision Gold Questions

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal HLE-Verified: A Systematic Verification and Structu… SCOPE: Selective Conformal Optimized Pairwise LLM J… CricBench: A Multilingual Benchmark for Evaluating…
Human Feedback Expert Verification, Critique EditPairwise PreferenceExpert Verification
Evaluation Modes Automatic MetricsAutomatic MetricsAutomatic Metrics
Benchmarks HLEMT Bench, LMSYS Chatbot ArenaDROP, BIRD
Metrics AccuracyError rateAccuracy
Quality Controls AdjudicationCalibrationGold Questions
Rater Population Domain ExpertsUnknownDomain Experts
Annotation Unit UnknownPairwiseUnknown
Suggested Reading Order (Extended)

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

  1. An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + expert verification. Focus: f1. Abstract: Phenotyping is fundamental to rare disease diagnosis, but manual curation.

  2. Cross-Cultural Expert-Level Art Critique Evaluation with Vision-Language Models

    Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Abstract: Existing benchmarks assess perception without interpretation, and common evaluation proxies, such as automated metrics and.

  3. Validating Political Position Predictions of Arguments

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: Real-world knowledge representation often requires capturing subjective, continuous attributes.

  4. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

  5. InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

    Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: Innoeval. Abstract: However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened.

Known Limitations

Known Limitations

  • Only 4.8% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (9.9% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Research Utility Snapshot (Detailed)

Research Utility Snapshot

Human Feedback Mix

  • Pairwise Preference (48)
  • Expert Verification (17)
  • Rubric Rating (16)
  • Critique Edit (13)

Evaluation Modes

  • Automatic Metrics (288)
  • Simulation Env (21)
  • Llm As Judge (19)
  • Human Eval (15)

Top Benchmarks

  • BrowseComp (3)
  • LiveCodeBench (3)
  • ALFWorld (2)
  • MMLU (2)

Top Metrics

  • Accuracy (42)
  • Cost (18)
  • Latency (13)
  • F1 (7)

Rater Population Mix

  • Domain Experts (66)
  • Crowd (2)
  • Mixed (2)

Quality Controls

  • Calibration (18)
  • Inter Annotator Agreement Reported (10)
  • Adjudication (6)
  • Gold Questions (3)
Coverage diagnostics (sample-based): human-feedback 85.0% · benchmarks 38.3% · metrics 60.0% · quality controls 20.0%.

Top Papers

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.