Skip to content
← Back to explorer

HFEPX Hub

CS.LG + Rubric Rating Papers

Updated from current HFEPX corpus (Mar 10, 2026). 11 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 10, 2026). 11 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Human Eval. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Inter Annotator Agreement Reported. Frequently cited benchmark: Onemillion-Bench. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Mar 9, 2026.

Papers: 11 Last published: Mar 9, 2026 Global RSS Tag RSS
Cs.LGRubric Rating

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

All Sampled Papers (11) Replication-Ready Only (1)

High-Signal Coverage

100.0%

11 / 11 sampled papers are not low-signal flagged.

Replication-Ready Set

1

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

0

Papers containing both `human_eval` and `llm_as_judge`.

  • 1 papers are replication-ready (benchmark + metric + explicit evaluation mode).
  • 0 papers support judge-vs-human agreement analysis.
  • 1 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Need evaluators for this research workflow?

Post a Job →

Why This Matters For Eval Research

  • 100% of papers report explicit human-feedback signals, led by rubric ratings.
  • automatic metrics appears in 36.4% of papers in this hub.
  • Onemillion-Bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

  • Most common quality-control signal is inter-annotator agreement reporting (9.1% of papers).
  • Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

  • Onemillion-Bench appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.
  • Scholareval appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • accuracy is reported in 9.1% of hub papers (1/11); compare with a secondary metric before ranking methods.
  • agreement is reported in 9.1% of hub papers (1/11); compare with a secondary metric before ranking methods.
Researcher Checklist (Expanded)

Researcher Checklist

  • Strong: Papers with explicit human feedback

    Coverage is strong (100% vs 45% target).

  • Gap: Papers reporting quality controls

    Coverage is a replication risk (9.1% vs 30% target).

  • Gap: Papers naming benchmarks/datasets

    Coverage is a replication risk (18.2% vs 35% target).

  • Moderate: Papers naming evaluation metrics

    Coverage is usable but incomplete (27.3% vs 35% target).

  • Strong: Papers with known rater population

    Coverage is strong (45.5% vs 35% target).

  • Strong: Papers with known annotation unit

    Coverage is strong (100% vs 35% target).

Strengths

  • Strong human-feedback signal (100% of papers).
  • Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
  • Agentic evaluation appears in 27.3% of papers.

Known Gaps

  • Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Benchmark coverage is thin (18.2% of papers mention benchmarks/datasets).

Suggested Next Analyses

  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
  • Stratify by benchmark (Onemillion-Bench vs Scholareval) before comparing methods.
  • Track metric sensitivity by reporting both accuracy and agreement.
Recommended Queries (Expanded)

Recommended Queries

Judge vs Human Agreement Benchmark Slice: Onemillion-Bench Metric Slice: accuracy IAA-Reported Evaluations Recent High-Signal Papers

Start with These 3

Use these when you need one protocol anchor, one benchmark anchor, and one recent comparison point before reading the wider hub.

Strongest recent paper

APEX-Agents

Useful for current practice scanning; published Jan 20, 2026.

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper HF Signal Eval Modes Benchmarks Metrics QC
\$OneMillion-Bench: How Far are Language Agents from Human Experts?

Mar 9, 2026

 Yes
Rubric Rating
Automatic Metrics Onemillion Bench Accuracy , Coherence Not Reported
MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages

Sep 30, 2025

 Yes
Pairwise Preference , Rubric Rating
Automatic Metrics Not Reported Agreement Inter Annotator Agreement Reported
APEX-Agents

Jan 20, 2026

 Yes
Rubric Rating , Expert Verification
Automatic Metrics Not Reported Pass@1 Not Reported
ScholarEval: Research Idea Evaluation Grounded in Literature

Oct 17, 2025

 Yes
Rubric Rating
Not Reported Scholareval Not Reported Not Reported
Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

Mar 3, 2026

 Yes
Pairwise Preference , Rubric Rating
Llm As Judge , Simulation Env Not Reported Not Reported Not Reported
Discovering Implicit Large Language Model Alignment Objectives

Feb 17, 2026

 Yes
Rubric Rating
Human Eval Not Reported Not Reported Not Reported
SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing

Feb 24, 2026

 Yes
Rubric Rating , Red Team
Not Reported Not Reported Not Reported Not Reported
Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

Sep 25, 2025

 Yes
Rubric Rating
Automatic Metrics Not Reported Not Reported Not Reported
Toward LLM-Supported Automated Assessment of Critical Thinking Subskills

Oct 14, 2025

 Yes
Rubric Rating
Not Reported Not Reported Not Reported Not Reported
Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference

May 19, 2025

 Yes
Rubric Rating
Not Reported Not Reported Not Reported Not Reported
RM-R1: Reward Modeling as Reasoning

May 5, 2025

 Yes
Pairwise Preference , Rubric Rating
Not Reported Not Reported Not Reported Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal \$OneMillion-Bench: How Far are Language Agents fro… MENLO: From Preferences to Proficiency -- Evaluatin… APEX-Agents
Human Feedback Rubric RatingPairwise Preference, Rubric RatingRubric Rating, Expert Verification
Evaluation Modes Automatic MetricsAutomatic MetricsAutomatic Metrics
Benchmarks Onemillion BenchNot reportedNot reported
Metrics Accuracy, CoherenceAgreementPass@1
Quality Controls Not reportedInter Annotator Agreement ReportedNot reported
Rater Population Domain ExpertsUnknownDomain Experts
Annotation Unit Multi Dim RubricPairwiseMulti Dim Rubric
Suggested Reading Order (Extended)

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

  1. \$OneMillion-Bench: How Far are Language Agents from Human Experts?

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + rubric ratings. Focus: Onemillion-Bench / accuracy. Abstract: We adopt a rubric-based evaluation protocol scoring factual.

  2. Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

    Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge + pairwise preferences. Abstract: Grocery shopping further amplifies these difficulties, as user requests are often underspecified, highly.

  3. SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing

    Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Abstract: Rubrics provide structured, interpretable supervision, but scaling rubric construction is difficult: expert rubrics are costly,.

  4. Discovering Implicit Large Language Model Alignment Objectives

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + rubric ratings. Abstract: Existing interpretation methods typically rely on pre-defined rubrics, risking the omission.

  5. APEX-Agents

    Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Focus: pass@1. Abstract: We open source the APEX-Agents benchmark.

  6. MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages

    Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: agreement. Abstract: Using MENLO, we create a dataset.

  7. ScholarEval: Research Idea Evaluation Grounded in Literature

    Adds evaluation protocol evidence with rubric ratings for broader protocol coverage within this hub. Signals: rubric ratings. Focus: Scholareval. Abstract: Our evaluation shows that ScholarEval achieves significantly higher.

  8. Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

    Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Abstract: To address this, we study rubric-based rewards.

Known Limitations

Known Limitations

  • Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Benchmark coverage is thin (18.2% of papers mention benchmarks/datasets).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Research Utility Snapshot (Detailed)

Research Utility Snapshot

Human Feedback Mix

  • Rubric Rating (11)
  • Pairwise Preference (3)
  • Expert Verification (1)
  • Red Team (1)

Evaluation Modes

  • Automatic Metrics (4)
  • Human Eval (1)
  • Llm As Judge (1)
  • Simulation Env (1)

Top Benchmarks

  • Onemillion Bench (1)
  • Scholareval (1)

Top Metrics

  • Accuracy (1)
  • Agreement (1)
  • Coherence (1)
  • Pass@1 (1)

Rater Population Mix

  • Domain Experts (5)

Quality Controls

  • Inter Annotator Agreement Reported (1)
Coverage diagnostics (sample-based): human-feedback 100.0% · benchmarks 18.2% · metrics 27.3% · quality controls 9.1%.

Top Papers

  • \$OneMillion-Bench: How Far are Language Agents from Human Experts?

    Qianyu Yang, Yang Liu, Jiaqi Li, Jun Bai, Hao Chen · Mar 9, 2026 · Citations: 0

    Rubric Rating Automatic Metrics Tool Use

    To this end, we introduce \OneMillion-Bench \OneMillion-Bench, a benchmark of 400 expert-curated tasks spanning Law, Finance, Industry, Healthcare, and Natural Science, built to evaluate agents across economically consequential scenarios.

  • APEX-Agents

    Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman · Jan 20, 2026 · Citations: 0

    Rubric RatingExpert Verification Automatic Metrics Long Horizon

    We introduce the AI Productivity Index for Agents (APEX-Agents), a benchmark for assessing whether AI agents can execute long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate…

  • Build, Judge, Optimize: A Blueprint for Continuous Improvement of Multi-Agent Consumer Assistants

    Alejandro Breen Herrera, Aayush Sheth, Steven G. Xu, Zhucheng Zhan, Charles Wright · Mar 3, 2026 · Citations: 0

    Pairwise PreferenceRubric Rating Llm As JudgeSimulation Env Long Horizon

    Conversational shopping assistants (CSAs) represent a compelling application of agentic AI, but moving from prototype to production reveals two underexplored challenges: how to evaluate multi-turn interactions and how to optimize tightly…

  • MENLO: From Preferences to Proficiency -- Evaluating and Modeling Native-like Quality Across 47 Languages

    Chenxi Whitehouse, Sebastian Ruder, Tony Lin, Oksana Kurylo, Haruka Takagi · Sep 30, 2025 · Citations: 0

    Pairwise PreferenceRubric Rating Automatic Metrics

    To address this, we introduce MENLO, a framework that operationalizes the evaluation of native-like response quality based on audience design-inspired mechanisms.

  • Discovering Implicit Large Language Model Alignment Objectives

    Edward Chen, Sanmi Koyejo, Carlos Guestrin · Feb 17, 2026 · Citations: 0

    Rubric Rating Human Eval

    To address these limitations, we introduce Obj-Disco, a framework that automatically decomposes an alignment reward signal into a sparse, weighted combination of human-interpretable natural language objectives.

  • ScholarEval: Research Idea Evaluation Grounded in Literature

    Hanane Nour Moussa, Patrick Queiroz Da Silva, Daniel Adu-Ampratwum, Alyson East, Zitong Lu · Oct 17, 2025 · Citations: 0

    Rubric Rating

    As AI tools become increasingly common for research ideation, robust evaluation is critical to ensure the validity and usefulness of generated ideas.

  • SibylSense: Adaptive Rubric Learning via Memory Tuning and Adversarial Probing

    Yifei Xu, Guilherme Potje, Shivam Shandilya, Tiancheng Yuan, Leonardo de Oliveira Nunes · Feb 24, 2026 · Citations: 0

    Rubric RatingRed Team

    We present SibylSense, an inference-time learning approach that adapts a frozen rubric generator through a tunable memory bank of validated rubric items.

  • Ice Cream Doesn't Cause Drowning: Benchmarking LLMs Against Statistical Pitfalls in Causal Inference

    Jin Du, Li Chen, Xun Xian, An Luo, Fangqiao Tian · May 19, 2025 · Citations: 0

    Rubric Rating

    Current benchmarks usually involve simplified tasks.

  • Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training

    Junkai Zhang, Zihao Wang, Lin Gui, Swarnashree Mysore Sathyendra, Jaehwan Jeong · Sep 25, 2025 · Citations: 0

    Rubric Rating Automatic Metrics

    Reinforcement fine-tuning (RFT) often suffers from reward over-optimization, where a policy model hacks the reward signals to achieve high scores while producing low-quality outputs.

  • Toward LLM-Supported Automated Assessment of Critical Thinking Subskills

    Marisa C. Peczuh, Nischal Ashok Kumar, Ryan Baker, Blair Lehman, Danielle Eisenberg · Oct 14, 2025 · Citations: 0

    Rubric Rating

    As the world becomes increasingly saturated with AI-generated content, disinformation, and algorithmic persuasion, critical thinking - the capacity to evaluate evidence, detect unreliable claims, and exercise independent judgment - is…

  • RM-R1: Reward Modeling as Reasoning

    Xiusi Chen, Gaotang Li, Ziqi Wang, Bowen Jin, Cheng Qian · May 5, 2025 · Citations: 0

    Pairwise PreferenceRubric Rating

    Reward modeling is essential for aligning large language models with human preferences through reinforcement learning.

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote