Skip to content
← Back to explorer

HFEPX Metric Hub

Agreement + General Metric Papers (Last 45 Days)

Updated from current HFEPX corpus (Apr 9, 2026). 11 papers are grouped in this metric page.

Read Full Context

Updated from current HFEPX corpus (Apr 9, 2026). 11 papers are grouped in this metric page. Common evaluation modes: Automatic Metrics, Human Eval. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Inter Annotator Agreement Reported. Frequently cited benchmark: Interaction2eval. Common metric signal: agreement. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Mar 31, 2026.

Papers: 11 Last published: Mar 31, 2026 Global RSS

Researcher Quick Triage

Use this page to compare metric behavior across protocols and benchmarks before selecting your reporting stack. Quality band: Developing .

Metric Coverage

90.9%

10 sampled papers include metric names.

Benchmark Anchoring

9.1%

Papers with explicit dataset/benchmark anchors for fair comparison.

Quality Controls

54.5%

6 papers report calibration/adjudication/IAA controls.

  • 11 papers are not low-signal flagged in this sample.
  • Use the protocol matrix below to avoid comparing metrics across incompatible eval setups.

Primary action: Treat this as directional signal only; metric reporting is present but benchmark anchoring is still thin.

Why This Matters (Expanded)

Why This Matters For Eval Research

  • 90.9% of papers report explicit human-feedback signals, led by rubric ratings.
  • automatic metrics appears in 54.5% of papers in this hub.
  • Interaction2eval is a recurring benchmark anchor for cross-paper comparisons in this page.
Metric Notes (Expanded)

Metric-Driven Protocol Takeaways

  • Most common quality-control signal is inter-annotator agreement reporting (54.5% of papers).
  • Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Metric Interpretation

  • agreement is reported in 100% of hub papers (11/11); compare with a secondary metric before ranking methods.
  • accuracy is reported in 18.2% of hub papers (2/11); compare with a secondary metric before ranking methods.

Benchmark Context

  • Interaction2eval appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.

Start Here (Metric-Reliable First 6)

Ranked for metric reporting completeness and comparability.

Metric Protocol Matrix (Top 10)

Compare metric, benchmark, and evaluation context side by side.

Paper Metrics Benchmarks Eval Modes Quality Controls
LLM Essay Scoring Under Holistic and Analytic Rubrics: Prompt Effects and Bias

Mar 31, 2026

 Kappa, Agreement Not reported Human Eval Inter Annotator Agreement Reported, Adjudication
More Human, More Efficient: Aligning Annotations with Quantized SLMs

Apr 1, 2026

 Agreement Not reported Automatic Metrics Inter Annotator Agreement Reported, Adjudication
When AI Meets Early Childhood Education: Large Language Models as Assessment Teammates in Chinese Preschools

Mar 25, 2026

 Agreement, Cost Interaction2eval Automatic Metrics Not reported
From Consensus to Split Decisions: ABC-Stratified Sentiment in Holocaust Oral Histories

Mar 30, 2026

 Kappa, Agreement Not reported Automatic Metrics Inter Annotator Agreement Reported
ReasonScaffold: A Scaffolded Reasoning-based Annotation Protocol for Human-AI Co-Annotation

Mar 22, 2026

 Accuracy, Agreement Not reported Automatic Metrics Inter Annotator Agreement Reported
Same Words, Different Judgments: Modality Effects on Preference Alignment

Feb 26, 2026

 Agreement Not reported Automatic Metrics Inter Annotator Agreement Reported
Strengthening Human-Centric Chain-of-Thought Reasoning Integrity in LLMs via a Structured Prompt Framework

Apr 6, 2026

 Accuracy, Agreement Not reported Human Eval, Automatic Metrics Inter Annotator Agreement Reported
Beyond the Illusion of Consensus: From Surface Heuristics to Knowledge-Grounded Evaluation in LLM-as-a-Judge

Mar 11, 2026

 Spearman Not reported Llm As Judge Not reported
Evaluating Austrian A-Level German Essays with Large Language Models for Automated Essay Scoring

Mar 6, 2026

 Agreement Not reported Human Eval Not reported
PONTE: Personalized Orchestration for Natural Language Trustworthy Explanations

Mar 6, 2026

 Agreement, Faithfulness Not reported Human Eval Not reported
Researcher Workflow (Detailed)

Checklist

  • Strong: Papers with explicit human feedback

    Coverage is strong (90.9% vs 45% target).

  • Strong: Papers reporting quality controls

    Coverage is strong (54.5% vs 30% target).

  • Gap: Papers naming benchmarks/datasets

    Coverage is a replication risk (9.1% vs 35% target).

  • Strong: Papers naming evaluation metrics

    Coverage is strong (100% vs 35% target).

  • Strong: Papers with known rater population

    Coverage is strong (45.5% vs 35% target).

  • Strong: Papers with known annotation unit

    Coverage is strong (54.5% vs 35% target).

Strengths

  • Strong human-feedback signal (90.9% of papers).
  • Quality-control evidence appears in 54.5% of papers.
  • Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.

Known Gaps

  • Benchmark coverage is thin (9.1% of papers mention benchmarks/datasets).

Suggested Next Analyses

  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
  • Track metric sensitivity by reporting both agreement and accuracy.

Recommended Queries

Judge vs Human AgreementBenchmark Slice: Interaction2evalMetric Slice: agreementIAA-Reported EvaluationsRecent High-Signal Papers
Known Limitations
  • Benchmark coverage is thin (9.1% of papers mention benchmarks/datasets).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
  • Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.
Research Utility Snapshot (Detailed)

Top Metrics

  • Agreement (11)
  • Accuracy (2)
  • Kappa (2)
  • Cost (1)

Evaluation Modes

  • Automatic Metrics (6)
  • Human Eval (4)
  • Llm As Judge (1)

Top Benchmarks

  • Interaction2eval (1)

Agentic Mix

Top Papers Reporting This Metric

Related Metric Hubs

Agreement + General Metric Papers (Last 60 Days) Agreement + General Metric Papers (Last 90 Days) Agreement + General Metric Papers (Last 120 Days) Agreement + General Metric Papers Agreement & Reliability Metric Papers + General Agreement Metric Papers Helpfulness In CS.CL Papers (12) Helpfulness Metric Papers (15) Relevance + Automatic Metrics Metric Papers (Last 120 Days) (18) Relevance + Automatic Metrics Metric Papers (Last 30 Days) (10) Relevance + Automatic Metrics Metric Papers (Last 45 Days) (14) Relevance + Automatic Metrics Metric Papers (Last 60 Days) (16) Relevance + Automatic Metrics Metric Papers (Last 90 Days) (17) Relevance + Automatic Metrics Metric Papers (22) Relevance + General Metric Papers (Last 120 Days) (11) Relevance + General Metric Papers (Last 45 Days) (10) Relevance + General Metric Papers (Last 60 Days) (11) Relevance + General Metric Papers (Last 90 Days) (11) Relevance + General Metric Papers (12) Relevance + Pairwise Preference Metric Papers (Last 120 Days) (10) Relevance + Pairwise Preference Metric Papers (11) Relevance In CS.CL Papers (40) Relevance In CS.LG Papers (13) Relevance Metric Papers (65)

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote