Skip to content
← Back to explorer

HFEPX Hub

Llm As Judge + General (Last 60 Days)

Updated from current HFEPX corpus (Mar 8, 2026). 11 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 11 papers are grouped in this hub page. Common evaluation modes: Llm As Judge, Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Pairwise. Frequent quality control: Adjudication. Frequently cited benchmark: Innoeval. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 16, 2026.

Papers: 11 Last published: Feb 16, 2026 Global RSS Tag RSS
Llm As JudgeGeneralLast 60d

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

All Sampled Papers (11) Replication-Ready Only (1)

High-Signal Coverage

100.0%

11 / 11 sampled papers are not low-signal flagged.

Replication-Ready Set

1

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

1

Papers containing both `human_eval` and `llm_as_judge`.

  • 1 papers are replication-ready (benchmark + metric + explicit evaluation mode).
  • 1 papers support judge-vs-human agreement analysis.
  • 1 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Why This Matters For Eval Research

  • 18.2% of papers report explicit human-feedback signals, led by pairwise preferences.
  • LLM-as-judge appears in 100% of papers in this hub.
  • Innoeval is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

  • 1 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.
  • Most common quality-control signal is adjudication (9.1% of papers).
  • Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.

Benchmark Interpretation

  • Innoeval appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.
  • Mind2Web appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • accuracy is reported in 27.3% of hub papers (3/11); compare with a secondary metric before ranking methods.
  • cost is reported in 18.2% of hub papers (2/11); compare with a secondary metric before ranking methods.
Researcher Checklist (Expanded)

Researcher Checklist

  • Gap: Papers with explicit human feedback

    Coverage is a replication risk (18.2% vs 45% target).

  • Gap: Papers reporting quality controls

    Coverage is a replication risk (9.1% vs 30% target).

  • Moderate: Papers naming benchmarks/datasets

    Coverage is usable but incomplete (27.3% vs 35% target).

  • Strong: Papers naming evaluation metrics

    Coverage is strong (63.6% vs 35% target).

  • Gap: Papers with known rater population

    Coverage is a replication risk (18.2% vs 35% target).

  • Moderate: Papers with known annotation unit

    Coverage is usable but incomplete (27.3% vs 35% target).

Strengths

  • Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
  • Agentic evaluation appears in 27.3% of papers.

Known Gaps

  • Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (18.2% coverage).
  • LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

  • Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
  • Stratify by benchmark (Innoeval vs Mind2Web) before comparing methods.
  • Track metric sensitivity by reporting both accuracy and cost.
  • Add inter-annotator agreement checks when reproducing these protocols.
Recommended Queries (Expanded)

Recommended Queries

Judge vs Human Agreement Benchmark Slice: Innoeval Metric Slice: accuracy Recent High-Signal Papers

Start with These 3

Use these when you need one protocol anchor, one benchmark anchor, and one recent comparison point before reading the wider hub.

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper HF Signal Eval Modes Benchmarks Metrics QC
HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

Jan 9, 2026

 Yes
Pairwise Preference , Rubric Rating
Human Eval , Llm As Judge Not Reported Agreement Not Reported
InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

Feb 16, 2026

 No
Not Reported
Llm As Judge Innoeval Not Reported Adjudication
ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

Mar 5, 2026

 No
Not Reported
Llm As Judge , Automatic Metrics Thaisafetybench F1 , F1 weighted Not Reported
Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

Feb 15, 2026

 Yes
Pairwise Preference , Rubric Rating
Llm As Judge Not Reported Not Reported Not Reported
World-Model-Augmented Web Agents with Action Correction

Feb 17, 2026

 No
Not Reported
Llm As Judge , Simulation Env VisualWebArena , Mind2Web Not Reported Not Reported
The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

Feb 19, 2026

 No
Not Reported
Llm As Judge , Automatic Metrics Not Reported Accuracy Not Reported
Overton Pluralistic Reinforcement Learning for Large Language Models

Feb 24, 2026

 No
Not Reported
Llm As Judge , Automatic Metrics Not Reported Accuracy Not Reported
Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning

Feb 24, 2026

 No
Not Reported
Llm As Judge , Automatic Metrics Not Reported Coherence Not Reported
Luna-2: Scalable Single-Token Evaluation with Small Language Models

Feb 20, 2026

 No
Not Reported
Llm As Judge , Automatic Metrics Not Reported Accuracy , Latency Not Reported
CAST: Character-and-Scene Episodic Memory for Agents

Jan 14, 2026

 No
Not Reported
Llm As Judge , Automatic Metrics Not Reported F1 , Recall Not Reported
Eval4Sim: An Evaluation Framework for Persona Simulation

Mar 3, 2026

 No
Not Reported
Llm As Judge , Simulation Env Not Reported Not Reported Not Reported

Protocol Diff (Top Papers)

Fast side-by-side comparison for the highest-ranked papers in this hub.

Signal HEART: A Unified Benchmark for Assessing Humans and… InnoEval: On Research Idea Evaluation as a Knowledg… ThaiSafetyBench: Assessing Language Model Safety in…
Human Feedback Pairwise Preference, Rubric RatingNot reportedNot reported
Evaluation Modes Human Eval, Llm As JudgeLlm As JudgeLlm As Judge, Automatic Metrics
Benchmarks Not reportedInnoevalThaisafetybench
Metrics AgreementNot reportedF1, F1 weighted
Quality Controls Not reportedAdjudicationNot reported
Rater Population UnknownDomain ExpertsUnknown
Annotation Unit PairwiseUnknownUnknown
Suggested Reading Order (Extended)

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

  1. ThaiSafetyBench: Assessing Language Model Safety in Thai Cultural Contexts

    Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: Thaisafetybench / f1. Abstract: Using ThaiSafetyBench, we evaluate 24 LLMs, with GPT-4.1 and Gemini-2.5-Pro serving as.

  2. Eval4Sim: An Evaluation Framework for Persona Simulation

    Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Abstract: However, current evaluation practices largely rely on LLM-as-a-judge approaches, offering limited grounding in observable human behavior.

  3. Overton Pluralistic Reinforcement Learning for Large Language Models

    Start here for detailed protocol reporting and quality-control evidence. Signals: LLM-as-judge. Focus: accuracy. Abstract: Additional evaluations using GPT-4.1 as a large language model judge further confirm the robustness.

  4. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: agreement. Abstract: For each dialogue history, we pair human and model.

  5. InnoEval: On Research Idea Evaluation as a Knowledge-Grounded, Multi-Perspective Reasoning Problem

    Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: LLM-as-judge. Focus: Innoeval. Abstract: However, existing idea evaluation methods often suffer from narrow knowledge horizons, flattened evaluation.

  6. Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric

    Adds LLM-as-judge with pairwise preferences for broader protocol coverage within this hub. Signals: LLM-as-judge + pairwise preferences. Abstract: Scalar reward models compress multi-dimensional human preferences into a single.

  7. World-Model-Augmented Web Agents with Action Correction

    Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which a judge.

  8. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

    Adds LLM-as-judge for broader protocol coverage within this hub. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational reasoning layers.

Known Limitations

Known Limitations

  • Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (18.2% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Research Utility Snapshot (Detailed)

Research Utility Snapshot

Human Feedback Mix

  • Pairwise Preference (2)
  • Rubric Rating (2)

Evaluation Modes

  • Llm As Judge (11)
  • Automatic Metrics (6)
  • Simulation Env (2)
  • Human Eval (1)

Top Benchmarks

  • Innoeval (1)
  • Mind2Web (1)
  • Thaisafetybench (1)
  • VisualWebArena (1)

Top Metrics

  • Accuracy (3)
  • Cost (2)
  • F1 (2)
  • Agreement (1)

Rater Population Mix

  • Domain Experts (2)

Quality Controls

  • Adjudication (1)
Coverage diagnostics (sample-based): human-feedback 18.2% · benchmarks 27.3% · metrics 63.6% · quality controls 9.1%.

Top Papers

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote