Skip to content
← Back to explorer

HFEPX Hub

CS.CL + Llm As Judge Papers

Updated from current HFEPX corpus (Feb 27, 2026). 9 papers are grouped in this hub page. Common evaluation modes: Llm As Judge, Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Inter Annotator Agreement Reported. Frequently cited benchmark: Caparena. Common metric signal: agreement. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 24, 2026.

Papers: 9 Last published: Feb 24, 2026 Global RSS Tag RSS
Cs.CLLlm As Judge

Research Narrative

Grounded narrative Model: deterministic-grounded Source: persisted

Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 9 papers for CS.CL + Llm As Judge Papers. Dominant protocol signals include LLM-as-judge, automatic metrics, human evaluation, with frequent benchmark focus on Caparena, Visualwebarena and metric focus on agreement, accuracy. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.

Why This Matters For Eval Research

Protocol Takeaways

Benchmark Interpretation

  • Caparena appears in 11.1% of hub papers (1/9); use this cohort for benchmark-matched comparisons.
  • Visualwebarena appears in 11.1% of hub papers (1/9); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • agreement is reported in 33.3% of hub papers (3/9); compare with a secondary metric before ranking methods.
  • accuracy is reported in 11.1% of hub papers (1/9); compare with a secondary metric before ranking methods.

Researcher Checklist

  • Tighten coverage on Papers with explicit human feedback. Coverage is usable but incomplete (33.3% vs 45% target).
  • Tighten coverage on Papers reporting quality controls. Coverage is usable but incomplete (22.2% vs 30% target).
  • Tighten coverage on Papers naming benchmarks/datasets. Coverage is usable but incomplete (22.2% vs 35% target).
  • Maintain strength on Papers naming evaluation metrics. Coverage is strong (77.8% vs 35% target).
  • Tighten coverage on Papers with known rater population. Coverage is usable but incomplete (33.3% vs 35% target).
  • Tighten coverage on Papers with known annotation unit. Coverage is usable but incomplete (22.2% vs 35% target).

Papers with explicit human feedback

Coverage is usable but incomplete (33.3% vs 45% target).

Papers reporting quality controls

Coverage is usable but incomplete (22.2% vs 30% target).

Papers naming benchmarks/datasets

Coverage is usable but incomplete (22.2% vs 35% target).

Papers naming evaluation metrics

Coverage is strong (77.8% vs 35% target).

Papers with known rater population

Coverage is usable but incomplete (33.3% vs 35% target).

Papers with known annotation unit

Coverage is usable but incomplete (22.2% vs 35% target).

Suggested Reading Order

  1. 1. Overton Pluralistic Reinforcement Learning for Large Language Models

    Start here for detailed protocol reporting, including rater and quality-control evidence.

  2. 2. Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning

    Start here for detailed protocol reporting, including rater and quality-control evidence.

  3. 3. MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

    Start here for detailed protocol reporting, including rater and quality-control evidence.

  4. 4. HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue

    Include a human-eval paper to anchor calibration against automated judge settings.

  5. 5. PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions

    Include a human-eval paper to anchor calibration against automated judge settings.

  6. 6. World-Model-Augmented Web Agents with Action Correction

    Adds LLM-as-judge for broader coverage within this hub.

  7. 7. Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios

    Adds LLM-as-judge for broader coverage within this hub.

  8. 8. DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

    Adds LLM-as-judge with expert verification for broader coverage within this hub.

Known Limitations

  • Annotation unit is under-specified (22.2% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
  • Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Links

human_eval vs llm_as_judge

both=2, left_only=0, right_only=7

2 papers use both Human Eval and Llm As Judge.

human_eval vs automatic_metrics

both=0, left_only=2, right_only=2

0 papers use both Human Eval and Automatic Metrics.

llm_as_judge vs automatic_metrics

both=2, left_only=7, right_only=0

2 papers use both Llm As Judge and Automatic Metrics.

Benchmark Brief

Caparena

Coverage: 1 papers (11.1%)

1 papers (11.1%) mention Caparena.

Examples: PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions

Benchmark Brief

Visualwebarena

Coverage: 1 papers (11.1%)

1 papers (11.1%) mention Visualwebarena.

Examples: World-Model-Augmented Web Agents with Action Correction

Metric Brief

agreement

Coverage: 3 papers (33.3%)

3 papers (33.3%) mention agreement.

Examples: MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents , HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue , Human-like Affective Cognition in Foundation Models

Metric Brief

accuracy

Coverage: 1 papers (11.1%)

1 papers (11.1%) mention accuracy.

Examples: Overton Pluralistic Reinforcement Learning for Large Language Models

Metric Brief

auroc

Coverage: 1 papers (11.1%)

1 papers (11.1%) mention auroc.

Examples: DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

Most Cited In This Hub

Fast path to methods with the strongest citation traction in this scope.

Papers: Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning , MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Most Recent

Fast path to latest protocol changes and newly published evaluation setups.

Papers: Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning , MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Best Protocol Detail

Papers with explicit rater/unit metadata and quality-control signals for reproducibility.

Papers: Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning , MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Top Papers

Related Hubs