HFEPX Hub

Llm As Judge Papers

Updated from current HFEPX corpus (Feb 27, 2026). 10 papers are grouped in this hub page. Common evaluation modes: Llm As Judge, Automatic Metrics. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Inter Annotator Agreement Reported. Frequently cited benchmark: Caparena. Common metric signal: agreement. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 24, 2026.

Papers: 10 Last published: Feb 24, 2026 Global RSS Tag RSS

Llm As Judge

Research Narrative

Grounded narrative Model: deterministic-grounded Source: persisted

Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 10 papers for Llm As Judge Papers. Dominant protocol signals include LLM-as-judge, automatic metrics, human evaluation, with frequent benchmark focus on Caparena, Visualwebarena and metric focus on agreement, accuracy. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.

Why This Matters For Eval Research

40% of papers report explicit human-feedback signals, led by expert verification.

Evidence: EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis , DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries , Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
LLM-as-judge appears in 100% of papers in this hub.

Evidence: Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning , MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents , World-Model-Augmented Web Agents with Action Correction
Caparena is a recurring benchmark anchor for cross-paper comparisons in this page.

Evidence: PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions , Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning , MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Protocol Takeaways

2 sampled papers report both human evaluation and LLM-as-judge, supporting direct agreement checks.

Evidence: HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue , PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions , Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Most common quality-control signal is inter-annotator agreement reporting (20% of papers).

Evidence: MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents , HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue , Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.

Evidence: World-Model-Augmented Web Agents with Action Correction , PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions , EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis , DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

Benchmark Interpretation

Caparena appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.
Visualwebarena appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

agreement is reported in 30% of hub papers (3/10); compare with a secondary metric before ranking methods.
accuracy is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.

Researcher Checklist

Tighten coverage on Papers with explicit human feedback. Coverage is usable but incomplete (40% vs 45% target).
Tighten coverage on Papers reporting quality controls. Coverage is usable but incomplete (20% vs 30% target).
Close gap on Papers naming benchmarks/datasets. Coverage is a replication risk (20% vs 35% target).
Maintain strength on Papers naming evaluation metrics. Coverage is strong (80% vs 35% target).
Maintain strength on Papers with known rater population. Coverage is strong (40% vs 35% target).
Close gap on Papers with known annotation unit. Coverage is a replication risk (20% vs 35% target).

Papers with explicit human feedback

Coverage is usable but incomplete (40% vs 45% target).

Papers reporting quality controls

Coverage is usable but incomplete (20% vs 30% target).

Papers naming benchmarks/datasets

Coverage is a replication risk (20% vs 35% target).

Papers naming evaluation metrics

Coverage is strong (80% vs 35% target).

Papers with known rater population

Coverage is strong (40% vs 35% target).

Papers with known annotation unit

Coverage is a replication risk (20% vs 35% target).

Known Limitations

Annotation unit is under-specified (20% coverage).
Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Links

Judge vs Human Agreement - Compares papers that evaluate with both human raters and LLM judges.
Benchmark Slice: Caparena - Prioritizes benchmark-specific protocol comparisons.
Metric Slice: agreement - Finds papers where reported metrics are directly comparable.
IAA-Reported Evaluations - Highlights evaluations that explicitly report inter-annotator agreement.
Recent High-Signal Papers - Keeps the hub connected to the latest HFEPX corpus updates.

human_eval vs llm_as_judge

both=2, left_only=0, right_only=8

2 papers use both Human Eval and Llm As Judge.

human_eval vs automatic_metrics

both=0, left_only=2, right_only=2

0 papers use both Human Eval and Automatic Metrics.

llm_as_judge vs automatic_metrics

both=2, left_only=8, right_only=0

2 papers use both Llm As Judge and Automatic Metrics.

Benchmark Brief

Caparena

Coverage: 1 papers (10%)

1 papers (10%) mention Caparena.

Examples: PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions

Benchmark Brief

Visualwebarena

Coverage: 1 papers (10%)

1 papers (10%) mention Visualwebarena.

Examples: World-Model-Augmented Web Agents with Action Correction

Metric Brief

agreement

Coverage: 3 papers (30%)

3 papers (30%) mention agreement.

Examples: MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents , HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue , Human-like Affective Cognition in Foundation Models

Metric Brief

accuracy

Coverage: 1 papers (10%)

1 papers (10%) mention accuracy.

Examples: Overton Pluralistic Reinforcement Learning for Large Language Models

Metric Brief

auroc

Coverage: 1 papers (10%)

1 papers (10%) mention auroc.

Examples: DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

Most Cited In This Hub

Fast path to methods with the strongest citation traction in this scope.

Papers: Overton Pluralistic Reinforcement Learning for Large Language Models , Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning , MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents

Most Recent

Fast path to latest protocol changes and newly published evaluation setups.

Best Protocol Detail

Papers with explicit rater/unit metadata and quality-control signals for reproducibility.

Top Papers

Overton Pluralistic Reinforcement Learning for Large Language Models
Yu Fu, Seongho Son, Ilija Bogunovic · Feb 24, 2026 · Citations: 0

Llm As JudgeAutomatic Metrics

Existing alignment paradigms remain limited in capturing the pluralistic nature of human values.
Stop-Think-AutoRegress: Language Modeling with Latent Diffusion Planning
Justin Lovelace, Christian Belardi, Sofian Zalouk, Adhitya Polavaram, Srivatsa Kundurthy · Feb 24, 2026 · Citations: 0

Llm As JudgeAutomatic Metrics

Evaluations show STAR-LDM significantly outperforms similar-sized models on language understanding benchmarks and achieves $>70\%$ win rates in LLM-as-judge comparisons for narrative coherence and commonsense reasoning.
MERRY: Semantically Decoupled Evaluation of Multimodal Emotional and Role Consistencies of Role-Playing Agents
Zhenyu Wang, Xiaofen Xing, Yirong Chen, Xiangmin Xu · Feb 24, 2026 · Citations: 0

Llm As Judge

Multimodal Role-Playing Agents (MRPAs) are attracting increasing attention due to their ability to deliver more immersive multimodal emotional interactions.
World-Model-Augmented Web Agents with Action Correction
Zhouzhou Shen, Xueyu Hu, Xiyun Li, Tianqing Fang, Juncheng Li · Feb 17, 2026 · Citations: 0

Llm As JudgeSimulation Env Multi Agent

Web agents based on large language models have demonstrated promising capability in automating web tasks.
HEART: A Unified Benchmark for Assessing Humans and LLMs in Emotional Support Dialogue
Laya Iyer, Kriti Aggarwal, Sanmi Koyejo, Gail Heyman, Desmond C. Ong · Jan 9, 2026 · Citations: 0

Pairwise PreferenceRubric Rating Human EvalLlm As Judge

Despite rapid progress in language models, we still lack a clear way to understand how their abilities in these interpersonal domains compare to those of humans.
PoSh: Using Scene Graphs To Guide LLMs-as-a-Judge For Detailed Image Descriptions
Amith Ananthram, Elias Stengel-Eskin, Lorena A. Bradford, Julia Demarest, Adam Purvis · Oct 21, 2025 · Citations: 0

Rubric Rating Human EvalLlm As Judge

While vision-language models (VLMs) have advanced into detailed image description, evaluation remains a challenge.
EpidemIQs: Prompt-to-Paper LLM Agents for Epidemic Modeling and Analysis
Mohammad Hossein Samaei, Faryad Darabi Sahneh, Lee W. Cohnstaedt, Caterina Scoglio · Sep 24, 2025 · Citations: 0

Expert Verification Llm As JudgeSimulation Env Multi Agent

We introduce EpidemIQs, a novel multi-agent LLM framework that integrates user inputs and autonomously conducts literature review, analytical derivation, network modeling, mechanistic modeling, stochastic simulations, data visualization and
Self-adaptive Dataset Construction for Real-World Multimodal Safety Scenarios
Jingen Qu, Lijun Li, Bo Zhang, Yichen Yan, Jing Shao · Sep 4, 2025 · Citations: 0

Llm As Judge

Multimodal large language models (MLLMs) are rapidly evolving, presenting increasingly complex safety challenges.
DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries
Heloisa Oss Boll, Antonio Oss Boll, Leticia Puttlitz Boll, Ameen Abu Hanna, Iacer Calixto · Jun 20, 2025 · Citations: 0

Expert Verification Llm As Judge

This study introduces DistillNote, an evaluation framework for LLM summaries that targets their functional utility by applying the generated summary downstream in a complex clinical prediction task, explicitly quantifying how much predictio
Human-like Affective Cognition in Foundation Models
Kanishk Gandhi, Zoe Lynch, Jan-Philipp Fränken, Kayla Patterson, Sharon Wambu · Sep 18, 2024 · Citations: 0

Llm As Judge

Understanding emotions is fundamental to human interaction and experience.

Llm As Judge Papers

Research Narrative

Why This Matters For Eval Research

Protocol Takeaways

Benchmark Interpretation

Metric Interpretation

Researcher Checklist

Suggested Reading Order

Known Limitations

Research Utility Links

Top Papers

Related Hubs