Automatic Metrics + General (Last 45 Days)

Why This Matters For Eval Research

• 39% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 100% of papers in this hub.
• BrowseComp is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is inter-annotator agreement reporting (5.1% of papers).
• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

• BrowseComp appears in 3.4% of hub papers (2/59); use this cohort for benchmark-matched comparisons.
• MMLU appears in 3.4% of hub papers (2/59); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 45.8% of hub papers (27/59); compare with a secondary metric before ranking methods.
• cost is reported in 18.6% of hub papers (11/59); compare with a secondary metric before ranking methods.

Researcher Checklist

• Moderate: Papers with explicit human feedback

  Coverage is usable but incomplete (39% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (8.5% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (11.9% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (78% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (11.9% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (22% vs 35% target).

Strengths

• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 52.5% of papers.

Known Gaps

• Only 8.5% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (11.9% coverage).
• Annotation unit is under-specified (22% coverage).

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (BrowseComp vs MMLU) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Spatio-Temporal Token Pruning for Efficient High-Resolution GUI Agents

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: precision. Abstract: Pure-vision GUI agents provide universal interaction capabilities but suffer from severe efficiency bottlenecks.

2. DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: cost. Abstract: Presentation generation requires deep content research, coherent visual design, and iterative refinement based.

3. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: Ama-Bench / accuracy. Abstract: Large Language Models (LLMs) are deployed as autonomous agents in increasingly.

4. SCOPE: Selective Conformal Optimized Pairwise LLM Judging

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: MT-Bench / error rate. Abstract: Large language models (LLMs) are increasingly.

5. MemoryArena: Benchmarking Agent Memory in Interdependent Multi-Session Agentic Tasks

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + pairwise preferences. Focus: Memoryarena / recall. Abstract: MemoryArena supports evaluation across web navigation, preference-constrained.

6. The Emergence of Lab-Driven Alignment Signatures: A Psychometric Framework for Auditing Latent Bias and Compounding Risk in Generative AI

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: accuracy. Abstract: As Large Language Models (LLMs) transition from standalone chat interfaces to foundational.

7. Personalized Prediction of Perceived Message Effectiveness Using Large Language Model Based Digital Twins

   Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Focus: accuracy. Abstract: Perceived message effectiveness (PME) by potential.

8. Yor-Sarc: A gold-standard dataset for sarcasm detection in a low-resource African language

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: agreement. Abstract: The dataset comprises 436 instances annotated.

Known Limitations

• Only 8.5% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (11.9% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot