Long Horizon + General (Last 30 Days)

Why This Matters For Eval Research

• 8.3% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 70.8% of papers in this hub.
• ALFWorld is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is rater calibration (4.2% of papers).
• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
• Stratify by benchmark (ALFWorld vs Ama-Bench) before comparing methods.

Benchmark Interpretation

• ALFWorld appears in 4.2% of hub papers (1/24); use this cohort for benchmark-matched comparisons.
• Ama-Bench appears in 4.2% of hub papers (1/24); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 41.7% of hub papers (10/24); compare with a secondary metric before ranking methods.
• cost is reported in 20.8% of hub papers (5/24); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (8.3% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (4.2% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (25% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (70.8% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (4.2% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (33.3% vs 35% target).

Strengths

• Agentic evaluation appears in 100% of papers.

Known Gaps

• Only 4.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (4.2% coverage).

Suggested Next Analyses

• Stratify by benchmark (ALFWorld vs Ama-Bench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: cost. Abstract: Presentation generation requires deep content research, coherent visual design, and iterative refinement based.

2. AMA-Bench: Evaluating Long-Horizon Memory for Agentic Applications

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: Ama-Bench / accuracy. Abstract: Large Language Models (LLMs) are deployed as autonomous agents in increasingly.

3. Replacing Multi-Step Assembly of Data Preparation Pipelines with One-Step LLM Pipeline Generation for Table QA

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Table Question Answering (TQA) aims to answer natural language questions over structured tables.

4. SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments. Focus: ALFWorld. Abstract: Large language models (LLMs) are increasingly deployed as multi-step decision-making agents, where.

5. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments. Focus: WebArena. Abstract: The paper introduces GUI-Owl-1.5, the latest native GUI agent model that features.

6. PMG: Parameterized Motion Generator for Human-like Locomotion Control

   Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: calibration. Abstract: Recent advances in data-driven reinforcement learning and motion tracking have substantially improved.

7. ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: Arlarena. Abstract: Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm.

8. Learning to Reason for Multi-Step Retrieval of Personal Context in Personalized Question Answering

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: Personalization in Question Answering (QA) requires answers that are both.

Known Limitations

• Only 4.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (4.2% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot