Long Horizon Papers (Last 30 Days)

Why This Matters For Eval Research

• 14.6% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 47.8% of papers in this hub.
• SWE-bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is adjudication (1.5% of papers).
• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
• Pair this hub with llm_as_judge pages to benchmark automated-vs-human evaluation tradeoffs.

Benchmark Interpretation

• SWE-bench appears in 6.3% of hub papers (3/67); use this cohort for benchmark-matched comparisons.
• SWE-bench Verified appears in 6.3% of hub papers (3/67); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 37.5% of hub papers (18/67); compare with a secondary metric before ranking methods.
• cost is reported in 20.8% of hub papers (10/67); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (14.6% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (4.2% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (31.3% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (70.8% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (8.3% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (43.8% vs 35% target).

Strengths

• Agentic evaluation appears in 100% of papers.

Known Gaps

• Only 4.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (8.3% coverage).

Suggested Next Analyses

• Pair this hub with llm_as_judge pages to benchmark automated-vs-human evaluation tradeoffs.
• Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. The Trinity of Consistency as a Defining Principle for General World Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: Cow-Bench. Abstract: CoW-Bench evaluates both video generation models and UMMs under a unified evaluation protocol.

2. Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Reasoning with large language models often benefits from generating multiple chains-of-thought, but existing.

3. DeepPresenter: Environment-Grounded Reflection for Agentic Presentation Generation

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: cost. Abstract: Presentation generation requires deep content research, coherent visual design, and iterative refinement based.

4. FrameRef: A Framing Dataset and Simulation Testbed for Modeling Bounded Rational Information Health

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation. Abstract: Human evaluation further confirms that FrameRef's generated framings measurably affect human judgment.

5. AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + expert verification. Focus: Ad-Bench / pass@1. Abstract: While Large Language Model (LLM) agents have.

6. KLong: Training LLM Agent for Extremely Long-horizon Tasks

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: rubric ratings. Focus: SWE-bench. Abstract: Then, we introduce Research-Factory, an automated pipeline that generates high-quality training.

7. Tutoring Large Language Models to be Domain-adaptive, Precise, and Safe

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Focus: precision. Abstract: The methodological trajectory moves from classical supervised adaptation.

8. MedXIAOHE: A Comprehensive Recipe for Building Medical MLLMs

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: To improve reliability in real-world use, MedXIAOHE integrates user-preference rubrics,.

Known Limitations

• Only 4.2% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (8.3% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot