Skip to content
← Back to explorer

Metric Hub

Cost + Long Horizon Metric Papers

Updated from current HFEPX corpus (Feb 27, 2026). 12 papers are grouped in this metric page. Common evaluation modes: Automatic Metrics, Simulation Env. Most common rater population: Domain Experts. Common annotation unit: Trajectory. Frequently cited benchmark: ALFWorld. Common metric signal: cost. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Feb 26, 2026.

Papers: 12 Last published: Feb 26, 2026 Global RSS

Research Narrative

Grounded narrative Model: deterministic-grounded Source: persisted

Updated from current HFEPX corpus (Feb 27, 2026). This page tracks 12 papers for Cost + Long Horizon Metric Papers. Dominant protocol signals include automatic metrics, simulation environments, with frequent benchmark focus on ALFWorld, BrowseComp and metric focus on cost, accuracy. Use the grounded sections below to prioritize reproducible protocol choices, benchmark-matched comparisons, and judge-vs-human evaluation checks.

Why This Matters For Eval Research

Protocol Takeaways

Benchmark Interpretation

  • ALFWorld appears in 8.3% of hub papers (1/12); use this cohort for benchmark-matched comparisons.
  • BrowseComp appears in 8.3% of hub papers (1/12); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • cost is reported in 100% of hub papers (12/12); compare with a secondary metric before ranking methods.
  • accuracy is reported in 41.7% of hub papers (5/12); compare with a secondary metric before ranking methods.

Abstract Evidence Highlights

Direct snippets from paper abstracts to ground protocol and benchmark interpretation.

Human-eval abstract signal: Reasoning with large language models often benefits from generating multiple chains-of-thought, but existing aggregation strategies are typically trajectory-level (e.g., selecting the best trace or voting on the final answer), discarding useful intermediate work from...

Human-eval abstract signal: Table Question Answering (TQA) aims to answer natural language questions over structured tables.

ALFWorld benchmark signal: We evaluate our framework on three embodied planning benchmarks-Robotouille Synchronous, Robotouille Asynchronous, and ALFWorld.

ALFWorld benchmark signal: Across math reasoning benchmarks, we find that step-level recombination is most beneficial on harder problems, and ablations highlight the importance of the final AR solver in converting stitched but imperfect rationales into accurate answers.

cost metric signal: Using low-confidence diffusion sampling with parallel, independent rollouts, our training-free framework improves average accuracy by up to 23.8% across six math and coding tasks.

cost metric signal: Experiments on two benchmark datasets show that, with the same LLM backbone, Operation-R1 achieves average absolute accuracy gains of 9.55 and 6.08 percentage points over multi-step preparation baselines, with 79\% table compression and a...

Protocol abstract signal: Recent deep research agents primarily improve performance by scaling reasoning depth, but this leads to high inference cost and latency in search-intensive scenarios.

Protocol abstract signal: Latent reasoning has been recently proposed as a reasoning paradigm and performs multi-step reasoning through generating steps in the latent space instead of the textual space.

Researcher Checklist

  • Close gap on Papers with explicit human feedback. Coverage is a replication risk (0% vs 45% target).
  • Close gap on Papers reporting quality controls. Coverage is a replication risk (0% vs 30% target).
  • Tighten coverage on Papers naming benchmarks/datasets. Coverage is usable but incomplete (25% vs 35% target).
  • Maintain strength on Papers naming evaluation metrics. Coverage is strong (100% vs 35% target).
  • Close gap on Papers with known rater population. Coverage is a replication risk (8.3% vs 35% target).
  • Maintain strength on Papers with known annotation unit. Coverage is strong (50% vs 35% target).

Papers with explicit human feedback

Coverage is a replication risk (0% vs 45% target).

Papers reporting quality controls

Coverage is a replication risk (0% vs 30% target).

Papers naming benchmarks/datasets

Coverage is usable but incomplete (25% vs 35% target).

Papers naming evaluation metrics

Coverage is strong (100% vs 35% target).

Papers with known rater population

Coverage is a replication risk (8.3% vs 35% target).

Papers with known annotation unit

Coverage is strong (50% vs 35% target).

Suggested Reading Order

  1. 1. Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching

    Start here for detailed protocol reporting, including rater and quality-control evidence.

  2. 2. Replacing Multi-Step Assembly of Data Preparation Pipelines with One-Step LLM Pipeline Generation for Table QA

    Start here for detailed protocol reporting, including rater and quality-control evidence.

  3. 3. Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization

    Start here for detailed protocol reporting, including rater and quality-control evidence.

  4. 4. How Do Latent Reasoning Methods Perform Under Weak and Strong Supervision?

    Adds automatic metrics for broader coverage within this hub.

  5. 5. SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

    Adds automatic metrics for broader coverage within this hub.

  6. 6. Continuous Telemonitoring of Heart Failure using Personalised Speech Dynamics

    Adds automatic metrics for broader coverage within this hub.

  7. 7. Do LLMs and VLMs Share Neurons for Inference? Evidence and Mechanisms of Cross-Modal Transfer

    Adds automatic metrics for broader coverage within this hub.

  8. 8. Watermarking LLM Agent Trajectories

    Adds automatic metrics for broader coverage within this hub.

Known Limitations

  • Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
  • Rater population is under-specified (8.3% coverage).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Links

automatic_metrics vs simulation_env

both=0, left_only=10, right_only=2

0 papers use both Automatic Metrics and Simulation Env.

Benchmark Brief

ALFWorld

Coverage: 1 papers (8.3%)

1 papers (8.3%) mention ALFWorld.

Examples: Embodied Task Planning via Graph-Informed Action Generation with Large Language Model

Benchmark Brief

BrowseComp

Coverage: 1 papers (8.3%)

1 papers (8.3%) mention BrowseComp.

Examples: Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization

Benchmark Brief

GAIA

Coverage: 1 papers (8.3%)

1 papers (8.3%) mention GAIA.

Examples: Search More, Think Less: Rethinking Long-Horizon Agentic Search for Efficiency and Generalization

Top Papers Reporting This Metric

Other Metric Hubs