Simulation Env + General (Last 60 Days)

Why This Matters For Eval Research

• 7.1% of papers report explicit human-feedback signals, led by pairwise preferences.
• simulation environments appears in 100% of papers in this hub.
• ALFWorld is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Benchmark Interpretation

• ALFWorld appears in 14.3% of hub papers (2/14); use this cohort for benchmark-matched comparisons.
• Arlarena appears in 7.1% of hub papers (1/14); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 14.3% of hub papers (2/14); compare with a secondary metric before ranking methods.
• coherence is reported in 14.3% of hub papers (2/14); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (7.1% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Strong: Papers naming benchmarks/datasets

  Coverage is strong (50% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (42.9% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (14.3% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (28.6% vs 35% target).

Strengths

• Most papers provide measurable evaluation context (50% benchmarks, 42.9% metrics).
• Agentic evaluation appears in 92.9% of papers.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (14.3% coverage).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Stratify by benchmark (ALFWorld vs Arlarena) before comparing methods.
• Track metric sensitivity by reporting both accuracy and coherence.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. ARLArena: A Unified Framework for Stable Agentic Reinforcement Learning

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: Arlarena. Abstract: Agentic reinforcement learning (ARL) has rapidly gained attention as a promising paradigm for.

2. LiLo-VLA: Compositional Long-Horizon Manipulation via Linked Object-Centric Policies

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: success rate. Abstract: General-purpose robots must master long-horizon manipulation, defined as tasks involving multiple kinematic.

3. Efficient Hierarchical Any-Angle Path Planning on Multi-Resolution 3D Grids

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Abstract: Hierarchical, multi-resolution volumetric mapping approaches are widely used to represent large and complex environments as.

4. Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + pairwise preferences. Focus: latency. Abstract: Fast-ThinkAct learns to reason efficiently with latent CoTs by.

5. Embodied Task Planning via Graph-Informed Action Generation with Large Language Model

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments. Focus: ALFWorld / pass@1. Abstract: While Large Language Models (LLMs) have demonstrated strong zero-shot reasoning.

6. World-Model-Augmented Web Agents with Action Correction

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which.

7. SELAUR: Self Evolving LLM Agent via Uncertainty-aware Rewards

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: ALFWorld. Abstract: Large language models (LLMs) are increasingly deployed as multi-step decision-making agents, where.

8. Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: WebArena. Abstract: The paper introduces GUI-Owl-1.5, the latest native GUI agent model that features.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (14.3% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot