Simulation Env Papers (Last 90 Days)

Why This Matters For Eval Research

• 17.2% of papers report explicit human-feedback signals, led by expert verification.
• simulation environments appears in 61.7% of papers in this hub.
• ALFWorld is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is adjudication (2.1% of papers).
• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

• ALFWorld appears in 10.3% of hub papers (3/47); use this cohort for benchmark-matched comparisons.
• WebShop appears in 6.9% of hub papers (2/47); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 17.2% of hub papers (5/47); compare with a secondary metric before ranking methods.
• coherence is reported in 10.3% of hub papers (3/47); compare with a secondary metric before ranking methods.

Researcher Checklist

• Gap: Papers with explicit human feedback

  Coverage is a replication risk (17.2% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (3.4% vs 30% target).

• Strong: Papers naming benchmarks/datasets

  Coverage is strong (37.9% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (44.8% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (20.7% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (34.5% vs 35% target).

Strengths

• Most papers provide measurable evaluation context (37.9% benchmarks, 44.8% metrics).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 79.3% of papers.

Known Gaps

• Only 3.4% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20.7% coverage).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (ALFWorld vs WebShop) before comparing methods.
• Track metric sensitivity by reporting both accuracy and coherence.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. The Trinity of Consistency as a Defining Principle for General World Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: Cow-Bench. Abstract: CoW-Bench evaluates both video generation models and UMMs under a unified evaluation protocol.

2. Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: ALFWorld. Abstract: Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large.

3. TherapyProbe: Generating Design Knowledge for Relational Safety in Mental Health Chatbots Through Adversarial Simulation

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments + expert verification. Abstract: As mental health chatbots proliferate to address the global treatment gap, a.

4. FrameRef: A Framing Dataset and Simulation Testbed for Modeling Bounded Rational Information Health

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation. Abstract: Human evaluation further confirms that FrameRef's generated framings measurably affect human judgment.

5. AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + expert verification. Focus: Ad-Bench / pass@1. Abstract: While Large Language Model (LLM) agents have.

6. World-Model-Augmented Web Agents with Action Correction

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: VisualWebArena. Abstract: A world model, specialized in environmental state transitions, simulates action outcomes, which.

7. Fast-ThinkAct: Efficient Vision-Language-Action Reasoning via Verbalizable Latent Planning

   Adds simulation environments with pairwise preferences for broader protocol coverage within this hub. Signals: simulation environments + pairwise preferences. Focus: latency. Abstract: Fast-ThinkAct learns to reason efficiently with.

8. Embodied Task Planning via Graph-Informed Action Generation with Large Language Model

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: ALFWorld / pass@1. Abstract: While Large Language Models (LLMs) have demonstrated strong zero-shot reasoning.

Known Limitations

• Only 3.4% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (20.7% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot