CS.LG + Coding Papers

Why This Matters For Eval Research

• 50% of papers report explicit human-feedback signals, led by pairwise preferences.
• automatic metrics appears in 40.9% of papers in this hub.
• Ad-Bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.
• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.
• Pair this hub with llm_as_judge pages to benchmark automated-vs-human evaluation tradeoffs.

Benchmark Interpretation

• Ad-Bench appears in 4.5% of hub papers (1/22); use this cohort for benchmark-matched comparisons.
• ALFWorld appears in 4.5% of hub papers (1/22); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 18.2% of hub papers (4/22); compare with a secondary metric before ranking methods.
• latency is reported in 13.6% of hub papers (3/22); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (50% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (22.7% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (45.5% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (22.7% vs 35% target).

• Strong: Papers with known annotation unit

  Coverage is strong (40.9% vs 35% target).

Strengths

• Strong human-feedback signal (50% of papers).
• Agentic evaluation appears in 59.1% of papers.

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (22.7% coverage).

Suggested Next Analyses

• Pair this hub with llm_as_judge pages to benchmark automated-vs-human evaluation tradeoffs.
• Stratify by benchmark (Ad-Bench vs ALFWorld) before comparing methods.
• Track metric sensitivity by reporting both accuracy and latency.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: ALFWorld. Abstract: Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large.

2. GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Open-source native GUI agents still lag behind closed-source systems on long-horizon navigation tasks.

3. SWE-Protégé: Learning to Selectively Collaborate With an Expert Unlocks Small Language Models as Software Engineering Agents

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: SWE-bench / pass@1. Abstract: Small language models (SLMs) offer compelling advantages in cost, latency, and.

4. Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Abstract: Large Language Models (LLMs) are widely used as judges to evaluate.

5. AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + expert verification. Focus: Ad-Bench / pass@1. Abstract: While Large Language Model (LLM) agents have.

6. Supervised Reinforcement Learning: From Expert Trajectories to Step-wise Reasoning

   Adds evaluation protocol evidence with demonstration data for broader protocol coverage within this hub. Signals: demonstration data. Abstract: Large Language Models (LLMs) often struggle with problems that require.

7. The Vision Wormhole: Latent-Space Communication in Heterogeneous Multi-Agent Systems

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: Our framework adopts a hub-and-spoke topology to reduce pairwise alignment.

8. CORE: Measuring Multi-Agent LLM Interaction Quality under Game-Theoretic Pressures

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: We apply CORE to pairwise LLM dialogs across competitive, cooperative,.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (22.7% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot