CS.AI + Coding Papers

Why This Matters For Eval Research

• 53.8% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 53.8% of papers in this hub.
• SWE-bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is rater calibration (1.9% of papers).
• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.
• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

• SWE-bench appears in 5.8% of hub papers (3/52); use this cohort for benchmark-matched comparisons.
• SWE-bench Verified appears in 5.8% of hub papers (3/52); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 23.1% of hub papers (12/52); compare with a secondary metric before ranking methods.
• cost is reported in 15.4% of hub papers (8/52); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (53.8% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (3.8% vs 30% target).

• Moderate: Papers naming benchmarks/datasets

  Coverage is usable but incomplete (26.9% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (59.6% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (26.9% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (26.9% vs 35% target).

Strengths

• Strong human-feedback signal (53.8% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 59.6% of papers.

Known Gaps

• Only 3.8% of papers report quality controls; prioritize calibration/adjudication evidence.
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (SWE-bench vs SWE-bench Verified) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. AgentDropoutV2: Optimizing Information Flow in Multi-Agent Systems via Test-Time Rectify-or-Reject Pruning

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: While Multi-Agent Systems (MAS) excel in complex reasoning, they suffer from the cascading.

2. Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: accuracy. Abstract: Reasoning with large language models often benefits from generating multiple chains-of-thought, but existing.

3. Hierarchy-of-Groups Policy Optimization for Long-Horizon Agentic Tasks

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: ALFWorld. Abstract: Group-based reinforcement learning (RL), such as GRPO, has advanced the capabilities of large.

4. Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learning

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Abstract: Large Language Models (LLMs) are widely used as judges to evaluate.

5. AD-Bench: A Real-World, Trajectory-Aware Advertising Analytics Benchmark for LLM Agents

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + expert verification. Focus: Ad-Bench / pass@1. Abstract: While Large Language Model (LLM) agents have.

6. LiveMCPBench: Can Agents Navigate an Ocean of MCP Tools?

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: Livemcpbench / task success. Abstract: We further introduce an LLM-as-a-Judge evaluation framework that directly.

7. CricBench: A Multilingual Benchmark for Evaluating LLMs in Cricket Analytics

   Adds automatic metrics with expert verification for broader protocol coverage within this hub. Signals: automatic metrics + expert verification. Focus: DROP / accuracy. Abstract: We evaluate six state-of-the-art.

8. Team of Thoughts: Efficient Test-time Scaling of Agentic Systems through Orchestrated Tool Calling

   Adds evaluation protocol evidence with expert verification for broader protocol coverage within this hub. Signals: expert verification. Focus: LiveCodeBench. Abstract: Existing Multi-Agent Systems (MAS) typically rely on static,.

Known Limitations

• Only 3.8% of papers report quality controls; prioritize calibration/adjudication evidence.
• LLM-as-judge appears without enough inter-annotator agreement reporting.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot