General + Demonstrations (Last 120 Days)

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (0% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (18.2% vs 35% target).

• Gap: Papers naming evaluation metrics

  Coverage is a replication risk (9.1% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (18.2% vs 35% target).

• Gap: Papers with known annotation unit

  Coverage is a replication risk (18.2% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).

Known Gaps

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (18.2% coverage).
• Annotation unit is under-specified (18.2% coverage).

Suggested Next Analyses

• Stratify by benchmark (Auditbench vs Fewmmbench) before comparing methods.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. TimeWarp: Evaluating Web Agents by Revisiting the Past

   Start here for detailed protocol reporting and quality-control evidence. Signals: demonstration data. Abstract: The improvement of web agents on current benchmarks raises the question: Do today's agents perform.

2. Optimizing In-Context Demonstrations for LLM-based Automated Grading

   Start here for detailed protocol reporting and quality-control evidence. Signals: rubric ratings. Abstract: Standard retrieval methods typically select examples based on semantic similarity, which often fails to capture.

3. ArgLLM-App: An Interactive System for Argumentative Reasoning with Large Language Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: demonstration data. Abstract: Argumentative LLMs (ArgLLMs) are an existing approach leveraging Large Language Models (LLMs) and computational argumentation.

4. Orchestration-Free Customer Service Automation: A Privacy-Preserving and Flowchart-Guided Framework

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + demonstration data. Focus: cost. Abstract: Customer service automation has seen growing demand within digital.

5. AuditBench: Evaluating Alignment Auditing Techniques on Models with Hidden Behaviors

   Adds evaluation protocol evidence with demonstration data for broader protocol coverage within this hub. Signals: demonstration data. Focus: Auditbench. Abstract: We introduce AuditBench, an alignment auditing benchmark.

6. FewMMBench: A Benchmark for Multimodal Few-Shot Learning

   Adds evaluation protocol evidence with demonstration data for broader protocol coverage within this hub. Signals: demonstration data. Focus: Fewmmbench. Abstract: As multimodal large language models (MLLMs) advance in.

7. Risk-Aware World Model Predictive Control for Generalizable End-to-End Autonomous Driving

   Adds evaluation protocol evidence with demonstration data for broader protocol coverage within this hub. Signals: demonstration data. Abstract: With advances in imitation learning (IL) and large-scale driving datasets,.

8. Fine-Tuning Without Forgetting In-Context Learning: A Theoretical Analysis of Linear Attention Models

   Adds evaluation protocol evidence with demonstration data for broader protocol coverage within this hub. Signals: demonstration data. Abstract: Transformer-based large language models exhibit in-context learning, enabling adaptation to.

Known Limitations

• Only 0% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (18.2% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot