Critique Edit Papers (Last 30 Days)

Why This Matters For Eval Research

• 100% of papers report explicit human-feedback signals, led by critique/edit feedback.
• automatic metrics appears in 45.5% of papers in this hub.
• ContentBench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is adjudication (9.1% of papers).
• Rater context is mostly domain experts, and annotation is commonly scalar scoring; use this to scope replication staffing.
• Stratify by benchmark (ContentBench vs HLE) before comparing methods.

Benchmark Interpretation

• ContentBench appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.
• HLE appears in 9.1% of hub papers (1/11); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 18.2% of hub papers (2/11); compare with a secondary metric before ranking methods.
• cost is reported in 18.2% of hub papers (2/11); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (100% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (9.1% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (18.2% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (36.4% vs 35% target).

• Gap: Papers with known rater population

  Coverage is a replication risk (9.1% vs 35% target).

• Gap: Papers with known annotation unit

  Coverage is a replication risk (18.2% vs 35% target).

Strengths

• Strong human-feedback signal (100% of papers).

Known Gaps

• Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (9.1% coverage).
• Annotation unit is under-specified (18.2% coverage).

Suggested Next Analyses

• Stratify by benchmark (ContentBench vs HLE) before comparing methods.
• Track metric sensitivity by reporting both accuracy and cost.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. The logic of KM belief update is contained in the logic of AGM belief revision

   Start here for detailed protocol reporting and quality-control evidence. Signals: critique/edit feedback. Abstract: For each axiom of KM belief update we provide a corresponding axiom in a modal.

2. Natural Language Declarative Prompting (NLD-P): A Modular Governance Method for Prompt Design Under Model Drift

   Start here for detailed protocol reporting and quality-control evidence. Signals: critique/edit feedback. Abstract: The rapid evolution of large language models (LLMs) has transformed prompt engineering from a localized.

3. Towards Better RL Training Data Utilization via Second-Order Rollout

   Start here for detailed protocol reporting and quality-control evidence. Signals: critique/edit feedback. Abstract: Reinforcement Learning (RL) has empowered Large Language Models (LLMs) with strong reasoning capabilities, but vanilla.

4. HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + expert verification. Focus: HLE / accuracy. Abstract: Overall, HLE-Verified improves HLE-style evaluations by reducing.

5. From Pixels to Policies: Reinforcing Spatial Reasoning in Language Models for Content-Aware Layout Design

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: simulation environments + critique/edit feedback. Focus: latency. Abstract: We introduce LaySPA, a reinforcement learning framework that equips.

6. Can Large Language Models Replace Human Coders? Introducing ContentBench

   Adds automatic metrics with critique/edit feedback for broader protocol coverage within this hub. Signals: automatic metrics + critique/edit feedback. Focus: ContentBench / agreement. Abstract: Among the 59 evaluated.

7. CAMEL: Confidence-Gated Reflection for Reward Modeling

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: accuracy. Abstract: Reward models play a fundamental role.

8. Unlocking Reasoning Capability on Machine Translation in Large Language Models

   Adds evaluation protocol evidence with critique/edit feedback for broader protocol coverage within this hub. Signals: critique/edit feedback. Abstract: Reasoning-oriented large language models (RLMs) achieve strong gains on tasks.

Known Limitations

• Only 9.1% of papers report quality controls; prioritize calibration/adjudication evidence.
• Rater population is under-specified (9.1% coverage).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot