Law Papers (Last 90 Days)

Why This Matters For Eval Research

• 40% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 60% of papers in this hub.
• Cow-Bench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is adjudication (10% of papers).
• Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
• Pair this hub with llm_as_judge pages to benchmark automated-vs-human evaluation tradeoffs.

Benchmark Interpretation

• Cow-Bench appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.
• HLE appears in 10% of hub papers (1/10); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 20% of hub papers (2/10); compare with a secondary metric before ranking methods.
• error rate is reported in 10% of hub papers (1/10); compare with a secondary metric before ranking methods.

Researcher Checklist

• Moderate: Papers with explicit human feedback

  Coverage is usable but incomplete (40% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (10% vs 30% target).

• Strong: Papers naming benchmarks/datasets

  Coverage is strong (40% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (50% vs 35% target).

• Moderate: Papers with known rater population

  Coverage is usable but incomplete (30% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (30% vs 35% target).

Strengths

• Most papers provide measurable evaluation context (40% benchmarks, 50% metrics).
• Agentic evaluation appears in 60% of papers.

Known Gaps

• Only 10% of papers report quality controls; prioritize calibration/adjudication evidence.

Suggested Next Analyses

• Pair this hub with llm_as_judge pages to benchmark automated-vs-human evaluation tradeoffs.
• Stratify by benchmark (Cow-Bench vs HLE) before comparing methods.
• Track metric sensitivity by reporting both accuracy and error rate.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. The Trinity of Consistency as a Defining Principle for General World Models

   Start here for detailed protocol reporting and quality-control evidence. Signals: simulation environments. Focus: Cow-Bench. Abstract: CoW-Bench evaluates both video generation models and UMMs under a unified evaluation protocol.

2. Whisper: Courtside Edition Enhancing ASR Performance Through LLM-Driven Context Generation

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics. Focus: error rate. Abstract: Domain-specific speech remains a persistent challenge for automatic speech recognition (ASR), even.

3. Vichara: Appellate Judgment Prediction and Explanation for the Indian Judicial System

   Start here for detailed protocol reporting and quality-control evidence. Signals: human evaluation. Focus: f1. Abstract: Human evaluation of the generated explanations across Clarity, Linking, and Usefulness metrics highlights.

4. HLE-Verified: A Systematic Verification and Structured Revision of Humanity's Last Exam

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: automatic metrics + expert verification. Focus: HLE / accuracy. Abstract: Overall, HLE-Verified improves HLE-style evaluations by reducing.

5. APEX-Agents

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: automatic metrics + rubric ratings. Focus: pass@1. Abstract: We open source the APEX-Agents benchmark (n=480) with.

6. Multimodal Multi-Agent Empowered Legal Judgment Prediction

   Adds simulation environments for broader protocol coverage within this hub. Signals: simulation environments. Focus: Lawbench. Abstract: Legal Judgment Prediction (LJP) aims to predict the outcomes of legal cases.

7. The Subjectivity of Respect in Police Traffic Stops: Modeling Community Perspectives in Body-Worn Camera Footage

   Adds evaluation protocol evidence with pairwise preferences for broader protocol coverage within this hub. Signals: pairwise preferences. Abstract: To this end, we (i) develop a domain-specific evaluation rubric.

8. Orthogonalized Policy Optimization:Policy Optimization as Orthogonal Projection in Hilbert Space

   Adds automatic metrics for broader protocol coverage within this hub. Signals: automatic metrics. Focus: MATH. Abstract: We propose Orthogonalized Policy Optimization (OPO), a principled framework for large language.

Known Limitations

• Only 10% of papers report quality controls; prioritize calibration/adjudication evidence.
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
• Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.

Research Utility Snapshot