Automatic Metrics + Medicine (Last 45 Days)

Why This Matters For Eval Research

• 58.3% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 100% of papers in this hub.
• long-horizon tasks appears in 16.7% of papers, indicating agentic evaluation demand.

Protocol Takeaways

• Most common quality-control signal is adjudication (8.3% of papers).
• Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Metric Interpretation

• accuracy is reported in 50% of hub papers (6/12); compare with a secondary metric before ranking methods.
• agreement is reported in 16.7% of hub papers (2/12); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (58.3% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (16.7% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (0% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (100% vs 35% target).

• Strong: Papers with known rater population

  Coverage is strong (58.3% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (25% vs 35% target).

Strengths

• Strong human-feedback signal (58.3% of papers).
• Agentic evaluation appears in 33.3% of papers.

Known Gaps

• Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (0% of papers mention benchmarks/datasets).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
• Track metric sensitivity by reporting both accuracy and agreement.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + expert verification. Focus: f1. Abstract: Phenotyping is fundamental to rare disease diagnosis, but manual curation.

2. Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

   High citation traction makes this a strong baseline for protocol comparison. Signals: automatic metrics + expert verification. Focus: agreement. Abstract: Human-in-the-loop validation is essential in safety-critical clinical AI,.

3. MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models

   High citation traction makes this a strong baseline for protocol comparison. Signals: automatic metrics + expert verification. Focus: accuracy. Abstract: Multimodal large language models (MLLMs) have shown great.

4. SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video

   High citation traction makes this a strong baseline for protocol comparison. Signals: automatic metrics + expert verification. Focus: accuracy. Abstract: Minimally invasive surgery has dramatically improved patient operative.

5. Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: bleu. Abstract: Recently, Large Language Models (LLMs) have gained significant traction in medical domain,.

6. Multi-Objective Alignment of Language Models for Personalized Psychotherapy

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: agreement. Abstract: While AI systems show therapeutic promise,.

7. CUICurate: A GraphRAG-based Framework for Automated Clinical Concept Curation for NLP applications

   Adds automatic metrics with expert verification for broader protocol coverage within this hub. Signals: automatic metrics + expert verification. Focus: precision. Abstract: Comparisons between the two LLMs found.

8. What Makes a Good Doctor Response? An Analysis on a Romanian Telemedicine Platform

   Adds automatic metrics with expert verification for broader protocol coverage within this hub. Signals: automatic metrics + expert verification. Focus: accuracy. Abstract: Text-based telemedicine has become a common.

Known Limitations

• Only 16.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (0% of papers mention benchmarks/datasets).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot