Medicine Papers (Last 120 Days)

Why This Matters For Eval Research

• 69.6% of papers report explicit human-feedback signals, led by expert verification.
• automatic metrics appears in 60.9% of papers in this hub.
• AIME is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

• Most common quality-control signal is adjudication (4.3% of papers).
• Rater context is mostly domain experts, and annotation is commonly multi-dimensional rubrics; use this to scope replication staffing.
• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.

Benchmark Interpretation

• AIME appears in 4.3% of hub papers (1/23); use this cohort for benchmark-matched comparisons.
• Correctbench appears in 4.3% of hub papers (1/23); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 26.1% of hub papers (6/23); compare with a secondary metric before ranking methods.
• agreement is reported in 13% of hub papers (3/23); compare with a secondary metric before ranking methods.

Researcher Checklist

• Strong: Papers with explicit human feedback

  Coverage is strong (69.6% vs 45% target).

• Gap: Papers reporting quality controls

  Coverage is a replication risk (8.7% vs 30% target).

• Gap: Papers naming benchmarks/datasets

  Coverage is a replication risk (8.7% vs 35% target).

• Strong: Papers naming evaluation metrics

  Coverage is strong (65.2% vs 35% target).

• Strong: Papers with known rater population

  Coverage is strong (60.9% vs 35% target).

• Moderate: Papers with known annotation unit

  Coverage is usable but incomplete (26.1% vs 35% target).

Strengths

• Strong human-feedback signal (69.6% of papers).
• Contains both human-eval and LLM-as-judge protocols for head-to-head methodology comparison.
• Agentic evaluation appears in 34.8% of papers.

Known Gaps

• Only 8.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (8.7% of papers mention benchmarks/datasets).
• LLM-as-judge appears without enough inter-annotator agreement reporting.

Suggested Next Analyses

• Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
• Stratify by benchmark (AIME vs Correctbench) before comparing methods.
• Track metric sensitivity by reporting both accuracy and agreement.
• Add inter-annotator agreement checks when reproducing these protocols.

Recommended Queries

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

1. An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

   Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + expert verification. Focus: f1. Abstract: Phenotyping is fundamental to rare disease diagnosis, but manual curation.

2. Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

   High citation traction makes this a strong baseline for protocol comparison. Signals: automatic metrics + expert verification. Focus: agreement. Abstract: Human-in-the-loop validation is essential in safety-critical clinical AI,.

3. pMoE: Prompting Diverse Experts Together Wins More in Visual Adaptation

   High citation traction makes this a strong baseline for protocol comparison. Signals: expert verification. Abstract: Parameter-efficient fine-tuning has demonstrated promising results across various visual adaptation tasks, such as.

4. TherapyProbe: Generating Design Knowledge for Relational Safety in Mental Health Chatbots Through Adversarial Simulation

   High citation traction makes this a strong baseline for protocol comparison. Signals: simulation environments + expert verification. Abstract: As mental health chatbots proliferate to address the global treatment.

5. Think$^{2}$: Grounded Metacognitive Reasoning in Large Language Models

   Include a human-eval paper to calibrate against judge-based evaluation settings. Signals: human evaluation + pairwise preferences. Focus: GSM8K. Abstract: Blinded human evaluations over 580 query pairs show an.

6. Assessing Large Language Models for Medical QA: Zero-Shot and LLM-as-a-Judge Evaluation

   Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge. Focus: bleu. Abstract: Recently, Large Language Models (LLMs) have gained significant traction in medical domain,.

7. Assessing Risks of Large Language Models in Mental Health Support: A Framework for Automated Clinical AI Red Teaming

   Adds simulation environments with red-team protocols for broader protocol coverage within this hub. Signals: simulation environments + red-team protocols. Abstract: Large Language Models (LLMs) are increasingly utilized for.

8. Multi-Objective Alignment of Language Models for Personalized Psychotherapy

   Adds automatic metrics with pairwise preferences for broader protocol coverage within this hub. Signals: automatic metrics + pairwise preferences. Focus: agreement. Abstract: While AI systems show therapeutic promise,.

Known Limitations

• Only 8.7% of papers report quality controls; prioritize calibration/adjudication evidence.
• Benchmark coverage is thin (8.7% of papers mention benchmarks/datasets).
• Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.

Research Utility Snapshot