Skip to content
← Back to explorer

HFEPX Hub

Automatic Metrics + Expert Verification + Medicine Papers

Updated from current HFEPX corpus (Mar 8, 2026). 16 papers are grouped in this hub page.

Read Full Context

Updated from current HFEPX corpus (Mar 8, 2026). 16 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Llm As Judge. Most common rater population: Domain Experts. Common annotation unit: Ranking. Frequent quality control: Adjudication. Frequently cited benchmark: Healthbench. Common metric signal: accuracy. Use this page to compare protocol setup, judge behavior, and labeling design decisions before running new eval experiments. Newest paper in this set is from Mar 3, 2026.

Papers: 16 Last published: Mar 3, 2026 Global RSS Tag RSS
Automatic MetricsExpert VerificationMedicine

Researcher Quick Triage

This hub is best used for protocol triage and replication planning from abstract-level evidence. Quality band: Developing .

All Sampled Papers (16) Replication-Ready Only (1)

High-Signal Coverage

100.0%

16 / 16 sampled papers are not low-signal flagged.

Replication-Ready Set

1

Benchmark + metric + eval mode explicitly present.

Judge/Human Comparability

0

Papers containing both `human_eval` and `llm_as_judge`.

  • 1 papers are replication-ready (benchmark + metric + explicit evaluation mode).
  • 0 papers support judge-vs-human agreement analysis.
  • 5 papers report explicit quality controls (calibration/adjudication/IAA).

Primary action: Use this page for scouting only; collect additional papers before attempting replication-critical comparisons.

Currently showing only replication-ready papers in ranking and matrix sections (1 papers).

Why This Matters For Eval Research

  • 100% of papers report explicit human-feedback signals, led by expert verification.
  • automatic metrics appears in 100% of papers in this hub.
  • Healthbench is a recurring benchmark anchor for cross-paper comparisons in this page.

Protocol Takeaways

  • Most common quality-control signal is adjudication (12.5% of papers).
  • Rater context is mostly domain experts, and annotation is commonly ranking annotation; use this to scope replication staffing.
  • Pair this hub with a human_eval-heavy hub to validate judge-model calibration.

Benchmark Interpretation

  • Healthbench appears in 6.3% of hub papers (1/16); use this cohort for benchmark-matched comparisons.
  • Pancanbench appears in 6.3% of hub papers (1/16); use this cohort for benchmark-matched comparisons.

Metric Interpretation

  • accuracy is reported in 43.8% of hub papers (7/16); compare with a secondary metric before ranking methods.
  • agreement is reported in 25% of hub papers (4/16); compare with a secondary metric before ranking methods.
Researcher Checklist (Expanded)

Researcher Checklist

  • Strong: Papers with explicit human feedback

    Coverage is strong (100% vs 45% target).

  • Strong: Papers reporting quality controls

    Coverage is strong (31.3% vs 30% target).

  • Gap: Papers naming benchmarks/datasets

    Coverage is a replication risk (6.3% vs 35% target).

  • Strong: Papers naming evaluation metrics

    Coverage is strong (100% vs 35% target).

  • Strong: Papers with known rater population

    Coverage is strong (100% vs 35% target).

  • Strong: Papers with known annotation unit

    Coverage is strong (43.8% vs 35% target).

Strengths

  • Strong human-feedback signal (100% of papers).
  • Quality-control evidence appears in 31.3% of papers.
  • Rater population and annotation-unit details are frequently specified.

Known Gaps

  • Benchmark coverage is thin (6.3% of papers mention benchmarks/datasets).

Suggested Next Analyses

  • Pair this hub with a human_eval-heavy hub to validate judge-model calibration.
  • Stratify by benchmark (Healthbench vs Pancanbench) before comparing methods.
  • Track metric sensitivity by reporting both accuracy and agreement.
Recommended Queries (Expanded)

Recommended Queries

LLM-as-Judge Protocols Benchmark Slice: Healthbench Metric Slice: accuracy IAA-Reported Evaluations Recent High-Signal Papers

Start with These 3

Use these when you need one protocol anchor, one benchmark anchor, and one recent comparison point before reading the wider hub.

Start Here (Best First 6)

Ranked for protocol completeness (human signal, benchmark + metric anchors, quality controls, and judge/human overlap).

Protocol Matrix (Top 12)

Use this to quickly compare protocol ingredients instead of scanning long prose.

Paper HF Signal Eval Modes Benchmarks Metrics QC
PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

Mar 2, 2026

 Yes
Rubric Rating , Expert Verification
Llm As Judge , Automatic Metrics Pancanbench , Healthbench Accuracy Not Reported
Suggested Reading Order (Extended)

This section is intentionally expanded only when needed; use “Start Here” above for a faster pass.

Suggested Reading Order

  1. Guideline-Grounded Evidence Accumulation for High-Stakes Agent Verification

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + expert verification. Focus: brier score. Abstract: As LLM-powered agents have been used for high-stakes decision-making,.

  2. An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + expert verification. Focus: f1. Abstract: Phenotyping is fundamental to rare disease diagnosis, but manual curation.

  3. An Agentic System for Rare Disease Diagnosis with Traceable Reasoning

    Start here for detailed protocol reporting and quality-control evidence. Signals: automatic metrics + expert verification. Focus: recall. Abstract: Rare diseases affect over 300 million individuals worldwide, yet timely.

  4. PrivMedChat: End-to-End Differentially Private RLHF for Medical Dialogue Systems

    High citation traction makes this a strong baseline for protocol comparison. Signals: automatic metrics + pairwise preferences. Focus: rouge. Abstract: Our design enforces differential privacy at every training.

  5. PanCanBench: A Comprehensive Benchmark for Evaluating Large Language Models in Pancreatic Oncology

    High citation traction makes this a strong baseline for protocol comparison. Signals: LLM-as-judge + rubric ratings. Focus: Pancanbench / accuracy. Abstract: Moreover, high rubric-based scores do not ensure.

  6. DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

    Include an LLM-as-judge paper to test judge design and agreement assumptions. Signals: LLM-as-judge + expert verification. Focus: auroc. Abstract: We contrasted DistillNote's results with evaluations from LLM-as-judge and.

  7. A Scalable Framework for Evaluating Health Language Models

    Adds automatic metrics with rubric ratings for broader protocol coverage within this hub. Signals: automatic metrics + rubric ratings. Focus: accuracy. Abstract: In this work, we introduce Adaptive.

  8. Modeling Expert AI Diagnostic Alignment via Immutable Inference Snapshots

    Adds automatic metrics with expert verification for broader protocol coverage within this hub. Signals: automatic metrics + expert verification. Focus: agreement. Abstract: Human-in-the-loop validation is essential in safety-critical.

Known Limitations

Known Limitations

  • Benchmark coverage is thin (6.3% of papers mention benchmarks/datasets).
  • Narrative synthesis is grounded in metadata and abstracts only; full-paper implementation details are not parsed.
  • Cross-page comparisons should be benchmark- and metric-matched to avoid protocol confounding.
Research Utility Snapshot (Detailed)

Research Utility Snapshot

Human Feedback Mix

  • Expert Verification (16)
  • Pairwise Preference (3)
  • Rubric Rating (2)

Evaluation Modes

  • Automatic Metrics (16)
  • Llm As Judge (2)

Top Benchmarks

  • Healthbench (1)
  • Pancanbench (1)

Top Metrics

  • Accuracy (7)
  • Agreement (4)
  • Recall (3)
  • Auroc (2)

Rater Population Mix

  • Domain Experts (14)
  • Mixed (2)

Quality Controls

  • Adjudication (2)
  • Calibration (1)
  • Gold Questions (1)
  • Inter Annotator Agreement Reported (1)
Coverage diagnostics (sample-based): human-feedback 100.0% · benchmarks 6.3% · metrics 100.0% · quality controls 31.3%.

Top Papers

Related Hubs

Need human evaluators for your AI research? Scale annotation with expert AI Trainers.

Post a Job Get a Quote