Skip to content
OpenTrain AIFor AI Companies
← Back to explorer

Tag: Medicine

Medicine evaluation papers that call for domain expertise or specialist review (182 papers).

Papers in tag: 182

Global RSS

Need Medicine evaluators for your project?

Post a Job →

Research Utility Snapshot

Evaluation Modes

  • Automatic Metrics (9)
  • Llm As Judge (2)
  • Simulation Env (1)

Human Feedback Types

  • Pairwise Preference (7)
  • Expert Verification (5)
  • Rubric Rating (3)

Required Expertise

  • Medicine (20)
  • Coding (4)
  • Multilingual (2)
Symphony for Text Generation: Benchmarking Clinical Note Generation

Daniel Varab, Victor Petrén Bach Hansen, Asbjørn W. Helge, Kevin Pelgrims, Mathias Baltzersen, Adrian Young-San Roessler · Oct 6, 2026 · Citations: 0

Pairwise Preference MedicineMultilingual
  • We introduce MedConv, a multilingual dataset of 300 clinical encounters in English, Danish, and German, and use it alongside the Ambient Clinical Intelligence benchmark (ACI-BENCH) to compare Corti, a clinical AI platform, with two leading,…
  • We present a controlled clinical evaluation framework that combines entailment metrics with LLM-judged pairwise comparisons across eight dimensions adopted from PDSQI-9.
Harness Engineering for Software Engineering via Modular Executable Dev-Primitives

Haibo Jin, Xinjie Li, Peng Kuang, Haohan Wang · Oct 6, 2026 · Citations: 0

Automatic Metrics MedicineCoding
  • However, existing agents remain brittle on long-horizon workflows, where they must repeatedly reconstruct program state scattered across source files, configurations, tests, dependencies, and runtime behavior, leading to increasingly long…
  • Each Dev-Primitive pairs a repository artifact with a resident LLM, which gives the artifact an agent-native interface grounded in its own implementation and dependencies, enabling natural-language reasoning, inter-component communication,…
KlinikeBench: Evaluating Language Models Beyond Diagnostic Accuracy

Xueting Fang, Zehui Li, Yang Yang, Camilla Giovino, Shubh K. Patel, Shailly Prajapati · Sep 29, 2026 · Citations: 0

Expert Verification Automatic Metrics MedicineCoding
  • Diagnostic accuracy alone therefore cannot establish whether an agent gathered essential information or conducted an appropriate clinical assessment.
  • To address this gap, we introduce KlinikeBench, a benchmark of 333 clinician-authored tasks, each providing an isolated sandbox environment with a virtual patient, clinical tools, and task-specific success criteria.
A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses

Zhangshu Joshua Jiang, Zina Ibrahim, James T. Teo · Sep 29, 2026 · Citations: 0

Rubric Rating Automatic Metrics Medicine
  • Rubrics support the structured evaluation of language models.
  • We propose a rubric for assessing expressed clinical reasoning in model responses, drawing on three bodies of work: medical education assessment frameworks (ART, SCT, Key Feature Problems and OSCE); clinical LLM benchmarks (MedR-Bench,…
Co-Annotator: Expert-Distilled ViT and VLM for Visual and Documentation Guidance in Age-Related Macular Degeneration

Ziheng "Leo" Li, Benjamin Freeman, Akshay Raman, Kavin Aravindhan Rajkumar, Xinxin Fang, Rishabh Srivastava · Aug 31, 2026 · Citations: 0

Expert Verification Automatic Metrics Medicine
  • We present Co-Annotator, which distills expert gaze and dictation into two guidance components: a gaze-aligned Vision Transformer producing fixation-aligned areas of interest (AOIs), and an ontology-bounded vision-language model (VLM) that…
Free-Text Evaluation of LLMs for 5G Domain Knowledge and Fault Analysis using LLM-as-Judge

Rishiraj Sengupta, Sotiris Chatzimiltis, Mohammad Shojafar, Xiatian Zhu · Aug 21, 2026 · Citations: 0

Pairwise PreferenceExpert Verification Llm As JudgeAutomatic Metrics Medicine
  • While existing benchmarks rely on restrictive MCQs with fixed answer keys, this paper evaluates 5G domain understanding and fault analysis in a free-text generation format.
  • To address this we evaluate three lightweight LLMs, Claude-Haiku-4.5, GPT-5.4-Mini, and Gemini-3.1-Flash-Lite, on free-text 5G domain knowledge and fault-analysis tasks across three benchmarks, TeleQNA ORAN FT, 5G-Faults FT, and TeleInter…
HealMed: Multilingual Evaluation of Large Language Models in Medicine

Yingjian Chen, Fan Gao, Sherry T. Tong, Haoyu Zhang, Aosong Feng, Kevin W. Jin · Aug 20, 2026 · Citations: 0

Critique Edit MedicineMultilingual
  • We present HealMed, an expert-reviewed benchmark for multilingual evaluation of large language models in medicine.
  • The benchmark was developed over two years by 23 physicians and medical experts based across nine countries and regions.
MARC v1: An Open-Source Multi-Agent Framework for Clinical AI Reasoning and Coordination

Saisha Shetty, Satvik Tripathi, Austin Lin, Colin Zhao, Theodore Kim, Don Enwerem · Aug 13, 2026 · Citations: 0

Expert Verification MedicineCoding
  • We present Multi-Agent Reasoning and Coordination (MARC), an open-source framework that replaces monolithic LLM prompting with deterministic multi-agent orchestration for clinical reasoning.
  • MARC coordinates role-specialized agents for extraction, reasoning, answer generation, and evaluation, with explicit context passing and traceable intermediate outputs, enabling stage-wise failure attribution.
CRAFT: LLM-Based Iterative Refinement for Temporal Reasoning over Clinical Narratives

Chengyang He, Tahreem Arif, Marko Zivkovic, Lijing Wang, Yue Ning, Ping Wang · Aug 13, 2026 · Citations: 0

Pairwise Preference Automatic Metrics Medicine
  • Understanding the temporal progression of symptoms in clinical narratives is critical for disease monitoring, safety surveillance, and causality assessment.
  • We conduct evaluation on MedTempo, a new benchmark of 5,347 vaccine adverse-event narratives spanning three COVID-19 vaccine types, with expert-validated temporal stage annotations for 3,166 reports.
A corpus-specific clinical RAG system matches or outperforms newer frontier LLMs on HealthBench

Praveen Reddy, Charuta Mandke, Suvrankar Datta, Sarah Khan, Siddharth Reddy Anthireddy, Shitij Arora · Aug 12, 2026 · Citations: 0

Rubric Rating Llm As JudgeAutomatic Metrics Medicine
  • On 4,023 English-language HealthBench questions (80.5% of the benchmark), scored with a GPT-4.1 judge, VITA ranked first with 51.9% of possible rubric points, ahead of GPT-5.4 (46.1%), o4-mini (44.3%), Gemini 3.1 Pro (42.6%), and Claude…
  • VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower.
Preference Tree Optimization: Enhancing Goal-Oriented Dialogue with Look-Ahead Simulations

Lior Baruch, Moshe Butman, Kfir Bar, Doron Friedman · Aug 12, 2026 · Citations: 0

Pairwise Preference Simulation Env Medicine
  • This research proposes a novel framework called Preference Tree Optimization (PTO), designed to iteratively improve agent models in such dialogue systems, by generating preference data using a method called Preference Tree with Look-Ahead.
  • Focusing on Motivational Interviewing (MI) -- a counseling technique aimed at facilitating behavioral change -- we leverage virtual patients and an oracle evaluator to simulate conversations and generate rich preference datasets.
Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL

Minglai Yang, Xinyu Guo, Utkarsh Tyagi, Mian Zhang, Razvan Dumitru, Sunjie Hou · Aug 12, 2026 · Citations: 0

Rubric Rating Medicine
  • Reinforcement learning against rubrics, lists of criteria graded by an LLM judge, has become a standard way to post-train language models on tasks with no deterministic answer.
  • Comparing no dropout against dropout at 30% and 50% on both benchmark pairs, dropout raises the OOD gold score at every matched checkpoint (+1 to +2 points on HealthBench-Hard, +6 to +7 points on ResearchQA), lowers the two hacking measures…
Social Chain of Thought: A Multi-Agent Architecture Grounded in Medical Differential Diagnosis Methodology

Del Coburn, Scott Sanner, Dan Silver · Aug 11, 2026 · Citations: 0

Automatic Metrics Medicine
  • We introduce Social Chain of Thought (SCoT),a multi-round pipeline for medical differential diagnosis that structures multi-agent interaction as a deliberative framework for collabora.
  • Evaluating SCoT against single-agent baselines, one-agent pipeline ablations, and best-of-n scaling, we show that its recall advantage is not reproduced by monolithic inference alone.