HFEPX Hub
Updated from current HFEPX corpus (Feb 27, 2026). 23 papers are grouped in this hub page. Common evaluation modes: Automatic Metrics, Llm As Judge. Most common rater population: Domain Experts. Common annotation unit: Multi Dim Rubric. Frequent quality control: Gold Questions. Frequently cited benchmark: Retrieval. Common metric signal: accuracy. Newest paper in this set is from Feb 25, 2026.
Updated from current HFEPX corpus (Feb 27, 2026). This page covers 23 papers centered on Expert Verification Papers. Common evaluation modes include Automatic Metrics, Llm As Judge, with benchmark emphasis on Retrieval, BIRD. Use the anchored takeaways below to compare protocol choices and identify papers with stronger evidence depth.
Evaluation emphasis: Automatic Metrics and Llm As Judge appear frequently in this slice.
Evidence: MEDSYN: Benchmarking Multi-EviDence SYNthesis in Complex Clinical Cases for Multimodal Large Language Models , SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video , SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery
Benchmark concentration: Retrieval, BIRD helps control cross-paper variance.
Evidence: SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video , SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery , "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems
Metric concentration: accuracy, cost is repeatedly reported in this group.
Evidence: SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery , "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems , An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
Compare papers that report both human_eval and llm_as_judge to quantify judge-human agreement drift.
Evidence: "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems , An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems , An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models
Stratify by benchmark (Retrieval vs BIRD) before comparing methods.
Evidence: An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems , An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models , Hyper-KGGen: A Skill-Driven Knowledge Extractor for High-Quality Knowledge Hypergraph Generation
Papers with explicit human feedback is visible in approximately 100% of papers in this set.
Evidence: An artificial intelligence framework for end-to-end rare disease phenotyping from clinical notes using large language models , Hyper-KGGen: A Skill-Driven Knowledge Extractor for High-Quality Knowledge Hypergraph Generation , CUICurate: A GraphRAG-based Framework for Automated Clinical Concept Curation for NLP applications
Papers with explicit human feedback
Coverage is strong (100% vs 45% target).
Papers reporting quality controls
Coverage is a replication risk (17.4% vs 30% target).
Papers naming benchmarks/datasets
Coverage is usable but incomplete (26.1% vs 35% target).
Papers naming evaluation metrics
Coverage is strong (82.6% vs 35% target).
Papers with known rater population
Coverage is strong (100% vs 35% target).
Papers with known annotation unit
Coverage is usable but incomplete (30.4% vs 35% target).
Start with this anchor paper for scope and protocol framing. Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
2. SurGo-R1: Benchmarking and Modeling Contextual Reasoning for Operative Zone in Surgical Video
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
3. SparkMe: Adaptive Semi-Structured Interviewing for Qualitative Insight Discovery
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
4. "Are You Sure?": An Empirical Study of Human Perception Vulnerability in LLM-Driven Agentic Systems
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
5. An Expert Schema for Evaluating Large Language Model Errors in Scholarly Question-Answering Systems
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
7. Hyper-KGGen: A Skill-Driven Knowledge Extractor for High-Quality Knowledge Hypergraph Generation
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
8. CUICurate: A GraphRAG-based Framework for Automated Clinical Concept Curation for NLP applications
Covers Automatic Metrics. Includes human-feedback signal: Expert Verification.
human_eval vs llm_as_judge
both=0, left_only=1, right_only=2
0 papers use both Human Eval and Llm As Judge.
human_eval vs automatic_metrics
both=0, left_only=1, right_only=19
0 papers use both Human Eval and Automatic Metrics.
llm_as_judge vs automatic_metrics
both=0, left_only=2, right_only=19
0 papers use both Llm As Judge and Automatic Metrics.
Boqi Chen, Xudong Liu, Jiachuan Peng, Marianne Frey-Marti, Bang Zheng · Feb 25, 2026 · Citations: 0
Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity.
Guanyi Qin, Xiaozhen Wang, Zhu Zhuo, Chang Han Low, Yuancan Xiao · Feb 25, 2026 · Citations: 0
Existing AI systems offer binary safety verification or static detection, ignoring the phase-dependent nature of intraoperative reasoning.
David Anugraha, Vishakh Padmakumar, Diyi Yang · Feb 24, 2026 · Citations: 0
Based on this formulation, we introduce SparkMe, a multi-agent LLM interviewer that performs deliberative planning via simulated conversation rollouts to select questions with high expected utility.
Xinfeng Li, Shenyu Dai, Kelong Zheng, Yue Xiao, Gelei Deng · Feb 24, 2026 · Citations: 0
Large language model (LLM) agents are rapidly becoming trusted copilots in high-stakes domains like software development and healthcare.
Anna Martin-Boyle, William Humphreys, Martha Brown, Cara Leckey, Harmanpreet Kaur · Feb 24, 2026 · Citations: 0
Current evaluation metrics for testing LLM reliability are primarily automated approaches that prioritize efficiency and scalability, but lack contextual nuance and fail to reflect how scientific domain experts assess LLM outputs in practic
Cathy Shyr, Yan Hu, Rory J. Tinker, Thomas A. Cassini, Kevin W. Byram · Feb 23, 2026 · Citations: 0
Existing artificial intelligence approaches typically optimize individual components of phenotyping but do not operationalize the full clinical workflow of extracting features from clinical text, standardizing them to Human Phenotype Ontolo
Rizhuo Huang, Yifan Feng, Rundong Xue, Shihui Ying, Jun-Hai Yong · Feb 23, 2026 · Citations: 0
Additionally, we present \textbf{HyperDocRED}, a rigorously annotated benchmark for document-level knowledge hypergraph extraction.
Victoria Blake, Mathew Miller, Jamie Novak, Sze-yuan Ooi, Blanca Gallego · Feb 20, 2026 · Citations: 0
The framework was evaluated on five lexically heterogeneous clinical concepts against a manually curated benchmark and gold-standard concept sets.
Adrian Cosma, Cosmin Dumitrache, Emilian Radoi · Feb 19, 2026 · Citations: 0
As platforms increasingly rely on patient ratings and feedback, clinicians face growing pressure to maintain satisfaction scores, even though these evaluations often reflect communication quality more than clinical accuracy.
Jeffrey T. H. Wong, Zixi Zhang, Junyi Liu, Yiren Zhao · Feb 18, 2026 · Citations: 0
Existing Multi-Agent Systems (MAS) typically rely on static, homogeneous model configurations, limiting their ability to exploit the distinct strengths of differently post-trained models.
Mehrab Beikzadeh, Yasaman Asadollah Salmanpour, Ashima Suvarna, Sriram Sankararaman, Matteo Malgaroli · Feb 17, 2026 · Citations: 0
While AI systems show therapeutic promise, current alignment approaches optimize objectives independently, failing to balance patient preferences with clinical safety.
Shiwei Hong, Lingyao Li, Ethan Z. Rong, Chenxinran Shen, Zhicong Lu · Feb 16, 2026 · Citations: 0
Prior work has explored multi-turn interaction and feedback for LLM writing, but evaluations still largely center on prompts and localized feedback, leaving persistent public reception in online communities underexamined.
Weiqi Zhai, Zhihai Wang, Jinghang Wang, Boyu Yang, Xiaogang Li · Feb 15, 2026 · Citations: 0
Humanity's Last Exam (HLE) has become a widely used benchmark for evaluating frontier large language models on challenging, multi-domain questions.
Bertie Vidgen, Austin Mann, Abby Fennelly, John Wright Stanly, Lucas Rothman · Jan 20, 2026 · Citations: 0
We introduce the AI Productivity Index for Agents (APEX-Agents), a benchmark for assessing whether AI agents can execute long-horizon, cross-application tasks created by investment banking analysts, management consultants, and corporate law
Vaibhav Devraj, Dhruv Kumar, Jagat Sesh Challa, Parth Agarwal, Navya Kommuri · Dec 26, 2025 · Citations: 0
To investigate this potential capability gap, we present CricBench, a comprehensive benchmark suite for evaluating LLMs on specialized cricket data.
Mohammad Hossein Samaei, Faryad Darabi Sahneh, Lee W. Cohnstaedt, Caterina Scoglio · Sep 24, 2025 · Citations: 0
We introduce EpidemIQs, a novel multi-agent LLM framework that integrates user inputs and autonomously conducts literature review, analytical derivation, network modeling, mechanistic modeling, stochastic simulations, data visualization and
Heloisa Oss Boll, Antonio Oss Boll, Leticia Puttlitz Boll, Ameen Abu Hanna, Iacer Calixto · Jun 20, 2025 · Citations: 0
This study introduces DistillNote, an evaluation framework for LLM summaries that targets their functional utility by applying the generated summary downstream in a complex clinical prediction task, explicitly quantifying how much predictio
Nitin Sharma, Thomas Wolfers, Çağatay Yıldız · Jun 9, 2025 · Citations: 0
Accurate domain-specific benchmarking of LLMs is essential, specifically in domains with direct implications for humans, such as law, healthcare, and education.
Zhaolu Kang, Junhao Gong, Jiaxu Yan, Wanke Xia, Yian Wang · Jun 4, 2025 · Citations: 0
However, current benchmarks for evaluating MLLMs primarily emphasize general knowledge and vertical step-by-step reasoning typical of STEM disciplines, while overlooking the distinct needs and potential of the Humanities and Social Sciences
Neil Mallinar, A. Ali Heydari, Xin Liu, Anthony Z. Faranesh, Brent Winslow · Mar 30, 2025 · Citations: 0
As LLM-driven health applications are increasingly adopted, rigorous and efficient one-sided evaluation methodologies are crucial to ensure response quality across multiple dimensions, including accuracy, personalization and safety.
Hsin-Ling Hsu, Cong-Tinh Dao, Luning Wang, Zitao Shuai, Thao Nguyen Minh Phan · Mar 23, 2025 · Citations: 0
Comprehensive evaluation demonstrates that our method significantly outperforms baseline approaches in both assessment accuracy and treatment plan quality.
Thomas Kwa, Ben West, Joel Becker, Amy Deng, Katharyn Garcia · Mar 18, 2025 · Citations: 0
Despite rapid progress on AI benchmarks, the real-world meaning of benchmark performance remains unclear.
Max Lamparth, Declan Grabb, Amy Franks, Scott Gershan, Kaitlyn N. Kunstman · Feb 22, 2025 · Citations: 0
Current medical language model (LM) benchmarks often over-simplify the complexities of day-to-day clinical practice tasks and instead rely on evaluating LMs on multiple-choice board exam questions.
