Human Feedback and Eval Paper Explorer

Match reason: Matches selected tags (Automatic Metrics, Coding).

Match reason: Matches selected tags (Automatic Metrics, Coding).

• We introduce PRBench, a benchmark of 30 expert-curated tasks spanning 11 subfields of physics.
• Using an agentified assessment pipeline, we evaluate a set of coding agents on PRBench and analyze their capabilities across key dimensions of scientific reasoning and execution.

Match reason: Matches selected tags (Automatic Metrics, Coding).

Match reason: Matches selected tags (Automatic Metrics, Coding).

• Multimodal large language models (MLLMs) have shown great potential in medical applications, yet existing benchmarks inadequately capture real-world clinical complexity.
• We introduce MEDSYN, a multilingual, multimodal benchmark of highly complex clinical cases with up to 7 distinct visual clinical evidence (CE) types per case.

Match reason: Matches selected tags (Automatic Metrics, Coding).

• Existing AI systems offer binary safety verification or static detection, ignoring the phase-dependent nature of intraoperative reasoning.
• We introduce ResGo, a benchmark of laparoscopic frames annotated with Go Zone bounding boxes and clinician-authored rationales covering phase, exposure quality reasoning, next action and risk reminder.

Match reason: Matches selected tags (Automatic Metrics, Coding).

• Based on this formulation, we introduce SparkMe, a multi-agent LLM interviewer that performs deliberative planning via simulated conversation rollouts to select questions with high expected utility.
• The code, datasets, and evaluation protocols for SparkMe are available as open-source at https://github.com/SALT-NLP/SparkMe.

Match reason: Matches selected tags (Automatic Metrics, Coding).

• However, evaluating these visual and textual search abilities is still difficult, and existing benchmarks have two major limitations.
• First, existing benchmarks are not visual search-centric: answers that should require visual search are often leaked through cross-textual cues in the text questions or can be inferred from the prior world knowledge in current MLLMs.

Match reason: Matches selected tags (Automatic Metrics, Coding).

• To investigate this potential capability gap, we present CricBench, a comprehensive benchmark suite for evaluating LLMs on specialized cricket data.
• We evaluate six state-of-the-art models, including GPT-4o, Claude 3.7 Sonnet, and open-source models, using a strict evaluation protocol.

Match reason: Matches selected tags (Automatic Metrics, Coding).

Match reason: Matches selected tags (Automatic Metrics, Coding).

• To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse project-level tasks.
• Furthermore, to overcome the inaccuracy of general LLM judges, we propose a highly reliable evaluation framework powered by a specialized, fine-tuned model.

Match reason: Matches selected tags (Automatic Metrics, Coding).

• However, having AI models generate full reviews in the same way as human reviewers risks exacerbating the irresponsible use of LLM-generated reviews and instigating intentional manipulation.
• We introduce several baseline approaches and an extendable automatic evaluation framework using top reasoning LLMs as judges to tackle the difficulty of recruiting domain experts for manual evaluation.