HFEPX Fortnight Archive: 2025-F06

Why This Matters For Eval Research

• 33.3% of papers report explicit human-feedback signals, led by expert verification.

  Evidence: MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation , Measuring AI Ability to Complete Long Software Tasks , Imitating AI agents increase diversity in homogeneous information environments but can reduce it in heterogeneous ones , EmoGRACE: Aspect-based emotion analysis for social media data

• automatic metrics appears in 66.7% of papers in this hub.

  Evidence: MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation , EmoGRACE: Aspect-based emotion analysis for social media data , Measuring AI Ability to Complete Long Software Tasks , Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes

• Retrieval is a recurring benchmark anchor for cross-paper comparisons in this page.

  Evidence: MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation , A Survey on the Optimization of Large Language Model-based Agents , Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes , Imitating AI agents increase diversity in homogeneous information environments but can reduce it in heterogeneous ones

Protocol Takeaways

• Quality-control reporting is sparse in this slice; prioritize papers with explicit calibration or adjudication steps.

  Evidence: MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation , Imitating AI agents increase diversity in homogeneous information environments but can reduce it in heterogeneous ones , EmoGRACE: Aspect-based emotion analysis for social media data , Measuring AI Ability to Complete Long Software Tasks

• Rater context is mostly domain experts, and annotation is commonly trajectory-level annotation; use this to scope replication staffing.

  Evidence: MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation , Measuring AI Ability to Complete Long Software Tasks , Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes , Imitating AI agents increase diversity in homogeneous information environments but can reduce it in heterogeneous ones

• Stratify by benchmark (Retrieval vs Re-Bench) before comparing methods.

  Evidence: MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation , Imitating AI agents increase diversity in homogeneous information environments but can reduce it in heterogeneous ones , EmoGRACE: Aspect-based emotion analysis for social media data , Measuring AI Ability to Complete Long Software Tasks

Benchmark Interpretation

• Retrieval appears in 50% of hub papers (3/6); use this cohort for benchmark-matched comparisons.
• Re-Bench appears in 16.7% of hub papers (1/6); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• accuracy is reported in 33.3% of hub papers (2/6); compare with a secondary metric before ranking methods.
• f1 is reported in 16.7% of hub papers (1/6); compare with a secondary metric before ranking methods.

Researcher Checklist

• Tighten coverage on Papers with explicit human feedback. Coverage is usable but incomplete (33.3% vs 45% target).
• Close gap on Papers reporting quality controls. Coverage is a replication risk (0% vs 30% target).
• Maintain strength on Papers naming benchmarks/datasets. Coverage is strong (66.7% vs 35% target).
• Maintain strength on Papers naming evaluation metrics. Coverage is strong (66.7% vs 35% target).
• Maintain strength on Papers with known rater population. Coverage is strong (50% vs 35% target).
• Close gap on Papers with known annotation unit. Coverage is a replication risk (16.7% vs 35% target).

Suggested Reading Order

1. 1. MedPlan: A Two-Stage RAG-Based System for Personalized Medical Plan Generation

   Start here for detailed protocol reporting, including rater and quality-control evidence.

2. 2. Imitating AI agents increase diversity in homogeneous information environments but can reduce it in heterogeneous ones

   Start here for detailed protocol reporting, including rater and quality-control evidence.

3. 3. EmoGRACE: Aspect-based emotion analysis for social media data

   Start here for detailed protocol reporting, including rater and quality-control evidence.

4. 4. Measuring AI Ability to Complete Long Software Tasks

   Adds automatic metrics with expert verification for broader coverage within this hub.

5. 5. A Survey on the Optimization of Large Language Model-based Agents

   Adds simulation environments for broader coverage within this hub.

6. 6. Integrating Chain-of-Thought and Retrieval Augmented Generation Enhances Rare Disease Diagnosis from Clinical Notes

   Adds automatic metrics for broader coverage within this hub.