HFEPX Daily Archive: 2025-10-10

Why This Matters For Eval Research

• Evaluation emphasis: Automatic Metrics appear frequently in this slice.

  Evidence: The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach , Chlorophyll-a Mapping and Prediction in the Mar Menor Lagoon Using C2RCC-Processed Sentinel 2 Imagery , Verifying Chain-of-Thought Reasoning via Its Computational Graph

• Benchmark concentration: Retrieval helps control cross-paper variance.

  Evidence: Chlorophyll-a Mapping and Prediction in the Mar Menor Lagoon Using C2RCC-Processed Sentinel 2 Imagery , Verifying Chain-of-Thought Reasoning via Its Computational Graph , Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

• Metric concentration: accuracy, calibration is repeatedly reported in this group.

  Evidence: Verifying Chain-of-Thought Reasoning via Its Computational Graph , Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs , FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs

Protocol Takeaways

• Track metric sensitivity by reporting both accuracy and calibration.

  Evidence: Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs , FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs , The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach

• Add inter-annotator agreement checks when reproducing these protocols.

  Evidence: FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs , The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach , Chlorophyll-a Mapping and Prediction in the Mar Menor Lagoon Using C2RCC-Processed Sentinel 2 Imagery

• Papers with explicit human feedback is visible in approximately 0% of papers in this set.

  Evidence: The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach , Chlorophyll-a Mapping and Prediction in the Mar Menor Lagoon Using C2RCC-Processed Sentinel 2 Imagery , Verifying Chain-of-Thought Reasoning via Its Computational Graph

Benchmark Interpretation

• Retrieval appears as a recurring benchmark anchor in this page.
• 1 papers (20%) mention Retrieval.
• Most common evaluation modes: Automatic Metrics.

Metric Interpretation

• accuracy is a common reported metric and should be paired with protocol context before ranking methods.
• 1 papers (20%) mention accuracy.
• Most common evaluation modes: Automatic Metrics.

Researcher Checklist

• Papers with explicit human feedback: Coverage is a replication risk (0% vs 45% target).
• Papers reporting quality controls: Coverage is usable but incomplete (20% vs 30% target).
• Papers naming benchmarks/datasets: Coverage is a replication risk (20% vs 35% target).
• Papers naming evaluation metrics: Coverage is strong (40% vs 35% target).
• Papers with known rater population: Coverage is a replication risk (0% vs 35% target).
• Papers with known annotation unit: Coverage is a replication risk (0% vs 35% target).

Suggested Reading Order

1. 1. The Speech-LLM Takes It All: A Truly Fully End-to-End Spoken Dialogue State Tracking Approach

   Start with this anchor paper for scope and protocol framing. Covers Automatic Metrics.

2. 2. Chlorophyll-a Mapping and Prediction in the Mar Menor Lagoon Using C2RCC-Processed Sentinel 2 Imagery

   Covers Automatic Metrics.

3. 3. Verifying Chain-of-Thought Reasoning via Its Computational Graph

   Covers Automatic Metrics.

4. 4. Multimodal Prompt Optimization: Why Not Leverage Multiple Modalities for MLLMs

   Covers Automatic Metrics.

5. 5. FinAuditing: A Financial Taxonomy-Structured Multi-Document Benchmark for Evaluating LLMs

   Covers Automatic Metrics.