HFEPX Weekly Archive: 2026-W01

Why This Matters For Eval Research

• 16.7% of papers report explicit human-feedback signals, led by pairwise preferences.

  Evidence: ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System , Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study , Fast-weight Product Key Memory , RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

• automatic metrics appears in 100% of papers in this hub.

  Evidence: ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System , Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study , Fast-weight Product Key Memory , RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

• Needle In A Haystack is a recurring benchmark anchor for cross-paper comparisons in this page.

  Evidence: ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System , Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study , Fast-weight Product Key Memory , RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

Protocol Takeaways

• Most common quality-control signal is rater calibration (16.7% of papers).

  Evidence: WISE: Web Information Satire and Fakeness Evaluation , ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System , Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study , Fast-weight Product Key Memory

• Rater context is mostly domain experts, and annotation is commonly pairwise annotation; use this to scope replication staffing.

  Evidence: Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss , ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System , Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study , Fast-weight Product Key Memory

• Stratify by benchmark (Needle In A Haystack vs Retrieval) before comparing methods.

  Evidence: ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System , Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study , Fast-weight Product Key Memory , RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

Benchmark Interpretation

• Needle In A Haystack appears in 16.7% of hub papers (1/6); use this cohort for benchmark-matched comparisons.
• Retrieval appears in 16.7% of hub papers (1/6); use this cohort for benchmark-matched comparisons.

Metric Interpretation

• cost is reported in 33.3% of hub papers (2/6); compare with a secondary metric before ranking methods.
• accuracy is reported in 16.7% of hub papers (1/6); compare with a secondary metric before ranking methods.

Researcher Checklist

• Close gap on Papers with explicit human feedback. Coverage is a replication risk (16.7% vs 45% target).
• Close gap on Papers reporting quality controls. Coverage is a replication risk (16.7% vs 30% target).
• Close gap on Papers naming benchmarks/datasets. Coverage is a replication risk (16.7% vs 35% target).
• Maintain strength on Papers naming evaluation metrics. Coverage is strong (66.7% vs 35% target).
• Close gap on Papers with known rater population. Coverage is a replication risk (16.7% vs 35% target).
• Close gap on Papers with known annotation unit. Coverage is a replication risk (16.7% vs 35% target).

Suggested Reading Order

1. 1. ARGUS: Adaptive Rotation-Invariant Geometric Unsupervised System

   Start here for detailed protocol reporting, including rater and quality-control evidence.

2. 2. Improving Variational Autoencoder using Random Fourier Transformation: An Aviation Safety Anomaly Detection Case-Study

   Start here for detailed protocol reporting, including rater and quality-control evidence.

3. 3. Fast-weight Product Key Memory

   Start here for detailed protocol reporting, including rater and quality-control evidence.

4. 4. RAIR: A Rule-Aware Benchmark Uniting Challenging Long-Tail and Visual Salience Subset for E-commerce Relevance Assessment

   Adds automatic metrics for broader coverage within this hub.

5. 5. WISE: Web Information Satire and Fakeness Evaluation

   Adds automatic metrics for broader coverage within this hub.

6. 6. Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss

   Adds automatic metrics for broader coverage within this hub.