HFEPX · Eval paper review
Mateusz Modrzejewski, Tomasz Radzikowski
Published
Oct 22, 2026
Citations
0
Trust level
Moderate
Usefulness score
65/100 (Medium)
Extraction confidence
70% (Moderate)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Oct 22, 2026
This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.
Use this for comparison and orientation, not as your only source.
Best use
Secondary protocol comparison source
Use if you need
A secondary eval reference to pair with stronger protocol papers.
What to verify
Validate the evaluation procedure and quality controls in the full paper before operational use.
Main weakness
No major weakness surfaced.
Useful as a secondary reference; validate protocol details against neighboring papers.
If you are doing eval pipeline work, start here
Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher. We ask whether giving the models tools makes such comparisons more stable. An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once. We propose a matched protocol that separates those steps: the answer readout, reasoning, the decision to call, the tool's answer, and the report supplied without a call. We apply it to six open models in five domains under five wordings. We find that tools raise accuracy where a program answers the question and widen the wording sensitivity of many pairwise comparisons. The widening follows how often a model calls, and call frequency follows the wording for some models and not others. It also depends on the interface and the instructions: a native tool-calling interface and a second tool description leave no detected widening in molecules, and sensitivity persists where tools return intermediate results. These findings show that deterministic reports do not guarantee stable comparisons. We recommend reporting call frequency and comparison stability under several wordings beside accuracy.
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Pairwise Preference
Directly usable for protocol triage.
"Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher."
Automatic Metrics
Includes extracted eval setup.
"Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher."
Not reported
No explicit QC controls found.
"Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher."
Not extracted
No benchmark anchors detected.
"Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher."
Accuracy
Useful for evaluation criteria comparison.
"An accuracy gain cannot answer this, because tools change several steps between the question and the scored answer at once."
No benchmark or dataset names were extracted from the available abstract.
Language models are compared on question-answering benchmarks that pose each question in one wording, and a reworded question can change which of two models scores higher.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Human feedback protocol is explicit
Detected: Pairwise Preference
Evaluation mode is explicit
Detected: Automatic Metrics
Quality control reporting appears
No calibration/adjudication/IAA control explicitly detected.
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
Metric reporting is present
Detected: accuracy
