HFEPX · Eval paper review
Sungwoo Jung, Seonil Son
Published
Apr 8, 2026
Citations
0
Trust level
Moderate
Usefulness score
65/100 (Medium)
Extraction confidence
70% (Moderate)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Apr 26, 2026
This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.
Use this for comparison and orientation, not as your only source.
Best use
Secondary protocol comparison source
Use if you need
A secondary eval reference to pair with stronger protocol papers.
What to verify
Validate the evaluation procedure and quality controls in the full paper before operational use.
Main weakness
No major weakness surfaced.
Useful as a secondary reference; validate protocol details against neighboring papers.
If you are doing eval pipeline work, start here
Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times. That observation raises a question asked less often than it should be: once the harness is serious, how much of an agent's competence does the harness itself already carry, and how much genuinely still needs the LLM? We study this in noisy Collaborative Battleship, a partially observable planning setting with belief update, information-gathering questions, and uncertainty-aware action selection. We externalize a planning harness into four progressively richer layers -- posterior belief tracking, declarative planning, symbolic reflection, and an LLM-backed revision gate -- and report per-layer contribution under a common runtime. We report \emph{win rate} as the primary, game-level metric and \emph{F1} as a secondary, local-targeting indicator, and pre-specify \emph{heavy lifting} as the single largest positive marginal to the primary metric. Across 54 games, the declarative planning layer does most of the heavy lifting under this criterion, raising win rate from 50.0\% (Wilson 95\% CI $[37.1,62.9]$) to 74.1\% ($[61.1,83.9]$) over a belief-only harness (+24.1pp, +0.017 F1). Symbolic reflection is mechanistically real but calibration-sensitive, shifting board-level outcomes by up to $\pm0.140$ F1 without being net-positive on aggregate. LLM-backed revision activates on only 4.3\% of turns at the strictest confidence threshold and yields a small, non-monotonic change (+0.005 F1, -3.7pp win rate). The contribution is methodological: once harness layers are made externally measurable, one can ask not only how far the harness already carries the agent, but also where the LLM's role is actually residual rather than central.
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
Critique Edit
Directly usable for protocol triage.
"Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times."
Automatic Metrics
Includes extracted eval setup.
"Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times."
Not reported
No explicit QC controls found.
"Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times."
Not extracted
No benchmark anchors detected.
"Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times."
F1, Win rate
Useful for evaluation criteria comparison.
"We report \emph{win rate} as the primary, game-level metric and \emph{F1} as a secondary, local-targeting indicator, and pre-specify \emph{heavy lifting} as the single largest positive marginal to the primary metric."
No benchmark or dataset names were extracted from the available abstract.
Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Human feedback protocol is explicit
Detected: Critique Edit
Evaluation mode is explicit
Detected: Automatic Metrics
Quality control reporting appears
No calibration/adjudication/IAA control explicitly detected.
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
Metric reporting is present
Detected: f1, win rate
