Human Feedback Types
missingNone explicit
No explicit feedback protocol extracted.
"Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation."
HFEPX · Eval paper review
Eugene Agichtein, Charles L. A. Clarke, Kaustubh Dhole, William Stern
Published
Nov 18, 2026
Citations
0
Trust level
Low
Usefulness score
2/100 (Low)
Extraction confidence
40% (Low)
Derived from extracted protocol signals and abstract evidence.
Rater population
Not reported
Signals refreshed
Nov 18, 2026
This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.
Use this as background context only. Do not make protocol decisions from this page alone.
All signals on this page are inferred from the abstract only and may be inaccurate. Do not use this page as a primary protocol reference.
Best use
Background context only
Use if you need
A secondary eval reference to pair with stronger protocol papers.
What to verify
Read the full paper before copying any benchmark, metric, or protocol choices.
Main weakness
This paper looks adjacent to evaluation work, but not like a strong protocol reference.
Treat as adjacent context, not a core eval-method reference.
If you are doing eval pipeline work, start here
Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation. However, each judgment requires a potentially expensive call to a frontier model, and the judgments are hard to interpret and may not be reproducible. In this paper, we ask whether an LLM can write a readable program for each topic that replaces its per-document judgments. To address these issues, we propose ExecJudges, a method for generating fast, inspectable and scalable judges for automatic relevance labeling. Specifically, ExecJudges uses a frontier LLM to turn assessment guidelines into executable programs, which rely only on lexical matching, standard NLP libraries like parsers, and pool statistics. An ExecJudges-L variant also uses examples of human labels if available. We evaluate ExecJudges on seven TREC collections by comparing the produced labels with trained NIST assessors, frontier LLM judges, and recently proposed per-topic classifiers. Our results show that ExecJudges agrees with NIST assessors at the level of the Claude Sonnet 5.5 LLM judges on TREC DL 2019-2021. With a small number of human labels per topic, the ExecJudges-L variant agrees with NIST more than per-topic classifiers with the same labels on all seven collections, and either exceeds the agreement of LLM judges provided the same labels or is within about 0.05 of it in Krippendorff's alpha. The labels rank the official TREC DL runs with Kendall's tau of 0.87 to 0.94, and writing the programs incurs a one-time cost of $0.17 to $0.49 per topic, with no additional costs for grading. Our results suggest that ExecJudges offers a reproducible, inspectable, efficient and cost-effective alternative to LLM judges for labeling deep result pools for system evaluation or ranking improvements.
These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.
None explicit
No explicit feedback protocol extracted.
"Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation."
Llm As Judge
Includes extracted eval setup.
"Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation."
Inter Annotator Agreement Reported
Calibration/adjudication style controls detected.
"Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation."
Not extracted
No benchmark anchors detected.
"Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation."
Not extracted
No metric anchors detected.
"Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation."
No benchmark or dataset names were extracted from the available abstract.
No metric terms were extracted from the available abstract.
Large language model (LLM) judges have become a common way to create large-scale relevance judgments for retrieval evaluation.
Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.
Human feedback protocol is explicit
No explicit human feedback protocol detected.
Evaluation mode is explicit
Detected: Llm As Judge
Quality control reporting appears
Detected: Inter Annotator Agreement Reported
Benchmark or dataset anchors are present
No benchmark/dataset anchor extracted from abstract.
Metric reporting is present
No metric terms extracted.