Skip to content
OpenTrain AIFor AI Companies

HFEPX · Eval paper review

RMIT-ADM+S at TREC RAG 2026: Research Agents, Retriever Ablations, and Open-Weight Replication

Oleg Zendel, Kun Ran, Chenglong Ma, Nguyen Khoi Vo +1 more

Published

Nov 12, 2026

Citations

0

Trust level

Low

Usefulness score

40/100 (Low)

Extraction confidence

45% (Low)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Nov 12, 2026

Open arXiv Find implementation Shortlist (0) RSS feed

Tags

General Rubric Rating

Should you rely on this paper?

This paper is adjacent to HFEPX scope and is best used for background context, not as a primary protocol reference.

Use this as background context only. Do not make protocol decisions from this page alone.

Best use

Background context only

Use if you need

Background context only.

What to verify

Read the full paper before copying any benchmark, metric, or protocol choices.

Main weakness

The available metadata is too thin to trust this as a primary source.

Human feedback signal
Detected
From extracted signals
Evaluation signal
Weak or implicit
Validate from full paper
Usefulness for eval research
40/100
Adjacent candidate

Treat as adjacent context, not a core eval-method reference.

If you are doing eval pipeline work, start here

Human Eval Papers LLM-as-Judge Papers Pairwise Preference Papers Tool-Use Evaluation Search

Abstract

We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus. We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows. Pre-submission ablations identified writer selection and adjacent-chunk expansion as the main factors affecting answer quality. In the official NIST evaluation, six of seven runs exceeded the track median across all three rubric/nugget collections. Our best-performing run, brief-hybrid, achieved the highest all-item coverage among our submissions and tied the track maximum (0.892) for Batch-1 vital coverage. Human side-by-side preferences differed across evaluation batches: the open-weight owa-glm5 led Batch 1 despite lower automated coverage, while sol-research-first led Batch 2. These findings highlight differences between automated coverage measures and human preferences, as well as the potential of lower-cost open-weight models for research-agent systems.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

partial

Rubric Rating

Directly usable for protocol triage.

"We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus."

Evaluation Modes

missing

None explicit

Validate eval design from full paper text.

"We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus."

Quality Controls

missing

Not reported

No explicit QC controls found.

"We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus."

Benchmarks / Datasets

missing

Not extracted

No benchmark anchors detected.

"We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus."

Reported Metrics

missing

Not extracted

No metric anchors detected.

"We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus."

Benchmarks and datasets

No benchmark or dataset names were extracted from the available abstract.

Reported metrics

No metric terms were extracted from the available abstract.

Human feedback details
Uses human feedback
Yes
Feedback types
Rubric Rating
Rater population
Not reported
Unit of annotation
Pairwise (inferred)
Expertise required
General
Evaluation details
Evaluation modes
None
Agentic eval
None
Quality controls
Not reported
Evidence quality
Low
Use this page as
Background context only

Research brief

Metadata summary

We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

  • We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus.
  • We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows.
  • Pre-submission ablations identified writer selection and adjacent-chunk expansion as the main factors affecting answer quality.

Researcher actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Rubric rating evaluation

Contribution summary

  • We describe the RMIT-ADM+S team's participation in the TREC 2026 Retrieval-Augmented Generation (RAG) track using the ClimbMix corpus.
  • We submitted seven research-agent runs using proprietary OpenAI and open-weight models, exploring different retrieval strategies, writer models, and agent workflows.

Why it matters for eval

  • In the official NIST evaluation, six of seven runs exceeded the track median across all three rubric/nugget collections.

Researcher checklist

  • Human feedback protocol is explicit

    Detected: Rubric Rating

  • Evaluation mode is explicit

    No clear evaluation mode extracted.

  • Quality control reporting appears

    No calibration/adjudication/IAA control explicitly detected.

  • Benchmark or dataset anchors are present

    No benchmark/dataset anchor extracted from abstract.

  • Metric reporting is present

    No metric terms extracted.