Shrey Shah, Yinheng Li · Oct 6, 2026 · Citations: 0
Automatic MetricsLaw
We study evaluation from a fixed matrix of K independent scores per candidate for a policy that selects using J fresh scores.
A single estimator based only on this matrix is exactly unbiased for expected judge reward under every independent, stable collection of candidate-specific score laws if and only if J<K, for every pool size M\ge N\ge2.
Nur A Zarin Nishat, Jens Lehmann, Andrei Aioanei, Sahar Vahdati · Oct 6, 2026 · Citations: 0
Automatic MetricsGeneral
Endpoint accuracy on abstract-reasoning benchmarks does not reveal whether a language model has acquired a transferable rule or fit distribution-specific regularities.
We study this distinction in small language models on the ARC-TGI benchmark, which organizes abstract grid transformations into controllable task families and supports resampling, spatial shifts, and cross-benchmark transfer.
Haoyu Huang, Zhongwei Xie, Jiaxin Bai, Yisen Gao, Hong Ting Tsang, Wuganjing Song · Oct 6, 2026 · Citations: 0
Pairwise PreferenceAutomatic MetricsGeneral
Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, and interaction experience.
In-context learning (ICL) and ICL-based agent harness remain flexible, but they consume context capacity and incur repeated discretized encoding cost that grows with context length.
Dan Ben-Ami, Kobi Cohen, Chaim Baskin · Oct 6, 2026 · Citations: 0
Automatic MetricsGeneral
It decodes in all seven models of a shared byte-identical evaluation (AUROC 0.733-0.905 under the strictest not-ready sampling, where a fitted clock is near chance), and a probe fitted without any of a benchmark family's footage still reads…
Readiness also beats uncertainty estimators and their supervised combination on latency-matched answer selection, and tracks independent human judgments more closely than confidence.
Language models promise tireless, consistent assessment, but can accurate raters disagree about individuals?
Two randomly drawn raters with area under the receiver operating characteristic curve (AUC) >= 0.70 disagreed on screening decisions for 40% of participants, on average.
Lasse B. Strand, Robert Jakob, Kevin O'Sullivan, Markus Kreft · Oct 6, 2026 · Citations: 0
Automatic MetricsGeneral
We introduce Agentic AutoRAG, an LLM-agent optimizer for multi-objective RAG hyperparameter optimization with retrieval-versus-generation failure attribution.
On three multi-hop QA benchmarks it reaches higher LLM-judge accuracy than every baseline we compare, and within its first 10 trials it matches or beats the statistical baselines' full 30-trial judge accuracy.
Jerzy Kamiński, Ilya Galyukshev, Artem Kuznetsov, Danil Fedorov, Kirill Redko, Sergey Chuprin · Oct 6, 2026 · Citations: 0
Automatic MetricsMath
We contribute a turn-labeled multi-turn benchmark (423 conversations, 1,661 labeled turn-states) and an evaluation harness with a simulated user who answers clarifying questions, and use them with six datasets and six open-weight LLMs to…
Shu-Kai Hsieh, Da-Chen Lian · Oct 6, 2026 · Citations: 0
Automatic MetricsMultilingual
Current evaluation of multilingual Large Language Models (LLMs) rests on an implicit Translation-Isomorphism Assumption (TIA): that semantic structures across languages are congruent and mutually mappable without loss of information.
We replicate a low-powered psycholinguistic finding at NLP scale, namely that human reading times at the main verb increase with dependency length, driven by syntactic embedding beyond linear distance.
Different language models -- spanning n-gram models, SSMs, and transformers -- partially mirror this graded difficulty profile, yet underestimate the integration cost humans incur, with a gap that persists across architectures and model…
