Jonathan Ivey, Aimee Liang, Arthur Y. S. Wang, Madeline Mandell, Ziang Xiao, Anjalie Field · Oct 8, 2026 · Citations: 0
Pairwise PreferenceSimulation EnvGeneral
Increasingly, AI interviewers are being developed to elicit open-ended responses in applications like market research, public polling, preference elicitation, and social science research.
To test whether our simulation-based evaluations predict performance with human participants, we conduct 15 real qualitative studies with five AI interviewers, three interview topics, and 450 human participants and compare them to simulated…
Sarim Hashmi, Mukul Ranjan, Kshitij Mishra, Mikhail Kuznetsov, Praneeth Vepakomma, Nils Lukas · Oct 6, 2026 · Citations: 0
Simulation EnvGeneral
Web agents complete user requests by reading and acting on pages that third parties write, so an instruction planted on a page can redirect the agent away from the user's goal.
We introduce AdvSim2Real, which co-evolves a task curriculum, an injection adversary, and the agent inside a frozen web world model.
Haoyu Huang, Zhongwei Xie, Jiaxin Bai, Yisen Gao, Hong Ting Tsang, Wuganjing Song · Oct 6, 2026 · Citations: 0
Pairwise PreferenceAutomatic MetricsGeneral
Recently Large Language Models (LLMs) and LLM-based agents increasingly need to incorporate knowledge acquired after pretraining, e.g., domain facts, user preferences, documents, and interaction experience.
In-context learning (ICL) and ICL-based agent harness remain flexible, but they consume context capacity and incur repeated discretized encoding cost that grows with context length.
Stephanie Buttigieg, Maeve Madigan, Parameswaran Kamalaruban, Stuart Burrell · Oct 6, 2026 · Citations: 0
Pairwise PreferenceGeneral
We study the linear debiasing direction obtained by contrasting the activations of anti-biased and biased prompts, and evaluate it as a steering intervention across bias and general knowledge benchmarks.
Steering along it does reduce measured bias, but this is a consequence of reducing model confidence: on QA benchmarks we find that this steering drives the model to abstain from answering, with a side effect of improving fairness metrics.
Kyojun Choo, Minsoo Song, Yunju Kang, Chanjun Park · Oct 6, 2026 · Citations: 0
Automatic MetricsGeneral
Before selecting a specific tool, an agent must first infer the capability requirements implied by the user query.
Using 1,600 balanced training queries from benchmarks, synthetic examples, and new domain scenarios, we train linear probes on pre-generation hidden states from four open-weight LLM families.
When using an LLM agent in a consequential domain, making an informed decision about whether to trust its output or intervene requires calibrated confidence in the agent's success.
To address these challenges, we introduce Confidence Reasoning Graphs (CRGs), an inference-time framework that estimates the probability an agent accomplished its task from a single trajectory, without privileged model access or training…
Yupeng Su, Jiayi Tian, Zheng Zhang, Souvik Kundu · Oct 6, 2026 · Citations: 0
Automatic MetricsGeneral
Long-horizon LLM agents act on an append-only interaction history that is re-sent to the model at every step, so the context and its cost grow with steps until the sessions exceed the context window.
It removes two kinds of inter-turn redundancy without an auxiliary predictor: content an earlier turn already displayed, replaced by a stub, and turns the agent itself reports finished, folded into a one-line note.
Sihyeon Lee, Jihun Song, Chanwoo Kim, Jiwoo Kum, Chanjun Park · Oct 6, 2026 · Citations: 0
Pairwise PreferenceGeneral
Yet omission bias remains underexplored in LLM evaluation, with the few existing studies limited in scale and focused largely on utilitarian-deontological conflicts.
To address this gap, we introduce OMIT, a benchmark consisting of 218 paired-frame scenarios across 10 conflict types, constructed by leveraging disagreement patterns from an LLM-based, five-perspective philosophical persona panel…
Decomposing long-context inference across cooperating agents bounds the active KV cache per call rather than total evidence, which matters when KV-cache memory binds.
We measure both on one three-tier agent architecture.
Muhammad Faraz Shoaib, Muhammad Qasim, Raisulhaq Mohammed Rizwan, Rahmatullah Safdar, Muzammil Adnan Shaik, Abdul Aleem Mohammed · Oct 6, 2026 · Citations: 0
Critique EditAutomatic MetricsGeneral
We present LOGIC, a controlled benchmark and evaluation framework in which locally deployable language models ground a request in a deterministic candidate-change inventory before selected changes are propagated through a typed electrical…
Multi-agent simulations with large language models (LLMs) often operate networks of agents with a single base model.
To show this, we simulate a heterogeneous social network powered by several different base models and show that the amount of engagement an agent receives depends more on its base model than on its assigned persona.
Wonjun Lee, Kyungsik Yang, Gaeun Ji, Vaidehi Patil, Haon Park, Bumsub Ham · Oct 5, 2026 · Citations: 0
Red TeamAutomatic MetricsGeneral
To address these limitations, we introduce LADE (Latent Safety Signals for Defense), which leverages latent safety signals extracted by contrasting harmful and benign queries from dark knowledge (i.e., information carried by the output…
We show that these signals consistently align across LLMs, forming a model-agnostic direction that emerges from safety alignment.
Long-horizon LLM agents are frequently implemented using sparse mixture-of-experts (MoE) models, yet the co-design of agentic behavior and MoE structures remains underexplored.
To address this, we introduce a hierarchical routing control framework for agentic tasks.
David I. Atkinson, Dillon Plunkett, David Bau · Oct 5, 2026 · Citations: 0
Pairwise PreferenceGeneral
Using low-rank adapters, we train models to make decisions on behalf of fictitious characters, according to latent linear preference functions.
We find sustained fine-tuning on an implicit decision task can lead to the emergence of accurate self-reporting of models' learned preferences, even without explicit self-report supervision.
