Skip to content
OpenTrain AIFor AI Companies

HFEPX · Eval paper review

UserProxyBench: Evaluating LLM User Simulators for Agent Benchmarks and Training

Ashish Jain, Armaan Sandhu

Published

Sep 29, 2026

Citations

0

Trust level

High

Usefulness score

67/100 (Medium)

Extraction confidence

80% (High)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Sep 29, 2026

Open arXiv Find implementation Shortlist (0) RSS feed

Tags

General Rubric Rating Simulation Env

Should you rely on this paper?

This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.

Use this as a practical starting point for protocol research, then validate against the original paper.

Best use

Secondary protocol comparison source

Use if you need

A secondary eval reference to pair with stronger protocol papers.

What to verify

Validate the evaluation procedure and quality controls in the full paper before operational use.

Main weakness

No major weakness surfaced.

Human feedback signal
Detected
From extracted signals
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
67/100
Moderate-confidence candidate

Useful as a secondary reference; validate protocol details against neighboring papers.

If you are doing eval pipeline work, start here

Human Eval Papers LLM-as-Judge Papers Pairwise Preference Papers Tool-Use Evaluation Search

Abstract

Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user. This simulated user controls what information the agent receives and when, yet current benchmarks score only the agent and do not directly measure whether the user correctly executed its assigned role. We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success. Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks changes mean task reward by 15.2 points, while 24.4% of successful episodes contain a user-specification violation. The dominant failure is premature disclosure: users provide information before it is requested. This behavior has little effect on task reward, yet among successful episodes it causes the agent to make 1.06 fewer tool calls on average, changing the interaction being evaluated while preserving the reward. Finally, across seven proxies we identify an empirical cost-fidelity frontier, enabling practitioners to select the least expensive simulator that satisfies a required fidelity level.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

strong

Rubric Rating

Directly usable for protocol triage.

"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user."

Evaluation Modes

strong

Simulation Env

Includes extracted eval setup.

"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user."

Quality Controls

missing

Not reported

No explicit QC controls found.

"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user."

Benchmarks / Datasets

strong

Tau Bench, Userproxybench

Useful for quick benchmark comparison.

"We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success."

Reported Metrics

missing

Not extracted

No metric anchors detected.

"Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user."

Benchmarks and datasets

Tau-BenchUserproxybench

Reported metrics

No metric terms were extracted from the available abstract.

Human feedback details
Uses human feedback
Yes
Feedback types
Rubric Rating
Rater population
Not reported
Unit of annotation
Multi Dim Rubric
Expertise required
General
Evaluation details
Evaluation modes
Simulation Env
Agentic eval
None
Quality controls
Not reported
Evidence quality
High
Use this page as
Secondary protocol comparison source

Research brief

Metadata summary

Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

  • Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user.
  • This simulated user controls what information the agent receives and when, yet current benchmarks score only the agent and do not directly measure whether the user correctly executed its assigned role.
  • We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of agent success.

Researcher actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Check the full text for explicit evaluation design choices (raters, protocol, and metrics).
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Rubric rating evaluation

Contribution summary

  • Interactive agent benchmarks and multi-turn reinforcement learning increasingly place a second language model in the role of the user.
  • We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of…
  • Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks changes mean task reward by 15.2 points, while 24.4% of successful episodes contain a user-specification violation.

Why it matters for eval

  • We introduce UserProxyBench, an evaluation layer over the tau-bench family, and the User Fidelity Score (UFS), which measures adherence to the benchmark's private user instructions using task-grounded rubric criteria scored independently of…
  • Holding the agent fixed at GPT-5.5 and varying only the user proxy across 375 enterprise tasks changes mean task reward by 15.2 points, while 24.4% of successful episodes contain a user-specification violation.

Researcher checklist

  • Human feedback protocol is explicit

    Detected: Rubric Rating

  • Evaluation mode is explicit

    Detected: Simulation Env

  • Quality control reporting appears

    No calibration/adjudication/IAA control explicitly detected.

  • Benchmark or dataset anchors are present

    Detected: Tau-Bench, Userproxybench

  • Metric reporting is present

    No metric terms extracted.