Skip to content
OpenTrain AIFor AI Companies

HFEPX · Eval paper review

AJAR: Adaptive Jailbreak Architecture for Red-teaming

Yipu Dou, Wang Yang

Published

Jan 16, 2026

Citations

0

Trust level

High

Usefulness score

67/100 (Medium)

Extraction confidence

80% (High)

Derived from extracted protocol signals and abstract evidence.

Rater population

Not reported

Signals refreshed

Mar 19, 2026

Open arXiv Find implementation Shortlist (0) RSS feed

Tags

Coding Red Team Simulation Env

Should you rely on this paper?

This paper has useful evaluation signal, but protocol completeness is partial; pair it with related papers before deciding implementation strategy.

Use this as a practical starting point for protocol research, then validate against the original paper.

Best use

Secondary protocol comparison source

Use if you need

A benchmark-and-metrics comparison anchor.

What to verify

Validate the evaluation procedure and quality controls in the full paper before operational use.

Main weakness

No major weakness surfaced.

Human feedback signal
Detected
From extracted signals
Evaluation signal
Detected
Eval setup described
Usefulness for eval research
67/100
Moderate-confidence candidate

Useful as a secondary reference; validate protocol details against neighboring papers.

If you are doing eval pipeline work, start here

Human Eval Papers LLM-as-Judge Papers Pairwise Preference Papers Tool-Use Evaluation Search

Abstract

Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops. Existing jailbreak frameworks still leave a gap between adaptive multi-turn attacks and agentic runtimes: attack algorithms are usually packaged as monolithic scripts, while agent harnesses rarely expose explicit abstractions for rollback, tool simulation, or strategy switching. We present AJAR, a red-teaming framework that exposes multi-turn jailbreak algorithms as callable MCP services and lets an Auditor Agent orchestrate them inside a tool-aware runtime built on Petri. AJAR integrates three representative attacks, namely Crescendo, ActorAttack, and X-Teaming, under a shared service interface for planning, prompt generation, optimization, evaluation, and context control. On 200 HarmBench validation behaviors, AJAR improves X-Teaming from 65.0% to 76.0% attack success rate (ASR), reaches 80% cumulative success one turn earlier than the native implementation, and reproduces Crescendo more effectively than PyRIT (91.0% vs. 87.5% ASR). Behavior-level analysis shows that these gains are concentrated in hard categories and frequently depend on rollback-enabled transcript repair. We further show that tool access reshapes rather than uniformly enlarges the attack surface: ActorAttack rises from 51.0% to 56.0% ASR with tools, whereas Crescendo drops from 91.0% to 78.0% and X-Teaming from 76.0% to 55.5%, with the sharpest declines appearing in categories that rely on long semantic buildup. These results position AJAR as a practical foundation for evaluating multi-turn jailbreaks under realistic agent constraints. Code and data are available at https://github.com/douyipu/ajar.

What we could verify

These are the protocol signals we could actually recover from the available paper metadata. Use them to decide whether this paper is worth deeper reading.

Human Feedback Types

strong

Red Team

Directly usable for protocol triage.

"Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops."

Evaluation Modes

strong

Simulation Env

Includes extracted eval setup.

"Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops."

Quality Controls

missing

Not reported

No explicit QC controls found.

"Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops."

Benchmarks / Datasets

strong

Harmbench

Useful for quick benchmark comparison.

"On 200 HarmBench validation behaviors, AJAR improves X-Teaming from 65.0% to 76.0% attack success rate (ASR), reaches 80% cumulative success one turn earlier than the native implementation, and reproduces Crescendo more effectively than PyRIT (91.0% vs."

Reported Metrics

strong

Success rate, Jailbreak success rate

Useful for evaluation criteria comparison.

"On 200 HarmBench validation behaviors, AJAR improves X-Teaming from 65.0% to 76.0% attack success rate (ASR), reaches 80% cumulative success one turn earlier than the native implementation, and reproduces Crescendo more effectively than PyRIT (91.0% vs."

Benchmarks and datasets

Harmbench

Reported metrics

success ratejailbreak success rate
Human feedback details
Uses human feedback
Yes
Feedback types
Red Team
Rater population
Not reported
Expertise required
Coding
Evaluation details
Evaluation modes
Simulation Env
Agentic eval
None
Quality controls
Not reported
Evidence quality
High
Use this page as
Secondary protocol comparison source

Research brief

Metadata summary

Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops.

Based on abstract + metadata only. Check the source paper before making high-confidence protocol decisions.

Key takeaways

  • Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops.
  • Existing jailbreak frameworks still leave a gap between adaptive multi-turn attacks and agentic runtimes: attack algorithms are usually packaged as monolithic scripts, while agent harnesses rarely expose explicit abstractions for rollback, tool simulation, or strategy switching.
  • We present AJAR, a red-teaming framework that exposes multi-turn jailbreak algorithms as callable MCP services and lets an Auditor Agent orchestrate them inside a tool-aware runtime built on Petri.

Researcher actions

  • Compare this paper against nearby papers in the same arXiv category before using it for protocol decisions.
  • Validate inferred eval signals (Simulation environment) against the full paper.
  • Use related-paper links to find stronger protocol-specific references.

Caveats

  • Generated from abstract + metadata only; no PDF parsing.
  • Signals below are heuristic and may miss details reported outside the abstract.

Recommended queries

Contribution summary

  • Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops.
  • Existing jailbreak frameworks still leave a gap between adaptive multi-turn attacks and agentic runtimes: attack algorithms are usually packaged as monolithic scripts, while agent harnesses rarely expose explicit abstractions for rollback,…
  • We present AJAR, a red-teaming framework that exposes multi-turn jailbreak algorithms as callable MCP services and lets an Auditor Agent orchestrate them inside a tool-aware runtime built on Petri.

Why it matters for eval

  • Large language model (LLM) safety evaluation is moving from content moderation to action security as modern systems gain persistent state, tool access, and autonomous control loops.
  • We present AJAR, a red-teaming framework that exposes multi-turn jailbreak algorithms as callable MCP services and lets an Auditor Agent orchestrate them inside a tool-aware runtime built on Petri.

Researcher checklist

  • Human feedback protocol is explicit

    Detected: Red Team

  • Evaluation mode is explicit

    Detected: Simulation Env

  • Quality control reporting appears

    No calibration/adjudication/IAA control explicitly detected.

  • Benchmark or dataset anchors are present

    Detected: Harmbench

  • Metric reporting is present

    Detected: success rate, jailbreak success rate