Skip to content
OpenTrain AIFor AI Companies
← Back to explorer

Tag: Red Team

Red Team papers in the current HFEPX explorer (98 papers).

Papers in tag: 98

Global RSS

Running a Red Team study?

Post a Job →

Research Utility Snapshot

Evaluation Modes

  • Automatic Metrics (9)
  • Llm As Judge (3)
  • Human Eval (1)

Human Feedback Types

  • Red Team (20)
  • Pairwise Preference (1)

Required Expertise

  • General (12)
  • Coding (4)
  • Math (3)
Synthetic Persona Pretraining: Alignment from Token Zero

Julian Minder, Viktor Moskvoretskii, Raghav Singhal, Difan Jiao, Andy Arditi, Shaobo Cui · Aug 13, 2026 · Citations: 0

Red Team General
  • As language-model-based AI is increasingly deployed in autonomous settings, aligning its goals and values with those of humans becomes critical.
Reconcile Once, Write Anytime: A Trust-Tiered Librarian and a Multi-Agent Writer for Drift-Free, Point-in-Time Research

Xing Zhang, Yanwei Cui, Guanghui Wang, Peiyang He · Aug 13, 2026 · Citations: 0

Red Team Automatic Metrics General
  • We present a two-tier agentic system that separates a maintained, point-in-time knowledge library from report writing.
  • A portable multi-agent "writer" runtime then composes a contradiction-free, evidence-grounded report at any knowledge cutoff T, reading only evidence with as_of <= T (no look-ahead); red-team verdicts flow back into the librarian.
Online Safety Monitoring for LLMs

Mona Schirmer, Metod Jazbec, Alexander Timans, Christian Naesseth, Maja Waldron, Eric Nalisnick · Jul 2, 2026 · Citations: 0

Red Team Math
  • Monitoring outputs online and raising an alarm when safety can no longer be assumed is therefore critical.
HaloGuard 1.0: An Open Weights Constitutional Classifier for Multilingual AI Safety

Navaneeth Sangameswaran, Preetham S, Ashmiya Lenin · Jul 2, 2026 · Citations: 0

Red Team Automatic Metrics Multilingual
  • We present HaloGuard 1.0, an open-weights implementation of the constitutional-classifier paradigm for input safety.
  • Across seven prompt-safety benchmarks, HaloGuard 1.0-0.8B attains the best average F1 (90.9) of any open guard we evaluate, outperforming baselines up to 27B parameters (over 30 times larger) while holding false-positive rate (FPR) to 4.3…
Safety Targeted Embedding Exploit via Refinement

Joshua Adrian Cahyono · Jul 2, 2026 · Citations: 0

Red Team Automatic Metrics CodingMultilingual
  • We show that this creates an epistemic gap in which models confidently generate harmful responses for inputs that fall outside the distribution of their safety training.
  • To study this phenomenon, we introduce STEER (Safety Targeted Embedding Exploit via Refinement), a gradient-guided attack that identifies words contributing most strongly to the model's refusal behavior and iteratively translates them into…
EvalSafetyGap: A Hybrid Survey and Conceptual Framework for LLM Evaluation-Safety Failures

Buğra Alperen Uluırmak, Rifat Kurban · Jun 29, 2026 · Citations: 0

Red Team Llm As Judge Law
  • LLM evaluation and AI safety face a shared measurement problem: benchmark scores, reward-model signals, and reported safety metrics can improve while the latent properties they are meant to represent remain difficult to verify.
  • We introduce EvalSafetyGap as an organizing hypothesis for comparing evaluation-side and alignment-side proxy failures under optimization pressure, using Goodhart's Law together with two constructs we develop here - an Instability…
Do Encoders Suffice? A Systematic Comparison of Encoder and Decoder Safety Judges for LLM Adversarial Evaluation

Han Jeon, Shiv Medler, Joseph Voyles, Matt Wood · Jun 24, 2026 · Citations: 0

Red Team Llm As JudgeAutomatic Metrics General
  • Safety evaluation of LLM outputs has generally relied on LLM-based judges, which can be effective but are often slow and expensive to deploy at scale.
  • In this paper, we evaluate whether fine-tuned modern encoder classifiers from the ModernBERT family, including ModernBERT and Ettin, can reliably identify harmful LLM outputs in user-model conversations without substantial performance loss…
RAS: Measuring LLM Safety Through Refusal Alignment

Chang-Chieh Huang, Yan-Lun Chen, Chia-Mu Yu, Wei-Bin Lee · Jun 24, 2026 · Citations: 0

Red Team Automatic Metrics General
  • Safety evaluation of large language models (LLMs) is commonly performed by querying models with unsafe or jailbreak prompts and judging whether their outputs violate a safety policy.
  • We propose **SafeVec**, a white-box evaluation procedure that measures safety from internal representations rather than generated answers.
How Reliable Is Your Jailbreak Judge? Calibration and Adversarial Robustness of Automated ASR Scoring

Yang Gao · Jun 24, 2026 · Citations: 0

Red Team Llm As Judge Coding
  • Almost every paper on LLM jailbreaks and prompt injection reports an attack-success rate (ASR), and that number is assigned not by people but by an automated judge: either a safety classifier trained for the task, or a general chat model…
  • Wrappers that leave the harmful text untouched and only add benign framing flip every LLM-judge between 57% and 100% of the time, and a single prepended refusal sentence accounts for much of this (39% to 88%).
A Red Teaming Framework for Large Language Models: A Case Study on Faithfulness Evaluation

Abrar Alotaibi, Raed Mughus, Moataz Ahmed · Jun 24, 2026 · Citations: 0

Red Team Automatic Metrics General
  • Large language models (LLMs) have demonstrated remarkable performance across natural language processing tasks, yet their deployment in high-stakes applications raises critical concerns regarding reliability, safety, and trustworthiness.
  • The approach identifies how structural constraints in summarization can shape vulnerability patterns, with format limitations yielding measurable gains in faithfulness, and shows that architectural design choices typically outweigh…
What Intermediate Layers Know: Detecting Jailbreaks from Entropy Dynamics

Sofiia Nikolenko, Michele Papucci, Mina Rezaei, Shireen Kudukkil Manchingal · Jun 23, 2026 · Citations: 0

Red Team General
  • Jailbreak attacks reveal a persistent weakness in aligned Large Language Models: carefully crafted prompts can elicit policy-violating responses despite safety training.
  • Across multiple models (Llama, Qwen, Gemma) and adversarial benchmarks, these entropy dynamics provide architecture-consistent separation without additional training.
AdversaBench: Automated LLM Red-Teaming with Multi-Judge Confirmation and Cross-Model Transferability

Khanak Khandelwal · Jun 23, 2026 · Citations: 0

Pairwise PreferenceRed Team Automatic Metrics Coding
  • We present AdversaBench, an end-to-end red-teaming pipeline that mutates seed prompts with five structured operators, queries a target model, and confirms failures through a three-judge panel with a meta-judge tiebreaker.
  • Third, pairwise judge agreement of 80-87% coexists with near-zero Cohen's kappa due to label skew; category-level disagreement rates are more informative.
Audio Jailbreaks in Large Audio-Language Models: Taxonomy, Attack-Defense Analysis, and Cost-Aware Evaluation

Bo-Han Feng, Yu-Hsuan Li Liang, Chien-Feng Liu, You-Hsuan Chang, Yun-Nung Chen · May 28, 2026 · Citations: 0

Red Team Automatic Metrics General
  • Existing work studies these risks under heterogeneous threat models and evaluation protocols, making it difficult to compare attack practicality or defense utility.
  • This paper provides a unified taxonomy and a controlled empirical evaluation of LALM jailbreak attacks and defenses.
Harder to Defend: Towards Chinese Toxicity Attacks via Implicit Enhancement and Obfuscation Rewriting

Jingyi Kang, Junyu Lu, Bo Xu, Hongbo Wang, Linlin zong, Roy Ka-Wei Lee · May 21, 2026 · Citations: 0

Red Team Human Eval General
  • We introduce Chinese Implicit Toxicity Attack (CITA), a controlled red-team evaluation and defense-data generation framework, not a deployable evasion tool.
  • On CITA-generated evaluation samples, the seven tested detectors exhibit substantial missed-detection risks, reaching an average ASR of 69.48%; human evaluation further confirms preserved harmfulness and increased implicitness/evasiveness.