Is Vibe Coding Safe? Benchmarking Vulnerability of Agent-Generated Code in Real-World Tasks

Research Summary

Contribution Summary

• Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision.
• Although vibe coding is increasingly adopted, are its outputs really safe to deploy in production?
• To answer this question, we propose SU S VI B E S, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations.

Why It Matters For Eval

• Vibe coding is a new programming paradigm in which human engineers instruct large language model (LLM) agents to complete complex coding tasks with little supervision.
• To answer this question, we propose SU S VI B E S, a benchmark consisting of 200 feature-request software engineering tasks from real-world open-source projects, which, when given to human programmers, led to vulnerable implementations.

Related Papers

• TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models Embedding Neighbor
• KNIGHT: Knowledge Graph-Driven Multiple-Choice Question Generation with Adaptive Hardness Calibration Embedding Neighbor
• PonderLM: Pretraining Language Models to Ponder in Continuous Space Embedding Neighbor
• ZeroSyl: Simple Zero-Resource Syllable Tokenization for Spoken Language Modeling Embedding Neighbor
• MolReasoner: Toward Effective and Interpretable Reasoning for Molecular LLMs Embedding Neighbor
• Agentic Adversarial QA for Improving Domain-Specific LLMs Embedding Neighbor
• NoLan: Mitigating Object Hallucinations in Large Vision-Language Models via Dynamic Suppression of Language Priors Embedding Neighbor
• The Statistical Signature of LLMs Embedding Neighbor
• SocialHarmBench: Revealing LLM Vulnerabilities to Socially Harmful Requests Embedding Neighbor
• Inner Speech as Behavior Guides: Steerable Imitation of Diverse Behaviors for Human-AI coordination Tag Overlap
• Simple Yet Effective: Extracting Private Data Across Clients in Federated Fine-Tuning of Large Language Models Tag Overlap
• EARL: Entropy-Aware RL Alignment of LLMs for Reliable RTL Code Generation Tag Overlap