Although LLM-based agents, powered by Large Language Models (LLMs), can use external tools and memory mechanisms to solve complex real-world tasks, they may also introduce critical security vulnerabilities. However, the existing literature does not comprehensively evaluate attacks and defenses against LLM-based agents.
To address this, we introduce Agent Security Bench (ASB), a comprehensive framework designed to formalize, benchmark, and evaluate the attacks and defenses of LLM-based agents, including 10 scenarios (e.g., e-commerce, autonomous driving, finance), 10 agents targeting the scenarios, over 400 tools, 27 different types of attack/defense methods, and 7 evaluation metrics.
Based on ASB, we benchmark 10 prompt injection attacks, a memory poisoning attack, a novel Plan-of-Thought backdoor attack, 4 mixed attacks, and 11 corresponding defenses across 13 LLM backbones.
Our benchmark results reveal critical vulnerabilities in different stages of agent operation, including system prompt, user prompt handling, tool usage, and memory retrieval, with the highest average attack success rate of 84.30%, but limited effectiveness shown in current defenses, unveiling important works to be done in terms of agent security for the community. We also introduce a new metric to evaluate the agents' capability to balance utility and security. Our code can be found at https://github.com/agiresearch/ASB.
💫 ASB is a comprehensive benchmarking framework designed to evaluate various adversarial attacks and defenses of LLM-based agents.
💫 Compared to other benchmarks, ASB's key advantages lie in its inclusion of multiple types of attacks and defense mechanisms across diverse scenarios.
💫 This not only allows the framework to test agents under more realistic conditions but also to cover a broader spectrum of vulnerabilities and protective strategies.
Five types of prompt injection types.
Defenses and the corresponding attacks they defend against.
IT Management: system_admin_agent is a system administrator responsible for managing and maintaining computer systems and networks, ensuring proper system operation and security, allocating re sources, managing user rights, performing system upgrades, and troubleshooting.
Investment: financial_analyst_agent is a financial analyst providing financial analysis and investment advice to clients, analyzing market trends, assessing investment risks and returns, and man aging investment portfolios.
Legal Advice: legal_consultant_agent is a legal advisor who provides legal counseling and advice to clients, drafting and reviewing legal documents to ensure legal compliance.
Medicine: medical_advisor_agent a medical consultant who provides medical advice and diagnosis to patients, develops treatment plans, and manages patient records.
Academic Advising: education_consultant_agent is an educational consultant who helps students choose courses and schools, assesses their performance, and provides academic counseling.
Counseling: psychological_counselor_agent is a counselor who provides psychological counseling and support to patients to help them deal with emotional problems and mental disorders.
E-commerce: ecommerce_manager_agent is an e-commerce manager responsible for managing and optimizing the e-commerce platform, managing the product catalog and inventory, and developing the marketing strategy.
Aerospace Design: aerospace_engineer_agent is an aerospace engineer responsible for the design and development of aerospace vehicles and systems, flight testing, and evaluation.
Research: academic_research_agent is an expert who is good at looking up and summarizing academic articles.
Autonomous Vehicles: autonomous_driving_agent is a self-driving technologist who monitors and controls the operation of self driving vehicles, optimizing self-driving algorithms and path planning.
We introduce the evaluation metrics in the figure above. Generally, a higher ASR indicates a more effective attack. After a defense, A lower ASR indicates a more effective defense. The refuse rate is measured to assess how agents recognize and reject unsafe user requests, ensuring safe and policy-compliant actions. Our benchmark includes both aggressive and non-aggressive tasks to evaluate this ability. Higher RR indicates more refusal of aggressive tasks by the agent. If BP is close to PNA, it indicates that the agent's actions for clean queries are unaffected by the attack. In addition, lower FPR and FNR indicate a more successful detection defense.
We have conducted extensive experiments on both attack and defense of agent, and the experiments are rich in results.
Here, we could give you an overview on our experiments. The experiments we conducted are as follows:
⚔️Agent Attack: We evaluated the agent attacks with 5 attack types on 13 LLM backbones.
🛡️Agent Defense: We evaluated the agent defenses against all four types of agent attacks.
💪LLM Capability vs ASR: We evaluated the correlation between backbone LLM leaderboard quality and average ASR across various attacks.
⚔️ 1) Mixed Attack is the Most Effective. 2) DPI is Widely Effective.
3) OPI Shows Moderate Effectiveness. 4) Memory Poisoning is the Least Effective.
5) PoT Backdoor Targets Advanced Models. 6) Partial Refusal of Aggressive Instructions.
🛡️Current prevention-based defenses are inadequate. (See Section 5.4)
🛡️Current prevention-based defenses are inadequate. (See Section 5.4)
⚔️PoT Backdoor Attacks are Effective across Different Triggers.
⚔️Unaffected Utility Performance for PoT Backdoored Agents.
⚔️Comparing Different Prompt Injection Ways.
⚔️Comparisons for Aggressive and Non-aggressive Tasks.
🛡️Slight Decline in Agent Performance from Defenses.
🛡️Ineffectiveness of Defenses for PoT Attack.
🛡️Ineffectiveness of Defenses Against Memory Attacks.
🛡️FPR vs. FNR curve for PPL detection in identifying memory poisoning attack. High perplexity indicates compromised content. The curve shows FNR and FPR variations across different thresholds. Shallower colors correspond to lower thresholds, while darker colors correspond to higher thresholds.
💪LLM Capability vs ASR.
📏We introduce ASB, a benchmark for evaluating the security of LLM agents under various attacks and defenses.
💥ASB reveals key vulnerabilities of LLM-based agents in every operational step.
🛡️ASB provides a crucial resource for developing stronger defenses and more resilient LLM agents.
💡In the future, we will focus on improving defenses and expanding attack scenarios.
@article{zhang2024agent,
title={Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents},
author={Zhang, Hanrong and Huang, Jingyuan and Mei, Kai and Yao, Yifei and Wang, Zhenting and Zhan, Chenlu and Wang, Hongwei and Zhang, Yongfeng},
journal={arXiv preprint arXiv:2410.02644},
year={2024}
}