Pentest of AI/LLM Systems
Protect Your AI Solution and LLM-Based Applications
Where AI and LLM Systems Are Vulnerable
AI-based applications and large language models (LLMs) are being rapidly integrated into business-critical processes. Companies are using them to boost internal productivity, power customer-facing applications, and enable automated decision-making and agent-based workflows.
As an integral part of corporate infrastructure, AI and LLM applications are subject to the same security requirements as traditional IT systems and are increasingly subject to regulatory requirements as well. At the same time, they currently pose particularly high risks: They are often developed under high time pressure, their failure behavior is still relatively poorly understood due to their stochastic nature, and they are typically closely interlinked with sensitive data sources, tools, APIs, and the extended internal organizational infrastructure.
Due to their central role and high degree of autonomy, LLMs present a new target for attacks. Instead of exploiting only code errors or misconfigurations, attackers now target model behavior to exfiltrate sensitive data via retrieval mechanisms or abuse complex agent functions through context manipulation and prompt injections. These risks often cannot be reliably detected using traditional security analyses. For companies, this means that security can only be thoroughly analyzed using specialized testing approaches, such as our pentest of AI/LLM systems.
Common Vulnerabilities in AI/LLM Systems Include:
- AI agents or AI chatbots produce unwanted, regulatory or liability‑relevant outputs through attacker‑controlled malicious inputs ("jailbreaks").
- Exploitation of the "Lethal Trifecta": Exfiltration of sensitive data via prompt injections from retrieval‑augmented (RAG) data sources.
- Misuse of connected tools and APIs: Unauthorized actions in downstream systems, lateral movement within internal networks, and execution of arbitrary code outside secure sandboxes.
How Does usd AG Approach Penetration Testing of AI/LLM Systems?
Our tests are based on realistic attack scenarios within a specific context of use. This is because not every theoretical prompt injection automatically poses a relevant vulnerability.
Our methodology combines qualitative analyses from an attacker’s perspective with quantitative assessments. Building on our established methodical pentest approach, we conduct threat modeling and develop targeted, application-specific threat scenarios that reveal undesirable behavior and expose architectural and design weaknesses. Additionally, we address the stochastic behavior of AI/LLM systems by conducting multiple attacks under realistic conditions and measuring success rates and reproducibility. This results in robust risk metrics rather than one-off proof-of-concepts.
This assessment is based on established standards, including the OWASP Top 10 for LLM and agents, the MITRE ATLAS Framework, and the OWASP Vendor Evaluation Criteria for AI Red Teaming Providers.
Since LLM applications often build upon existing system landscapes, we combine our AI-specific assessments with web, mobile, or API penetration tests as needed. This allows us to address traditional vulnerabilities and security-relevant interfaces between LLM stacks and existing software.
What Checks Are Included in the Penetration Test of Your AI/LLM System?
These checks are included in the pentest of your AI/LLM system, among others:
- Taint analysis of the information flow throughout the entire system, with a focus on context poisoning, to detect direct and indirect prompt injections
- Assessment of data exfiltration possibilities
- Impact of attacker-induced malicious LLM outputs on downstream systems
- Analysis of potential tool calls for “excessive agency”
- Identification of LLM-based broken access control
- Assessing the effectiveness of alignment training for deployed models with regard to:
- Misinformation
- Unethical outputs
- Regulatory-relevant statements
- Liability-relevant statements
Tip: AI Security Training
Only those who understand the relevant threats posed by AI/LLM systems can implement effective protective measures. In our AI Security training course, we provide practical insights into real-world attack scenarios and demonstrate which measures are truly effective in a corporate environment. Upon request, we can tailor the content and technology stacks specifically to your organization and align the agenda with your requirements.
