Tested how easily LLMs leak sensitive data through tool calls - here’s what happened

The LLM Agent Security Testbed is an empirical framework that tests whether LLM-powered agents connected to backend tools can be coerced into unauthorized data exfiltration via prompt injection and related attacks. It contrasts a naive backend that returns all requested data without validation against a hardened backend that enforces access control and redacts sensitive fields. The testbed executes five attack categories against both backends using the same LLM model. Results show that while direct prompt attacks are blocked by the model's safety mechanisms, indirect or subtle prompt injections can cause naive backends to leak sensitive information such as admin passwords. The core vulnerability is the trust boundary between the LLM's intent and the backend execution, analogous to SQL injection vulnerabilities caused by unvalidated inputs. The hardened backend successfully blocks or sanitizes all tested attacks. The project provides detailed attack taxonomies, mock backend implementations, grading engines, and comprehensive reports to aid understanding and defense against these flaws.

Sensitive data such as admin passwords can be leaked if an LLM agent is connected to a backend that blindly trusts the model's tool call arguments without validation. Direct blunt attacks are typically blocked by the LLM's safety guardrails, but more subtle or indirect prompt injections can bypass naive backend defenses, resulting in unauthorized data disclosure. This represents a confused-deputy style vulnerability where the backend executes privileged actions based on untrusted LLM input. The hardened backend with access control and field redaction prevents these leaks, demonstrating that the vulnerability is primarily in the backend trust boundary rather than the LLM itself.

The vendor advisory indicates that the vulnerability is mitigated by enforcing access control and sanitizing sensitive fields at the backend boundary. Prompt alignment and LLM safety guardrails alone are insufficient. Implementing a hardened backend that validates all tool call arguments, refuses restricted queries, and strips sensitive data such as passwords effectively prevents data leakage. Users should adopt such backend validation and sanitization mechanisms when integrating LLM agents with privileged tool calls. No patch or fix is applicable to the LLM model itself; the defense is architectural and procedural at the application boundary.