Data Exfiltration Through Large Context Windows: A Hidden Vulnerability in Generative AI

There is a category of security risk in AI that most teams never think to defend against. It has nothing to do with hacking the model from the outside. It happens when you feed a model information it should never see, and the model finds a way to give that information to someone who should not have it.

Large context windows made this problem significantly worse. When a modern AI model can hold a million tokens of information in a single session, the question of what you put into that context becomes a serious security decision, not just a workflow choice.

What a Context Window Actually Holds

A context window is the total space an AI model uses to process everything in a session: system instructions, conversation history, documents you upload, retrieved data from connected knowledge bases, and your latest prompt. All of it sits in the same buffer.

Current models carry enormous amounts of information in that buffer. Leading models now offer context windows from 200,000 tokens to over one million in beta configurations. At one million tokens, a model can hold roughly 750,000 words in a single session, the equivalent of three to four full novels. When you feed that model a test log, a codebase, meeting notes, or a database extract, all of that content is processed as part of the same space with no separation between trusted instructions and the data you just loaded.

The fundamental security problem is that LLMs process everything in the context window as a unified stream. The model does not distinguish between your system prompt, your proprietary documents, and an injected instruction buried in a PDF you uploaded. It treats them with equal weight. This is the architecture that makes sensitive data exfiltration possible.

How Exfiltration Happens in Practice

The Samsung incident from March 2023 became the most widely cited real world example of this risk. Engineers at Samsung's semiconductor division pasted proprietary source code and internal meeting notes directly into ChatGPT to get help with debugging and meeting transcription. The AI Incident Database records that ChatGPT retained the inputted data, causing an inadvertent breach of confidentiality. Samsung responded by limiting prompt input to 1024 bytes as an emergency measure.

That incident was unintentional. Deliberate attacks look different and often more sophisticated.

Researchers at Harvard and Carnegie Mellon published a study in 2024 showing that adversaries can exploit instruction-following behavior in large language models to extract verbatim text from the private datastores of Retrieval Augmented Generation systems. Their attack worked by crafting queries that caused the model to repeat retrieved content instead of summarizing or analyzing it. They demonstrated this against multiple open source models including Llama 2 and Mistral, and found the exploitability increased as model size scaled up.

A second attack class exploits context overflow. Security researchers have documented that when an attacker sends extremely long prompts that exceed what a guardrail system can inspect, they can fragment malicious instructions across the overflow, with each individual segment appearing benign on its own. The guardrail inspects a window smaller than the full context the model processes, so the injected content passes inspection while remaining actionable to the model.

The OWASP LLM02:2025 Sensitive Information Disclosure guidance identifies three core scenarios that lead to data leakage at the context layer: PII leakage during normal interactions, proprietary algorithm exposure through poorly configured outputs, and sensitive business data appearing in generated responses. OWASP notes that these exposures can be unpredictable, with the model surfacing sensitive data in response to seemingly unrelated queries.

The Shared Infrastructure Problem

Context window exfiltration becomes substantially more dangerous in AI deployments that serve multiple tenants, where multiple users or organizations share the same model infrastructure.

Without proper session isolation, data from one user's context window can bleed into another user's session. A second user asking about a previous conversation, a related topic, or simply running a session that shares backend infrastructure may receive content that originated in a different user's context. OWASP LLM02:2025 explicitly lists data bleed across users as a common vulnerability pattern, where inadequate data sanitization causes one user to receive another user's personal data in a response.

This is not theoretical. RAG pipelines that pull documents into context without checking whether the requesting user is authorized to view those documents create exactly this exposure. The retrieval system may correctly authenticate the user at the query layer, then load documents the user should never see into the shared context.

Defenses That Address the Root Cause

Treating every piece of content as potentially sensitive before it enters the context window is the most effective starting point.

Sanitize before you load. Strip or mask credentials, personal identifiers, internal hostnames, and proprietary strings before any document or log goes into an AI context. This applies whether you are using a public service or a private deployment. The data that never enters the context cannot be extracted from it.

Apply access controls at the retrieval layer. In RAG systems, the check for whether a user should see a document must happen before that document is loaded into context, not after. A retrieval pipeline that fetches documents and then relies on the model to filter what it reveals is not a security control. The model may reveal it anyway.

Session isolation for systems that share infrastructure. Each user session should have a context window that cannot be read by another session. Shared memory architectures, persistent conversation logs, and agent state shared across sessions all create paths for data to move between contexts.

Audit what enters long-running agent contexts. Agentic AI systems that operate over multiple steps accumulate context over time. A document loaded in step one may still be accessible in step twenty. Audit logs should track what was loaded into context at each step, making it possible to investigate disclosure incidents after the fact.

Choose deployment environments based on data sensitivity. Content that includes real customer data, production credentials, or regulated information should not go into public AI services. Private cloud or on-premise deployments give organizations control over what is retained in logs and session storage. Public AI services process prompts on external infrastructure and retain data according to their own policies.

Sizing the Risk Against Real Data Volumes

The scale of modern context windows changes the risk calculation compared to earlier AI tools. At 200,000 tokens, which is a standard context size for current enterprise models, a single session can hold approximately 150,000 words. At one million tokens, that becomes 750,000 words. A full customer database extract, an entire codebase, or years of internal communications can fit within a single AI session.

When that volume of data enters a public context window, and an attacker has any ability to influence subsequent queries against that session, the extraction potential is enormous. Researchers studying inadvertent context leakage have demonstrated that even when a model correctly refuses direct extraction requests, the mere presence of secrets in the context window can create statistical correlations in the model's output that a sufficiently adaptive attacker can exploit.

The answer is not to avoid large context windows. They are genuinely useful. The answer is to treat every token you load into a context window as if it could appear in the response. If you would not want it quoted back by the model, it should not be in the context.

Teams building and working with AI tools need to understand these risks at a technical level, not just at a policy level. The AI security and test automation courses at RCV Academy walk through how these vulnerabilities manifest in real projects and what engineers actually do to contain them.