Related Big Read: The outbound channel fallacy: why AI egress is the governance problem your framework has not named
Some data loss can be contained.
Credentials can be rotated. Files can be removed. Access can be revoked. Notifications can be issued, damage might be still serious, however, the response followsa fairly standard process
AI ingestion is different.
Once sensitive data has been absorbed into a model, fine-tuning set, retrieval layer, prompt history, or opaque vendor pipeline, the organisation may not be able to recall it, audit it properly, or prove it has been forgotten.
That’s the irreversibility trap.
How is this being monetised by attackers?
The LunaLock ransomware story was important because it changed the operational shape of extortion.
The threat was to feed stolen material into AI training data. That matters because training ingestion carries a different kind of harm. Publication can be seen. Copies can sometimes be traced. But once data is used to train or enrich a model, the loss becomes harder to scope and harder to reverse.
[[CODEX: add find and add internal hyperlink Anthropic’s copyright settlement makes the same point from a different angle. ]] Training data provenance is no longer an abstract debate about ethics or licensing. It has become a financial, legal, and operational risk. If a provider cannot prove where training data came from and what rights attach to it, the customer may inherit a risk it cannot see.
Then there is the execution layer.
PromptLock and Hexstrike-AI point toward a separate but related control failure. AI-enabled tools can assemble behaviour dynamically at runtime. That makes static review less useful. Security teams can’t rely only on recognising known code or known patterns. They need to know what the system is technically allowed to do once it starts acting.
Together, these stories change the executive question.
It is no longer enough to ask whether data left the organisation. Leaders have to ask what kind of boundary it crossed.
Why the traditional data loss mindset is no longer enough
It assumes the organisation can identify the affected records. It assumes the records exist as records. It assumes access can be stopped. It assumes remediation can be described in terms of containment, notification, recovery, and monitoring.
AI ingestion weakens each assumption.
If a confidential document is pasted into a public AI tool, used in a fine-tuning workflow, embedded in a retrieval index, or passed into a third-party model pipeline, the organisation may not know where that information now sits. It may not know whether it is retained. It may not know whether it will be used to improve a model. It may not know whether a deletion request actually deletes the operational trace.
That makes it a control problem, not only a privacy problem.
The organisation has lost the ability to demonstrate what happened to its own information.
The assurance gap
This month has also seen buyers and regulators start asking for proof rather than accepting claims.
Switzerland’s Apertus model [Codex insert link] mattered because its openness put transparency d into the centre of the architecture. Microsoft’s MAI initiative mattered because it suggested that even large technology providers were uncomfortable depending entirely on external intelligence infrastructure. The UK Trusted Third-Party AI Assurance Roadmap mattered because it showed that AI trust was starting to move from vendor claim to inspectable assurance.
Leaders need to starting to ask for proof before trust.
That proof has to cover more than model quality. It has to cover data provenance, contractual rights, retention posture, customer-data separation, logging, model update practices, incident notification, and the technical boundary between low-risk use and sensitive workflows.
The vendor that cannot explain its data boundary is asking the customer to accept a risk the customer cannot govern.
Where model poisoning fits in
Model poisoning belongs inside the same control frame.
If the organisation cannot verify the integrity and provenance of the data entering an AI system, it faces two risks at once. Sensitive data may flow out into a model or vendor pipeline. Corrupted, malicious, or untrusted data may flow in and shape model behaviour.
The direction differs; the control question is the same.
Can we prove what entered the system, where it came from, whether it was permitted, whether it was intact, and what authority it has inside the workflow?
What leaders should be asking
Leaders don’t need to review every AI architecture diagram.
They do, however, need to know whether the organisation can answer a few basic questions before sensitive data enters an AI system:
- What data categories are prohibited from entering public or external AI tools?
- Can you technically enforce that prohibition?
- Do your vendors use customer inputs for training, improvement, evaluation, or retention?
- Can your vendors prove training data provenance and rights?
- Can you see AI-specific egress, including prompt content, retrieval context, API calls, and fine-tuning submissions?
- If sensitive data has already been ingested, do we have an incident classification that recognises the difference between copying and irreversible absorption?
If you cannot answer clearly, do not treat the risk as accepted.
You have not understood the risk well enough to accept it.
High-level checklist: proof before ingestion
Use these six questions before sensitive data enters an AI system.
If the answers to any of the questions are unknown then take a minute to think about the potential implications of this data being ingested into a model and how you’ll resolve the issue.
Data boundary
What data must not enter an external AI system?
- Define prohibited data categories, including prompts, uploaded files, retrieval context, fine-tuning data, logs, and metadata.
- Confirm whether the boundary is technically enforced or only written into policy.
- Name the owner responsible for maintaining the boundary.
Evidence: The data classification, policy rule, technical enforcement point, and named owner.
Provenance
Can the provider explain where its training and operational data came from?
- Ask for the origin of training data and the rights or licences that matter to the use case.
- Check whether legal or procurement has reviewed the provenance position.
- Record exclusions, limitations, and unresolved claims.
Evidence: Provider documentation, contractual terms, provenance statements, and review records.
Vendor use of inputs
What happens to prompts, files, outputs, embeddings, logs, and telemetry?
- Identify whether customer inputs are used for training, improvement, evaluation, or retention.
- Confirm that any opt-out is contractual, technical, and auditable.
- Require a plain-language explanation of retention and deletion.
Evidence: Contract and DPA terms, product settings, retention schedules, deletion evidence, and vendor responses.
Integrity before use
Has incoming data been verified before it enters training, tuning, retrieval, or evaluation?
- Trace input data to its source before use.
- Scan, test, quarantine, or otherwise validate external and third-party data.
- Assign an owner for approving untrusted data sources.
Evidence: A source register, validation record, quarantine process, approval owner, and exception log.
Runtime containment
What can the AI system do once it starts acting?
- Map file access, network calls, code execution, database queries, and external API calls.
- Constrain permissions to the minimum required for the workflow.
- Block, sandbox, or require human approval for destructive or irreversible actions.
Evidence: A permissions map, runtime policy, confirmation gate, revocation path, and test evidence.
Incident readiness
Can the organisation respond if ingestion cannot be reversed?
- Maintain a playbook for suspected AI training or retrieval ingestion of sensitive data.
- Define when the event becomes notifiable or board-reportable.
- Test the response with legal, privacy, procurement, security, and the business owner.
Evidence: The incident playbook, decision thresholds, notification criteria, exercise output, and named responders.
Decision rule
If the organisation cannot prove provenance, permission, visibility, containment, and recoverability, the workflow should not proceed with sensitive data.
If the organisation cannot prove provenance, permission, visibility, containment, and recoverability, the workflow should not proceed with sensitive data.
That is the minimum position when the loss may not be reversible.
