Skip to content
WorakornKoonawootrittriron
Back to blog
Network security

AI model security by lifecycle stage: poisoning, pickles and theft

A five-stage checklist for protecting AI models, from data poisoning and code-running pickle files to model theft, based on NIST, NCSC/CISA, OWASP and ATLAS.

Advanced11 min read

On this page
  1. The map: which frameworks I used
  2. Stage 1: Data, where poisoning starts
  3. Stage 2: Training, where the crown jewels are made
  4. Stage 3: Packaging and the model supply chain
  5. Stage 4: Deployment, where the model meets strangers
  6. Stage 5: Monitoring and audit
  7. How this differs from LLM application risks
  8. A last field note

Most AI security conversations start and end with the prompt box. Prompt injection matters, but it is only the part of the system we can see. Behind the chat window is a model, and that model is several things at once: a dataset someone collected, a training job running on someone's infrastructure, a file of weights that gets copied around, and an API that answers anyone who is allowed to ask. Each of those can be attacked, and most of the defences are ones network and security people already know: integrity checks, access control, logging, and treating downloads as untrusted.

These are my field notes from reading the public guidance on this, arranged as a checklist by lifecycle stage. I've kept to what the sources actually say and linked every claim, so you can check it against the original.

The map: which frameworks I used

NIST AI RMF. The AI Risk Management Framework 1.0 was released on 26 January 2023 and is "intended for voluntary use". Its core has four functions: Govern, Map, Measure and Manage. Govern is described as a "cross-cutting function" that runs through the other three. In other words, the framework is less a checklist than a way to organise who owns which risk.

NIST AI 600-1, the Generative AI Profile. Published in July 2024, it lists 12 risks that are unique to or made worse by generative AI. Two of them matter most for this post:

  • Information Security, which includes a larger attack surface that "may compromise a system's availability or the confidentiality or integrity of training data, code, or model weights".
  • Value Chain and Component Integration, about third-party datasets, pre-trained models and libraries that may be "improperly obtained or not properly vetted".

The joint guidelines. In November 2023 the UK NCSC and CISA published the Guidelines for secure AI system development, which CISA describes as co-sealed by 23 cybersecurity organisations. They are organised into four areas: secure design, secure development, secure deployment, and secure operation and maintenance (PDF). In May 2025 the NSA's AI Security Center, CISA, the FBI and partners added a data-focused sheet, AI Data Security, which follows the six NIST AI RMF lifecycle stages, from Plan & Design to Operate & Monitor.

Attack catalogues. For naming attacks I used the OWASP Machine Learning Security Top Ten (its 2023 list; the project says the work "is in draft and is being modified frequently") and MITRE ATLAS, "a public knowledge base of adversary TTPs targeting AI systems".

For this post I've squeezed those into five stages: data, training, packaging, deployment and monitoring.

Five stages, one check per stage. The poisoned sample is stopped at the provenance check before it ever reaches training.

Stage 1: Data, where poisoning starts

A model learns its decision logic from data. The AI Data Security sheet puts it directly: an attacker who can manipulate the data can also manipulate the logic of the system. ATLAS lists this as technique AML.T0020, Training Data Poisoning: adversaries may add, remove or modify samples, alter labels, or manipulate feedback and data-collection processes. OWASP calls it ML02:2023 Data Poisoning Attack.

Two examples from the AI Data Security sheet show how cheap this can be:

  • Split-view poisoning. Curated web-scale datasets are often lists of links. When a domain on that list expires, whoever buys it controls what the link now serves. The sheet says enough control for effective poisoning can cost roughly 1,000 USD, and in some cases as little as 60 USD (it names COYO-700M).
  • Frontrunning poisoning. Wikipedia publishes snapshots twice a month at known times, so an attacker can time edits to land just before a snapshot, before anyone corrects them. The sheet cites industry analysis that as much as 6.5% of Wikipedia could be poisoned this way.

Checklist for the data stage (all from the same sheet):

  • Use data from authoritative sources where you can, and track provenance: where each record came from and the path it took.
  • Keep provenance in a database that is cryptographically signed and append-only, so no single person can change history without it showing.
  • Hash datasets and verify the hash at download time; throw away anything that fails.
  • Sign original data and every later revision, with the person who made the change signing it. The sheet recommends quantum-resistant signature standards.
  • For web-scale data, use only snapshots that a trusted party has verified.

Stage 2: Training, where the crown jewels are made

Training is where data turns into weights, so the training environment holds both. The NCSC guidelines ask you to apply access controls to your APIs, models and data, and to their training and processing pipelines, with segregation of environments that hold sensitive code or data. Same idea as a management network: the place where the valuable thing is built should not be reachable from everywhere.

The same guidelines widen the definition of an asset. Models, data (including user feedback), prompts, software, documentation, logs and assessments all count, and logs are to be treated as sensitive data. You should know where each asset lives and be able to restore to a known good state.

Fine-tuning a model you didn't train is its own risk. OWASP lists ML07 Transfer Learning Attack, where a model is trained on one task and fine-tuned on another to make it misbehave, and ML10 Model Poisoning, where the attacker manipulates the model's parameters directly. NIST AI 600-1 suggests that you verify fine-tuning does not compromise safety and security controls (action MS-2.7-008).

Checklist for the training stage:

  • Training pipelines, data stores and weights sit behind access control, in a segregated environment.
  • Every model, dataset and system prompt is documented: sources of training data, intended scope and limits, guardrails, hashes or signatures, retention time. The NCSC suggests model cards, data cards and SBOMs as useful formats.
  • Red-team the result. NIST AI 600-1's action MS-2.7-007 lists ML attacks to test against, including data poisoning, membership inference and model extraction.

Stage 3: Packaging and the model supply chain

This is the part that surprises people coming from traditional IT: a model file can be code.

Many model files are Python pickles. Hugging Face notes that pickle is the default format for PyTorch model weights. The Python documentation's warning is blunt: the pickle module "is not secure", and malicious pickle data can "execute arbitrary code during unpickling". The docs even show a tiny hand-made pickle whose payload calls os.system when it is loaded.

Why? A pickle is a small program of opcodes. Hugging Face explains that the dangerous ones are GLOBAL/STACK_GLOBAL, which tell the unpickler to import something, and REDUCE, which calls it with arguments. Import os.system, call it, and loading the "weights" runs a command.

Same file, two loaders. A plain pickle.load lets GLOBAL import a callable and REDUCE run it; a restricted loader refuses the import.

ATLAS has names for this. AML.T0010, AI Supply Chain Compromise covers hardware, data, the AI software stack or the model itself. AML.T0011.000, Unsafe AI Artifacts describes artifacts that cause harm when loaded and that "may continue to work as expected from the victim's perspective". That last part is the uncomfortable one: the model can behave normally while the payload has already run. OWASP's ML06 ML Supply Chain Attacks includes a scenario where an attacker impersonates an organisation's account on a model hub and publishes a malicious model that employees then download.

What helps:

Checklist for packaging:

  • Treat third-party models and serialised weights as untrusted third-party code. The NCSC asks for scanning and isolation/sandboxing when importing them.
  • Prefer safetensors; if you must load pickles, use a restricted loader and never turn weights_only off for files you didn't produce.
  • Compute and publish hashes and/or signatures of model files and datasets, including checkpoints, as soon as the model is trained, and keep the keys well managed.
  • Verify the integrity of packages in the ML stack and keep them up to date, as OWASP ML06 recommends, and track supply-chain attestations with a framework such as SLSA.

Stage 4: Deployment, where the model meets strangers

Once a model is served, attackers have two routes, and the NCSC describes both: reconstructing a model's functionality or training data either directly, by acquiring model weights, or indirectly, by querying it through an application.

Attack What it is Catalogue entry
Model theft The attacker gains access to the model's parameters OWASP ML05
Model extraction Repeated queries collect outputs to train a copy that mimics the target ATLAS AML.T0024.002
Model inversion Reverse-engineering the model to extract information from it OWASP ML03

The direct route is classic security: protect the weights like any secret, with access control and encryption. OWASP's ML05 entry lists encryption, access control, monitoring and auditing, among others.

The indirect route is about the query interface. The NCSC asks for controls on the query interface that detect and prevent attempts to access, modify and exfiltrate confidential information. For LLM APIs, OWASP's LLM10:2025 Unbounded Consumption gets specific: apply rate limiting and user quotas, and restrict or obfuscate logit_bias and logprobs in API responses. The same entry warns that attackers may collect enough API outputs to replicate a partial model or create a shadow model.

Checklist for deployment:

  • Weights are stored encrypted, with access limited to the serving system and a small set of people.
  • Every API caller is authenticated; rate limits and quotas are per caller, not just global.
  • Responses expose only what the use case needs (no raw probabilities unless they're required).
  • An incident plan exists for AI-specific cases. The NCSC expects responders trained for AI-related incidents and offline backups of critical resources.

Stage 5: Monitoring and audit

A deployed model changes even if no one touches it, because the world it sees changes. The AI Data Security sheet calls this data drift: the statistical properties of input data shifting away from the training data. The sheet calls the degradation this causes "a natural and expected occurrence", which is why models need regular updates. Its rule of thumb for telling drift from attack is useful: slow, gradual change is more likely drift; abrupt, dramatic change in one or more dimensions is more likely someone trying to compromise the model.

The NCSC guidelines ask for two kinds of monitoring (PDF):

  • Behaviour: measure outputs and performance so you can see sudden and gradual changes that affect security, and tell intrusions apart from natural drift.
  • Inputs: in line with privacy rules, log inference requests, queries or prompts for audit, investigation and remediation, including detection of out-of-distribution and adversarial inputs.

They also ask you to treat major updates to data, models or prompts like new versions, because changing any of them changes behaviour. That's change management, the same discipline as a firewall upgrade: version, test, and keep a way back.

Checklist for monitoring:

  • Baselines exist for accuracy, output distribution and query volume, with alerts on abrupt change.
  • Inference logs are kept, protected as sensitive data, and searchable during an incident.
  • Per-caller query patterns are reviewed for extraction-style behaviour (high volume, systematic inputs).
  • Model, data and prompt changes go through change control with a rollback path.
  • Policies and procedures exist for continuously monitoring third-party AI systems in deployment, as NIST AI 600-1 suggests in action GV-6.2-004.

How this differs from LLM application risks

Prompt injection and its relatives are covered by the OWASP Top 10 for LLM Applications (2025), which I walk through in a separate post on prompt injection. That list looks mostly at the application around the model: prompt injection, sensitive information disclosure, improper output handling, excessive agency, system prompt leakage and so on. The attacker talks to the model and tries to make the app misbehave.

This post is about the model as an asset: what it was built from, the file it lives in, and who can copy it. The two overlap on purpose. The LLM list includes LLM03 Supply Chain, LLM04 Data and Model Poisoning and LLM10 Unbounded Consumption, which are the same problems seen from the application side.

LLM application view Model lifecycle view
Main question Can someone make the app do the wrong thing? Can someone corrupt, replace or copy the model?
Typical attacks Prompt injection, excessive agency, output handling Data poisoning, unsafe model files, theft, extraction
Where controls live Prompt design, output validation, tool permissions Provenance, signing, safe formats, access control, monitoring

A secure AI system needs both views. A well-hardened prompt doesn't help if the weights came from a hijacked hub account, and a signed model doesn't help if the app gives it permission to delete things.

A last field note

What struck me most while reading these documents is how familiar the controls are. Hash what you download. Sign what you publish. Don't run code from strangers. Put the valuable thing behind access control and watch who touches it. The new part is noticing that a dataset, a weights file and an inference API are each things you download, publish, run and protect.

Sources

Keep reading

Related posts

Back to blog