The AI Supply Chain Threat: Malicious Models in Public Repositories

In February 2024, security researchers at JFrog published findings that transformed how we should think about AI supply chains. Their research identified approximately 100 malicious models on Hugging Face—the largest public repository of machine learning models, hosting over 1 million models as of 2024.

What Researchers Found

The JFrog research team discovered models containing various forms of malicious code:

• Code execution via pickle files — Python's pickle serialization format, commonly used to save model weights, can execute arbitrary code during deserialization. Attackers embedded reverse shells, data exfiltration code, and other malicious payloads.
• Hidden functionality in model architecture — Some models contained deliberately backdoored neural network layers that activated under specific input conditions.
• Credential harvesting — Models designed to capture and exfiltrate API keys, tokens, and other credentials from the development environment.

The most concerning aspect: several of these malicious models had thousands of downloads before being detected and removed.

The Pickle Problem

At the core of this vulnerability is Python's pickle serialization format. When you "unpickle" a file, Python executes the instructions to reconstruct the serialized object—and those instructions can include arbitrary code execution.

# This looks innocent
import pickle

# But loading an untrusted pickle file can execute anything
model = pickle.load(open("malicious_model.pkl", "rb"))

# Attacker's code runs during deserialization - before you
# ever get the model object back

This isn't a bug—it's how pickle works by design. The Python documentation explicitly warns: "Warning: The pickle module is not secure. Only unpickle data you trust." Yet the ML ecosystem relies heavily on pickle-based formats for model storage and distribution.

Hugging Face's Response

To their credit, Hugging Face has implemented multiple security measures:

• Malware scanning — Automated scanning of uploaded models for known malicious patterns
• Pickle scanning — Specific detection of dangerous operations in pickle files
• SafeTensors format — A safer serialization format that doesn't support arbitrary code execution
• Secret scanning — Detection of exposed API keys and credentials in model files

These measures have caught and removed thousands of problematic uploads. But the cat-and-mouse nature of security means new evasion techniques continue to emerge. As of late 2024, researchers continue to find ways to bypass detection.

Beyond Hugging Face

While Hugging Face receives the most attention due to its size, this isn't a platform-specific problem. Any source of pre-trained models carries supply chain risk:

• Model zoos and registries (TensorFlow Hub, PyTorch Hub, ONNX Model Zoo)
• GitHub repositories containing model weights
• Direct downloads from research paper links
• Shared drives and internal model repositories
• Third-party fine-tuned versions of foundation models

Defense Recommendations

Organizations should treat model downloads with the same rigor as third-party code dependencies:

Implications for Standards

This threat vector illustrates why AI Security requires its own frameworks. Traditional software security addresses code dependencies, but model supply chain security involves:

• Unique serialization risks (pickle, joblib, torch.save)
• Embedded code in model architecture (custom layers, callbacks)
• Behavioral backdoors that survive fine-tuning
• Provenance verification for model weights
• Model integrity verification during deployment

These concerns are part of what AISF will address in the AI Security Controls Matrix—providing organizations with concrete controls for model supply chain security alongside other AI-specific security domains.

Sources & Further Reading