Sztuczna inteligencja (AI)
This Is How Your LLM Gets Compromised
Poisoned data. Malicious LoRAs. Trojan model files. AI attacks are stealthier than ever—often invisible until it’s too late. Here’s how to catch them before they catch you.
Save to Folio
This Is How Your LLM Gets Compromised
Plainly speaking, Artificial intelligence is no longer a fringe technology. It has become a core component of modern business, from customer service chatbots to complex data analysis. We often treat the Large Language Models (LLMs) that are at the core of this technology as trusted black boxes. But like any software, they can be tampered with, manipulated, and turned against their creators. Understanding the ways an AI model can be compromised is the first step toward building a secure and resilient AI infrastructure.
There are three primary ways that I want to cover, in which I will explain how an AI model can be compromised and made to act in unpredictable (or malicious!) ways:
- Embedding malicious executable instructions in a model’s file.
- Retraining the model with poisoned data.
- Using an “adapter” (LoRA) to manipulate the way the model behaves.
Supply Chain Attacks: The Trojan Horse
The AI community thrives on collaboration, with platforms like Hugging Face making it easy to download and build upon powerful pre-trained models. This open ecosystem, however, creates a significant new attack surface. An adversary doesn't need to build a malicious model from scratch; they only need to trick you into using their compromised version.
Payloads in Model Files
To understand this threat, it helps to know what an AI model file actually is. It’s not just code; it’s a data file containing the model's "brain" - a complex web of millions of numbers (you might have heard them being called "parameters" or "weights") organized into structures called tensors. To save and share this "brain," it must be packaged into a single file through a process called serialization. When another computer uses the model, it un-packages it (deserializes it). Think about it like being zipped, or turned into a batched file.
The danger lies in how this packaging happens. Older formats like Python's pickle were designed to package not just data, but also executable instructions. This flexibility creates a massive security hole. A malicious actor can hide harmful code within the model file. When an unsuspecting user loads the model, their computer "un-packages" not just the AI's brain, but also the hidden instructions, which could be anything from "steal all passwords" to "install ransomware" - It's the digital equivalent of a Trojan horse. A good reason why you should always be wary when loading pickle tensor files.
While safer formats like safetensors have been developed to mitigate this specific risk, the danger of compromised model files remains a fundamental concern.
Malicious Adapters: The LoRA Threat
To make models more versatile, developers often need to tweak their behavior for specific tasks. The old way was to retrain the entire model, which is like rebuilding a professional camera from scratch - incredibly expensive and time-consuming. A newer, more efficient method is Low-Rank Adaptation, or LoRA.
Think of a base AI model as a high-end digital camera. The camera itself is a complex, powerful piece of equipment. A LoRA file is like adding a special filter to the camera's lens. The camera's core mechanics remain untouched, but by adding a small, lightweight filter, you can instantly change how it captures images - a polarizing filter can make skies look more dramatic, or a soft-focus filter can be used for portraits. The filter is tiny and inexpensive compared to the camera, and you can easily swap it for another. The LoRA adapter does the same for an AI, altering its output with a file that is often less than 1% of the original model's size.
This creates a new supply chain problem. A malicious actor can distribute a seemingly benign and helpful LoRA (the lens filter) that promise enhancing the capabilities of a model in some way or another, but when applied to a trusted base model (the camera), injects hidden backdoors, introduces dangerous biases, or creates triggers for data exfiltration. Traditional security checks are useless here because the base model remains untouched and appears perfectly safe. The purported 'benefit' of the LoRA might be actually a thing - riding along the harmful bits. That malicious logic only activates when the small, easily-overlooked adapter is applied. Detecting these unauthorized modifications requires new, specialized tools that can analyze a model's structure and configuration for the tell-tale signs of tampering.
Data Poisoning: Corrupting the Source of Truth
An AI model is a reflection of the data it was trained on. If an attacker can manipulate that data, they can fundamentally warp the model's behavior in ways that are incredibly difficult to detect.
Backdoor Attacks
In a backdoor attack, the adversary injects a small amount of poisoned data containing a specific trigger. The model learns to associate this trigger with a malicious action. For example:
An image recognition model for an access control system could be poisoned with images of random individuals that all contain a specific, almost invisible watermark. The model learns that anyone with this watermark should be granted access.
A language model could be backdoored to produce harmful content or leak confidential information whenever it encounters a specific, unusual phrase.
The model behaves normally in all other circumstances, making the backdoor dormant and nearly impossible to find through standard testing. It only activates when the secret trigger is presented.
Direct Model Compromise: Unauthorized Retraining
This approach is more brute-force than using an adapter. If an attacker gains access to your trained model, they can deliberately retrain or fine-tune it for their own purposes by directly altering the core weights of the model file itself.
For example, an attacker could take a customer service chatbot and retrain it to subtly steer customers toward competitor products or to phish for sensitive financial information. Because the retrained model is a direct evolution of the original, its behavior might appear plausible at first glance, making the manipulation hard to spot until significant damage has been done. This highlights the importance of strict access controls and integrity monitoring for trained model artifacts, such as checking the file's hash to detect unauthorized changes.
Conclusion: Toward a Secure AI Lifecycle
The vulnerabilities in AI models are not theoretical; they are active and evolving threats. From malicious code hidden in model files to subtle backdoors created by poisoned data, the attack surface is broad and requires a multi-faceted defense.
Understanding the differences between these attacks is key. Both embedded tensor file payloads and unauthorized retraining involve direct tampering with the core model file. This makes them detectable with file integrity monitoring (like checking a file's hash) if a trusted baseline exists. However, both can be deployed without any changes to the application's code, executing their malicious logic the moment the file is loaded. In contrast, a LoRA attack is more insidious from a file-checking perspective, as it leaves the base model untouched, rendering file-hash checks useless. The trade-off is that it often requires a visible code change in the application to load the malicious adapter, providing a different kind of audit trail.
Securing the AI supply chain is no longer optional. Organizations must move beyond trusting models based on their source and adopt a "verify, then trust" approach. This includes:
- Static Analysis: Scanning model files and configurations for signs of tampering or the presence of unexpected adapters.
- Data Integrity: Implementing rigorous data validation and sanitization pipelines to defend against poisoning attacks.
- Access Control & Monitoring: Treating trained models as critical intellectual property, with strict access controls and continuous monitoring to detect unauthorized changes or anomalous behavior.
To ensure safe and responsible use of AI, it is critical to treat AI security with the same rigor and discipline as traditional cybersecurity in any other aspect.