Your developers are already running AI locally: Why on-device inference is the CISO’s new blind spot

For the last 18 months, the CISO playbook for generative AI has been relatively simple: Control the browser.

Security teams tightened cloud access security broker (CASB) policies, blocked or monitored traffic to well-known AI endpoints, and routed usage through sanctioned gateways. The operating model was clear: If sensitive data leaves the network for an external API call, we can observe it, log it, and stop it. But that model is starting to break.

A quiet hardware shift is pushing large language model (LLM) usage off the network and onto the endpoint. Call it Shadow AI 2.0, or the “bring your own model” (BYOM) era: Employees running capable models locally on laptops, offline, with no API calls and no obvious network signature. The governance conversation is still framed as “data exfiltration to the cloud,” but the more immediate enterprise risk is increasingly “unvetted inference inside the device."

When inference happens locally, traditional data loss prevention (DLP) doesn’t see the interaction. And when security can’t see it, it can’t manage it.

Why local inference is suddenly practical

Two years ago, running a useful LLM on a work laptop was a niche stunt. Today, it’s routine for technical teams. Three things converged:

• Consumer-grade accelerators got serious: A MacBook Pro with 64GB unified memory can often run quantized 70B-class models at usable speeds. What once required multi-GPU servers is now feasible on a high-end laptop.
• Quantization went mainstream: It’s now easy to compress models into smaller, faster formats that fit within laptop memory with acceptable quality tradeoffs.
• Distribution is frictionless: Open-weight models are a single command away, and the tooling ecosystem makes “download → run → chat” trivial.

The result: An engineer can pull down a multi‑GB model artifact, turn off Wi‑Fi, and run sensitive workflows locally—source code review, document summarization, or exploratory analysis. No outbound packets, no proxy logs, no cloud audit trail.

The risk isn’t only data leaving the company anymore

If the data isn’t leaving the laptop, why should a CISO care? Because the dominant risks shift from exfiltration to integrity, provenance, and compliance. Local inference creates three classes of blind spots:

1. Code and decision contamination (integrity risk): Local models are often unvetted. A developer might use a community-tuned model that subtly degrades security posture by suggesting unsafe defaults or weak validation.
2. Licensing and IP exposure (compliance risk): Many models have non-commercial licenses. Local usage bypasses legal review, potentially inheriting risks that show up during M&A or litigation.
3. Model supply chain exposure (provenance risk): Older Pickle-based PyTorch files can execute malicious payloads when loaded. Without a software bill of materials (SBOM) for models, developers might unknowingly download exploits.

Mitigating BYOM: treat model weights like software artifacts

You can’t solve local inference by blocking URLs. You need endpoint-aware controls:

• Move governance to the endpoint: Scan for indicators like .gguf files, processes like Ollama, and high GPU utilization from unapproved runtimes.
• Provide a paved road: Offer a curated internal catalog of approved models with verified licenses and pinned hashes.
• Update policy language: Ensure acceptable use policies explicitly cover downloading and running model artifacts on corporate endpoints.