What actually breaks when you take AI fully offline

Air-gapped deployment of AI infrastructure sounds simple in a meeting. Cut the network connection, run everything on-prem, done. The reality is that modern AI tooling is deeply assumption-laden about network access. The assumptions are subtle, they are scattered across multiple layers of the stack, and almost none of them announce themselves loudly. They fail silently, fail slowly, or fail at the worst possible moment — usually after you have already deployed to the air-gapped environment.

This post goes through every class of outbound network call that a typical AI stack makes, why it exists, what breaks when it can’t complete, and how to design for zero outbound traffic from the start.

The assumption of connectivity in AI tooling

LLM tooling was designed for cloud-native environments. The original use case — calling an API over the internet — assumed connectivity by definition. As the ecosystem has expanded into self-hosted models, the tooling has not always shed those connectivity assumptions. Libraries download models on first use, tokenizers fetch vocabulary files, telemetry data is flushed to remote endpoints, and license validators phone home.

This is not malicious. It is the natural consequence of building developer tools for the majority use case (internet-connected dev environments) without designing explicitly for the minority case (air-gapped deployment). For teams in regulated industries — defense, healthcare data processing, financial infrastructure — the minority case is the only acceptable case.

Understanding what calls exist is the first step to eliminating them.

Model weight distribution

The most obvious challenge, and usually the one teams solve first. Transformer models are large — 7B parameter models run 14–30GB in typical quantization schemes, and production-capable models at 70B run 40–80GB. Getting these weights into an air-gapped environment requires a deliberate distribution mechanism.

Common approaches:

Sneakernet — physically transport weights on hard drives or tapes — works but is operationally painful and creates version tracking headaches. Did the drive you loaded last Tuesday contain the same weights as what’s running in the other data center?

Artifact registry synchronization — use a tool like a container registry or an artifact manager (Harbor, Artifactory, Nexus) to hold model weights as versioned artifacts. Synchronize from an internet-connected staging environment to the air-gapped registry on a controlled schedule. The air-gapped environment pulls from the internal registry, never from the internet.

Container-bundled weights — for smaller models (1B–7B range), bundle weights into a container image layer and distribute via your existing container registry. This works at small scale but becomes unwieldy above 20GB per model.

For Kubernetes deployments, the practical approach is a combination: a private OCI-compatible registry inside the air-gapped boundary that serves model artifacts, populated by a synchronization pipeline that runs in a DMZ or a controlled internet-connected environment and pushes artifacts inbound after verification.

The version tracking question is load-bearing: you need to know exactly which weights are running in each environment, when they were updated, and what provenance they have. Without this, you cannot reason about model behavior or reproduce inference results for audit purposes.

Tokenizer and vocabulary file downloads

This is the failure mode that catches teams most often. Tokenizer files (the vocabulary, merges, and special token definitions that convert text to token IDs and back) are typically fetched on first use from Hugging Face’s model hub or a similar registry. Frameworks like Hugging Face Transformers, tiktoken (OpenAI’s tokenizer library), and LlamaIndex tokenizers make this fetch transparently, with no warning or configuration option that announces the behavior.

In an internet-connected environment, the tokenizer downloads on first call and is cached locally. In an air-gapped environment, the first call fails with a connection error. If the application does not handle the error explicitly, it may silently fall back to a default tokenizer — which produces incorrect token counts, breaks context window management, and causes subtle downstream failures that are hard to diagnose.

The fix:

Pre-populate the tokenizer cache before moving to the air-gapped environment. For Hugging Face-based tokenizers, set TRANSFORMERS_OFFLINE=1 and HF_DATASETS_OFFLINE=1 environment variables — this forces the library to fail loudly (rather than silently fall back) if the cache is empty. Cache the required tokenizer files in your artifact registry alongside the model weights.

For tiktoken (used by OpenAI’s library and many wrappers), set TIKTOKEN_CACHE_DIR to a local path and pre-populate it with the required .tiktoken vocabulary files. These files are small (under 2MB) but must be present before the library is called.

Audit every model serving library in your stack for tokenizer fetch behavior before deploying to an air-gapped environment. Test by running with network access blocked at the OS level (iptables or Windows Firewall) in a staging environment that mirrors the air-gapped configuration.

Embedding model fetches

A RAG (retrieval-augmented generation) pipeline requires an embedding model to convert documents and queries into vectors. If your embedding model is fetched from a remote API (OpenAI’s embedding endpoint, Cohere, etc.), you do not have an air-gapped pipeline — you have a pipeline with a dependency on an external service.

For air-gapped deployment, you need a locally served embedding model. Common choices are sentence-transformers models (all-MiniLM-L6-v2 for small/fast, all-mpnet-base-v2 for higher quality) or larger models like BGE or E5, served by a local inference server.

The gap teams miss: the embedding model for ingestion (creating the vector store) and the embedding model for query time must be identical. If you ingest with text-embedding-3-small and query with all-MiniLM-L6-v2, you will get nonsensical retrieval results. If you switch embedding models, you must re-embed your entire corpus. Plan for this before you build the corpus.

Also check your vector database. Some managed vector databases (Pinecone, Weaviate Cloud) are SaaS services that do not run in your environment. For air-gapped deployment you need a self-hosted vector database: pgvector in PostgreSQL, Weaviate self-hosted, Qdrant, Milvus, or Chroma. Each has different operational characteristics; choose based on your data volume, query patterns, and team’s operational familiarity.

Telemetry and observability pipelines

Tracing, metrics, and log forwarding all make outbound connections. OpenTelemetry instrumentation sends traces to a collector. Prometheus pushes or is scraped. Log shippers (Fluentd, Vector, Filebeat) forward to a remote endpoint.

In an air-gapped environment, these need to point at internal endpoints — an OpenTelemetry collector running inside the boundary, a Prometheus instance inside the boundary, a log aggregator inside the boundary. This is straightforward to configure and teams usually handle it. The failure mode is less about the primary telemetry path and more about the secondary, unexpected telemetry calls.

Secondary telemetry calls to watch for:

• Sentry and similar error-tracking SDKs — many LLM frameworks include these. They send exception reports to remote endpoints. Set SENTRY_DSN to empty, or configure them to point at a self-hosted Sentry instance.
• LangSmith, LangFuse, and similar LLM observability platforms — the open-source versions can be self-hosted; the SDK defaults to the cloud endpoint.
• Hugging Face Hub API calls for model metadata — even when not downloading weights, some libraries call the Hub API to fetch metadata (model card, config). Set HUGGINGFACE_HUB_VERBOSITY=warning and ensure the offline flags described above are set.
• Python package update checks — some libraries check PyPI for available updates on startup. Minimal impact, but adds a DNS lookup and TCP connection on the critical path at startup.

Instrument your air-gapped environment with network monitoring (at the OS/kernel level, not just application logging) and capture all outbound connections during a representative workload. DNS queries alone will reveal hidden dependencies that no application-level audit would surface.

License validation and activation calls

Commercial software and some open-weight model licenses include runtime validation components that phone home to verify entitlement. This is common in enterprise software generally and increasingly present in AI tooling.

For truly open-weight models (Apache 2.0, MIT) there are no license validation calls — the license is a legal document, not a software check. For models with restricted licenses (Llama community license, various commercial licenses), read the license carefully before assuming you can deploy offline. Some restrict modification, commercial use, or derivative works, but do not include software-level validation. Others do include runtime validation.

For commercial model serving software, license server configuration is typically an explicit deployment step, and the vendor provides documentation for air-gapped scenarios. Ensure you have a local license server or a pre-activated offline entitlement before deploying, and test the failure mode: what happens when the license server is unreachable? Graceful degradation (continue serving but alert) is preferable to hard failure, but the behavior is vendor-determined.

Designing for zero outbound traffic

A reliable air-gapped AI stack requires treating zero outbound traffic as a design constraint, not an afterthought:

Set offline environment variables at the system level, not in application code. Application code can be updated and re-deployed; a system-level environment variable in your base image or VM template is harder to accidentally remove.

# In your base container image or systemd environment
TRANSFORMERS_OFFLINE=1
HF_DATASETS_OFFLINE=1
TIKTOKEN_CACHE_DIR=/opt/ai/tokenizer-cache
HF_HOME=/opt/ai/hf-cache

Block outbound traffic by default, with explicit allowlisting. Configure egress firewall rules to deny all outbound connections except to explicitly approved internal endpoints. This catches hidden dependencies during staging that would otherwise surface in production.

Mirror everything you need. Build a mirroring pipeline that, from an internet-connected environment, pulls all required artifacts — model weights, tokenizer files, container images, Python packages — verifies their checksums, and pushes them to internal registries. The air-gapped environment never fetches from the internet directly; it always fetches from the internal mirror.

Test with a simulated air-gap during staging. Before moving to production, run your full workload in a staging environment with egress blocked. Any outbound connection attempt should be logged and investigated. Address every one before promoting to the air-gapped environment.

Plan for model updates. The air-gap does not mean the model never changes. Define a controlled process for introducing new model versions: pull from internet-connected environment, verify checksums and scan for supply chain risks, push to internal registry, update the model serving configuration, validate performance, promote to production. This process should be documented and practiced.

The infrastructure that makes it sustainable

Managing all of the above by hand for a single deployment is tractable. Managing it across multiple models, multiple teams, and multiple update cycles requires purpose-built infrastructure.

A model registry inside the air-gapped boundary, with versioning and provenance tracking, is the foundation. Kubernetes-based model serving (with KEDA or similar for autoscaling) provides the compute layer. A gateway layer that handles routing, authentication, budgets, and guardrails without external dependencies keeps the complexity from exploding across individual model endpoints.

ManyLayers Deploy is designed for this architecture — Kubernetes-native model serving in air-gapped environments, with ManyLayers Gateway providing the API surface that applications call. The gateway runs entirely on your infrastructure; there are no cloud endpoints in the request path. If you are designing or operating an air-gapped AI deployment, it is worth examining whether the operational overhead of managing these layers independently justifies the integration.

The bottom line: taking AI offline is achievable, but it requires intentional design from the start. Retrofit is expensive. The teams that do it well treat zero outbound traffic as a first-class requirement during architecture, not a constraint they discover during security review.