On-Premises vs. Cloud AI for Manufacturing

The shop floor reality

Every AI demo I've ever seen runs on a MacBook Pro with a strong WiFi connection in a conference room. The real deployment target for manufacturing AI is a lot less comfortable: a control room adjacent to a CNC floor, a dusty cabinet next to a conveyor line, or a server rack in a facility with spotty connectivity in a rural industrial park.

The assumption baked into most AI tooling — that you have reliable broadband, that sending data to a third-party API is fine, that your customer can just sign up for an OpenAI account — falls apart in practice. Here's what you actually encounter:

• Connectivity gaps. Many manufacturing facilities, especially in rural areas, have unreliable or limited internet. A system that goes down when the WAN link drops is not a system a plant manager will trust.
• IT security constraints. Enterprise manufacturers have network segmentation — the OT (operational technology) network running SCADA and PLCs is often air-gapped or severely restricted from the internet. Your AI system may never be allowed to touch an external API.
• Proprietary process data. A manufacturer's SOPs, process parameters, quality specs, and machine configurations are competitive IP. Sending them through a shared API endpoint is a non-starter for many customers, regardless of the provider's privacy policies.
• Procurement friction. "Just sign up for an API key" is not how enterprise procurement works. Some manufacturers have policies against any new SaaS vendor without a full security review. On-premises removes the vendor entirely.

None of this means cloud AI is wrong for manufacturing. It means you need to design for both deployment modes from day one, not bolt on an on-prem option as an afterthought.

What data sovereignty actually means here

Data sovereignty in manufacturing isn't abstract. Here are the concrete things your customers are worried about:

Process recipes. A food manufacturer's seasoning ratios or a chemical processor's reaction parameters are trade secrets. They may have been developed over decades. The idea that this data leaves the facility and gets processed on shared infrastructure — even infrastructure with strong privacy guarantees — is genuinely alarming to the people responsible for protecting it.

Quality data. Defect rates, rejection patterns, process capability indices — this is information competitors would pay to have. It also contains signals about what a manufacturer is making, how much of it, and for whom.

Maintenance history. Equipment maintenance records reveal what machines you have, how hard you run them, and when they're likely to fail. For a contract manufacturer, this is operational intelligence they don't want outside the building.

The legal and regulatory angle is real too: ITAR-controlled manufacturers (defense supply chain) often face hard restrictions on where controlled technical data can be processed. CMMC compliance for DoD contractors has specific requirements about data residency.

The practical implication: your AI architecture needs a clear answer to "where does the data go?" before you get to the pilot stage with enterprise customers. "To OpenAI" is often not an acceptable answer.

The latency argument

There's a second reason to care about local inference beyond data sovereignty: latency on control-adjacent workloads.

An API call to a cloud LLM takes 500ms to 5 seconds under normal conditions. For a document Q&A interface where a maintenance tech is looking something up, that's fine. For an anomaly detection system that needs to assess a sensor reading and decide whether to trigger an alert before the next PLC scan cycle, it's not.

The PLC scan cycle in most manufacturing automation is 10-100ms. You're not going to run LLM inference in that loop — that's still deterministic control logic. But the layer above it — the AI advisor that interprets what the anomaly means and recommends action — needs to respond in seconds, not tens of seconds. Cloud API latency plus network overhead adds up.

Local inference on a decent GPU gives you 2-8 tokens/second on a 7B model, 1-3 tokens/second on a 14B model. For a 150-token explanation, that's under 10 seconds on a 7B model running on an RTX 4090. Acceptable for an advisory layer. Infeasible over a WAN link with a loaded API.

Building a fully local AI stack

Harmony Core is my on-premises AI agent stack for manufacturing. The design constraint: nothing leaves the building. Local auth, local inference, local embeddings, local storage. All via Docker Compose so it deploys on any Linux box without an internet connection after initial setup.

Here's what the full local stack looks like:

services:
  # PostgreSQL + pgvector for all storage
  postgres:
    image: pgvector/pgvector:pg16
    environment:
      POSTGRES_DB: harmony
      POSTGRES_USER: harmony
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - postgres_data:/var/lib/postgresql/data
    restart: unless-stopped

  # Local LLM inference — no API keys, no internet
  ollama:
    image: ollama/ollama:latest
    volumes:
      - ollama_models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]
    restart: unless-stopped

  # Self-hosted embeddings
  embeddings:
    image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.5
    command: --model-id BAAI/bge-base-en-v1.5
    volumes:
      - embedding_models:/data
    restart: unless-stopped

  # API server
  api:
    build: ./apps/api
    environment:
      DATABASE_URL: postgresql://harmony:${DB_PASSWORD}@postgres:5432/harmony
      LLM_BASE_URL: http://ollama:11434/v1   # OpenAI-compatible endpoint
      LLM_MODEL: ${LLM_MODEL:-qwen2.5:14b}
      EMBEDDING_URL: http://embeddings:80
    depends_on: [postgres, ollama, embeddings]
    ports:
      - "4000:4000"
    restart: unless-stopped

  # Next.js web UI
  web:
    build: ./apps/web
    environment:
      NEXT_PUBLIC_API_URL: http://api:4000
    ports:
      - "3000:3000"
    depends_on: [api]
    restart: unless-stopped

volumes:
  postgres_data:
  ollama_models:
  embedding_models:

The critical detail is LLM_BASE_URL: http://ollama:11434/v1. Ollama exposes an OpenAI-compatible API endpoint. That means the application code that makes LLM calls doesn't know whether it's talking to Ollama running locally or the real OpenAI API — the interface is identical. Switching between on-prem and cloud is a configuration change, not a code change.

import OpenAI from "openai";  // Works with any OpenAI-compatible endpoint

const llm = new OpenAI({
  baseURL: process.env.LLM_BASE_URL,    // http://ollama:11434/v1 locally
  apiKey: process.env.LLM_API_KEY ?? "ollama",  // Ollama ignores this
});

export async function complete(
  system: string,
  user: string,
  opts: { model?: string; maxTokens?: number } = {}
): Promise {
  const response = await llm.chat.completions.create({
    model: opts.model ?? process.env.LLM_MODEL ?? "qwen2.5:14b",
    messages: [
      { role: "system", content: system },
      { role: "user", content: user },
    ],
    max_tokens: opts.maxTokens ?? 512,
  });
  return response.choices[0].message.content ?? "";
}

Same function whether it's running against Ollama locally or Azure OpenAI in the cloud. The model name changes; nothing else does.

Hybrid inference routing

The interesting architectural case isn't pure on-prem or pure cloud — it's the hybrid, where you route different workloads to different inference backends based on their characteristics.

The routing logic is straightforward:

type WorkloadKind =
  | "anomaly_explanation"
  | "document_qa"
  | "complex_reasoning"
  | "report_generation"
  | "safety_procedure";

interface InferenceConfig {
  baseURL: string;
  apiKey: string;
  model: string;
}

const LOCAL: InferenceConfig = {
  baseURL: process.env.LLM_BASE_URL_LOCAL ?? "http://ollama:11434/v1",
  apiKey: "ollama",
  model: process.env.LLM_MODEL_LOCAL ?? "qwen2.5:14b",
};

const CLOUD: InferenceConfig = {
  baseURL: "https://api.anthropic.com/v1",
  apiKey: process.env.ANTHROPIC_API_KEY ?? "",
  model: "claude-sonnet-4-5",
};

// Workloads that must never leave the facility
const ALWAYS_LOCAL = new Set([
  "anomaly_explanation",
  "document_qa",
  "safety_procedure",
]);

export function routeInference(kind: WorkloadKind): InferenceConfig {
  // Respect explicit on-prem-only mode (e.g., air-gapped facility)
  if (process.env.INFERENCE_MODE === "local_only") {
    return LOCAL;
  }

  // Some workloads are always local regardless of mode
  if (ALWAYS_LOCAL.has(kind)) {
    return LOCAL;
  }

  // Cloud available and workload is cloud-eligible
  if (process.env.ANTHROPIC_API_KEY && process.env.INFERENCE_MODE !== "local_only") {
    return CLOUD;
  }

  // Fallback to local if cloud isn't configured
  return LOCAL;
}

The INFERENCE_MODE=local_only environment variable is a single switch that makes the entire system air-gap safe. Set it in the deployment config for facilities with network restrictions; leave it unset for facilities where cloud calls are acceptable for non-sensitive workloads.

When cloud AI is the right call

I've spent most of this article on the case for on-prem. To be fair: cloud AI is the right answer for a lot of manufacturing use cases.

Use cloud AI when:

• The data involved isn't sensitive — scheduling optimization, general engineering Q&A, non-proprietary document summarization
• You need frontier model capability — complex multi-step reasoning, code generation, sophisticated analysis that a 14B local model genuinely can't match
• The customer is a smaller shop without IT infrastructure to maintain a local AI stack
• You're in a demo or pilot phase and need to move fast before validating that on-prem is worth the complexity
• The workload is batch and async — nightly report generation, weekly trend analysis — where API latency doesn't matter

The hybrid answer for most customers: Use cloud for the heavy reasoning tasks where quality matters and data sensitivity is low. Use local for real-time, data-sensitive, and latency-critical workloads. Design the routing layer so customers can tune the balance based on their specific constraints.

Practical hardware

If you're recommending hardware for an on-premises deployment, here's what actually works in a manufacturing context:

For most SMB manufacturing deployments, the entry or mid tier is right. A 14B model running on an RTX 4090 is remarkably capable for document Q&A and anomaly explanation — the delta from a frontier cloud model is smaller than you'd expect for domain-specific industrial use cases where you control the prompts and context.

The CPU-only option is worth calling out: 2-4 tokens/second sounds slow, but for a use case where the operator submits a query and waits 30-60 seconds for a detailed answer, it's often acceptable. And it means you can run on existing server hardware with no GPU procurement at all.

One more hardware consideration: industrial environment durability. A consumer GPU in a rack near a CNC floor is going to encounter vibration, temperature swings, and metal dust. Use server-grade hardware with proper airflow and dust filtration. The RTX 4090 is a gaming card — it's not rated for industrial environments. If the system is going on the shop floor rather than in a climate-controlled server room, use industrial edge compute (Advantech, OnLogic) with appropriate enclosures.

What I'd do differently

Design for offline-first from day one. I retrofitted offline capability into Harmony Core partway through development. It's much harder to add than to design in from the start. If your system uses any service that requires network access — even for licensing or telemetry — document it explicitly and have a plan for how the system behaves when that service is unavailable.

Build a model management layer early. Ollama makes it easy to pull and run models, but in a production facility you need controlled model versions — not "whatever the latest pull gives you." Build a manifest that specifies exact model versions (including quantization), automates pulling them on deployment, and validates model hashes before use. You want model updates to be a deliberate change, not an accidental drift.

Test on real hardware before the customer pilot. The performance difference between an M2 MacBook Pro running Ollama in development and an RTX 4060 in a mini PC running the same model is significant and non-obvious. A 7B model that feels fast on dev hardware may feel unacceptably slow on the deployment target. Test on something close to production hardware before you make promises about response times.

Quantize aggressively, benchmark honestly. A Q4_K_M quantized 14B model running on 12GB VRAM delivers maybe 85% of the capability of the full-precision version at a fraction of the memory cost. For most manufacturing use cases — document Q&A, anomaly explanation, work order drafting — that 15% capability gap is invisible. Benchmark on your actual use cases, not on generic benchmarks, and quantize to fit your hardware.

Harmony Core — the on-premises agent stack this architecture is based on — is a commercial product at Harmony AI. The architecture patterns here are complete enough to implement independently.

Questions or pushback: email me or find me on LinkedIn.

Next in this series: Multi-Tenant AI Agents — session isolation patterns, pluggable adapter architecture, and role-based tool access in FastAPI.