The self-hosting vs. API decision for LLM inference has shifted dramatically in 2026. Open-weight models now rival proprietary APIs on many benchmarks, inference engines have matured to production-grade, and GPU capacity in Southeast Asia is finally obtainable without six-month lead times. For Malaysian enterprises with data sovereignty requirements, self-hosting is no longer a compromise — it's often the better architecture.

But "let's self-host" is not one decision. It's two: should we self-host, and which engine do we run? This article answers both — with a decision matrix, working deployment patterns on Azure, and the real cost math.

Why Teams Are Rethinking the API Default

Three forces push enterprise teams away from pure-API inference:

  • Data sovereignty. When prompts contain personal data — customer records, employee files, financial information — sending them to an offshore API endpoint creates a PDPA exposure. For financial institutions, BNM's Risk Management in Technology (RMiT) standard raises the bar further with explicit expectations on data residency and third-party risk.
  • Cost unpredictability. API pricing shifts with model generations and vendor decisions. If your unit economics depend on someone else's pricing page, you don't have unit economics — you have a bet.
  • Latency and control. Customer-facing agents need consistent time-to-first-token, and API rate limits become a hard capacity ceiling exactly when traffic spikes.

Self-hosting addresses all three — but only if you size it correctly. A GPU VM running at 1% utilization is not cost optimization; it's a donation to Microsoft.

The 2026 Engine Landscape

Three inference engines dominate production deployments.

vLLM: The Production Default

vLLM has become the de facto standard for self-hosted LLM inference. Its PagedAttention mechanism — which manages GPU memory the way an operating system manages virtual memory — delivers 2-4x higher throughput than naive implementations.

Key capabilities:

  • OpenAI-compatible API server (drop-in replacement for the OpenAI SDK)
  • Continuous batching (dynamic request grouping for maximum GPU utilization)
  • Tensor parallelism (split models across multiple GPUs)
  • Speculative decoding (draft model + verification for 2-3x decode speedup)
  • Prefix caching (automatic KV cache reuse for shared prompt prefixes)
  • Multi-LoRA serving (serve multiple fine-tuned variants on one GPU)

Production readiness: Battle-tested at scale; it's the serving layer behind most major open-model deployments.

pip install vllm

# Serve a 284B open-weight MoE model on 4x A100 80GB
# (native FP4 experts + FP8 remainder ≈ 142GB+ of weights —
#  two 80GB GPUs leave no room for KV cache)
vllm serve deepseek-ai/DeepSeek-V4-Flash \
    --tensor-parallel-size 4 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.9 \
    --enable-prefix-caching \
    --host 0.0.0.0 --port 8000

SGLang: The Speed Champion

SGLang (Structured Generation Language) is optimized for structured output and complex multi-step programs. Its RadixAttention mechanism provides more aggressive prefix caching than vLLM's implementation.

Key capabilities:

  • RadixAttention (automatic KV cache sharing across requests with common prefixes)
  • Structured output generation (JSON, regex, grammar-constrained — native, not post-processed)
  • Multi-turn conversation optimization (session-aware caching)
  • FlashInfer kernels (custom CUDA kernels for maximum throughput)

When SGLang wins:

  • Structured output workloads (JSON extraction, function calling)
  • Multi-turn conversations (chat applications, agent loops)
  • Workloads with high prompt overlap (RAG against a shared knowledge base)
pip install "sglang[all]"

python -m sglang.launch_server \
    --model-path deepseek-ai/DeepSeek-V4-Flash \
    --tp 4 \
    --mem-fraction-static 0.9 \
    --host 0.0.0.0 --port 8000

llama.cpp: The Edge and CPU Option

llama.cpp brings LLM inference to commodity hardware — CPUs, Apple Silicon, and edge devices. It's the only production-grade option for deployment without datacenter GPUs.

Key capabilities:

  • GGUF quantized models (2-bit through 8-bit — runs on consumer hardware)
  • Metal acceleration (Apple Silicon native)
  • Vulkan/ROCm support (AMD GPUs, older NVIDIA GPUs)
  • Minimal dependencies (single binary, no Python environment)
  • HTTP server with an OpenAI-compatible API

When llama.cpp wins:

  • CPU-only inference (no GPU budget)
  • Edge deployment (retail, IoT, on-premises kiosks)
  • Apple Silicon development (M-series Macs)
  • Small model deployment (under 14B parameters)
git clone https://github.com/ggml-org/llama.cpp && cd llama.cpp
make -j$(nproc)

# Start the server with a quantized model
./llama-server \
    -m qwen3-14b-q4_k_m.gguf \
    --host 0.0.0.0 --port 8000 \
    -ngl 99 -c 8192

Decision Matrix: Which Engine for Which Workload

FactorvLLMSGLangllama.cpp
Best forGeneral production servingStructured output, multi-turnEdge, CPU, development
GPU requirementNVIDIA (A100, H100, L40S)NVIDIA (A100, H100)CPU or any GPU
Practical max modelMulti-GPU, any sizeMulti-GPU, any size~70B quantized
Throughput vs naive2-4x3-5x (structured workloads)0.5-1x (CPU)
OpenAI API compatYesYesYes
Structured outputGood (guided decoding)Excellent (native)Good (grammars)
QuantizationAWQ, GPTQ, FP8AWQ, GPTQ, FP8GGUF (Q2-Q8)
Multi-LoRAYesYesLimited
Prefix cachingYesYes (RadixAttention)No

Standing Up Inference on Azure

Before the software, the hardware. One Azure-specific gotcha first: GPU quota defaults to zero cores for the NC/ND families on most subscriptions. Check and request increases before you commit to any launch date:

# Check current GPU quota in Malaysia West
az vm list-usage --location malaysiawest \
  --query "[?contains(name.value,'NC')].{name:name.value,current:currentValue,max:limit}" \
  -o table

# Request quota increase (40 vCPUs = one NC40ads_H100_v5)
az quota create \
  --resource-name StandardNCadsH100v5Family \
  --scope subscriptions/<subscription-id>/providers/Microsoft.Compute/locations/malaysiawest \
  --limit 40

# Confirm the SKU is actually offered in your target region before committing to a launch date
az vm list-skus --location malaysiawest --query "[?contains(name,'NC') || contains(name,'NV')].name" -o table

That last command matters more than most teams expect. As of August 2026, az vm list-skus --location malaysiawest returns the NVadsA10_v5 (A10 24GB), NCads/NCadis_H100_v5 (H100), and NCas_T4_v3 families — but not the A100 v4 family. The NC24/NC48/NC96ads_A100_v4 SKUs are offered in southeastasia (Singapore), while the ND96amsr_A100_v4 8-GPU node is only offered in regions such as eastus, westus2, eastus2, westeurope, and uksouth. If in-country residency is non-negotiable, size against H100 v5 or A10 v5 SKUs in Malaysia West; if you need A100-class capacity, plan for a Singapore deployment and check whether your regulator accepts it. Always verify with az vm list-skus --location <region> --all before promising any specific GPU SKU in a specific region — availability changes quietly.

Then provision the VM and lock down the network path (example uses southeastasia, where the A100 v4 family is offered):

az group create --name rg-inference-prod --location southeastasia

az vm create \
  --resource-group rg-inference-prod \
  --name vm-vllm-01 \
  --size Standard_NC48ads_A100_v4 \
  --image Ubuntu2204 \
  --admin-username azureuser \
  --ssh-key-values ~/.ssh/id_rsa.pub \
  --os-disk-size-gb 512

# Inference port reachable from the app subnet only — never the internet
az network nsg rule create \
  --resource-group rg-inference-prod \
  --nsg-name vm-vllm-01NSG \
  --name Allow-AppSubnet-8000 \
  --priority 100 --direction Inbound \
  --source-address-prefixes 10.0.2.0/24 \
  --destination-port-ranges 8000 \
  --access Allow --protocol Tcp

For IaC-managed estates, the same VM in Bicep (network and disk resources omitted for brevity):

resource vllmVm 'Microsoft.Compute/virtualMachines@2024-07-01' = {
  name: 'vm-vllm-01'
  location: 'southeastasia' // A100 v4 family not offered in malaysiawest — verify with az vm list-skus
  properties: {
    hardwareProfile: { vmSize: 'Standard_NC48ads_A100_v4' }
    storageProfile: {
      imageReference: { publisher: 'Canonical', offer: '0001-com-ubuntu-server-jammy', sku: '22_04-lts-gen2', version: 'latest' }
      osDisk: { createOption: 'FromImage', diskSizeGB: 512 }
    }
  }
}

Because all three engines speak the OpenAI API, application code is engine-agnostic — switching engines later is a base_url change, not a rewrite:

from openai import OpenAI

client = OpenAI(
    base_url="http://10.0.1.10:8000/v1",
    api_key="your-vllm-api-key",
)

resp = client.chat.completions.create(
    model="deepseek-ai/DeepSeek-V4-Flash",
    messages=[{"role": "user", "content": "Summarize PDPA obligations for a KL-based insurer."}],
    max_tokens=512,
)
print(resp.choices[0].message.content)

Container and Kubernetes Patterns

For production, containerize the engine. Isolation, reproducibility, and rolling updates all come free.

docker pull vllm/vllm-openai:latest

docker run --gpus all -d \
  --name vllm-server \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model deepseek-ai/DeepSeek-V4-Flash \
  --tensor-parallel-size 4 \
  --gpu-memory-utilization 0.9 \
  --enable-prefix-caching

For teams running AKS, a production deployment uses GPU node selectors and resource limits:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: vllm-inference
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm
  template:
    metadata:
      labels:
        app: vllm
    spec:
      nodeSelector:
        node.kubernetes.io/instance-type: Standard_NC96ads_A100_v4
      containers:
      - name: vllm
        image: vllm/vllm-openai:latest
        args:
        - "--model"
        - "deepseek-ai/DeepSeek-V4-Flash"
        - "--tensor-parallel-size"
        - "4"
        - "--gpu-memory-utilization"
        - "0.9"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 4

This pattern gives you self-healing, rolling updates, and horizontal scaling by adding replicas — critical once inference becomes a business-critical dependency.

The Real Cost Math: Self-Hosting vs. API

Numbers below are rough estimates based on published Azure pay-as-you-go pricing as of mid-2026; reserved capacity cuts GPU compute costs by 30-60%. API baseline: flagship-class pricing at $3 per million tokens.

Scenario 1: 10M tokens/month (small workload)

API: 10M × $3/1M = $30/month. No infrastructure, no operations.

Self-hosted: A 14-32B open-weight model on NC24ads_A100_v4 (1x A100 80GB), ≈ $2,600-2,900/month at pay-as-you-go rates. Capacity is roughly 1-2B tokens/month — utilization under 1%.

Verdict: Self-hosting is roughly 90x more expensive here. Use the API.

Scenario 2: 500M tokens/month (medium workload)

API: 500M × $3/1M = $1,500/month.

Self-hosted: Same 1x A100 VM ≈ $2,600-2,900/month, now 25-40% utilized.

Verdict: The API is still cheaper in raw dollars. Self-hosting at this volume is a sovereignty decision (PDPA/RMiT), not a cost decision.

Scenario 3: 10B tokens/month (large workload)

API: 10B × $3/1M = $30,000/month.

Self-hosted: DeepSeek-V4-Flash (native FP4/FP8 weights) on ND96amsr_A100_v4 (8x A100 80GB — US/Europe regions only, not Singapore) ≈ $23,000-25,000/month pay-as-you-go — or roughly $10,000-13,000/month with 3-year reserved capacity. With continuous batching, sustained aggregate throughput of 250-350K tokens/minute is realistic — capacity of 10-15B tokens/month.

Verdict: Self-hosting saves $5,000-7,000/month (~20%) at pay-as-you-go rates, and $17,000-20,000/month (55-65%) once reserved capacity is committed — with headroom for spikes.

The Break-Even Point

Break-even tokens/month ≈ GPU VM monthly cost ÷ API price per token
  • $2,600-2,900/month 1x A100 VM vs $3/1M API → ~900M-1B tokens/month
  • $23,000-25,000/month 8x A100 node (pay-as-you-go) vs $3/1M API → ~8B tokens/month, dropping to ~3.5-4B with 3-year reserved pricing

Two caveats. First, if your realistic API baseline is a budget-tier model ($0.10-0.50 per million tokens), break-even shifts 6-30x higher — compare against the capability class you would actually run, not headline prices. Second, self-hosting adds operational cost that never appears on the VM invoice: patching, monitoring, model updates, and on-call.

The Malaysian Enterprise Decision

Three factors specific to Malaysian enterprises push toward self-hosting regardless of the raw cost math:

1. Data Sovereignty

If your workload processes personal data — customer records, employee data, financial information — PDPA compliance may require that the data never leaves your control. Financial institutions face BNM RMiT on top of that. Self-hosting on Azure VMs in Malaysia West keeps everything in-country and makes the audit story straightforward.

2. Latency

Malaysian enterprises serving ASEAN customers need sub-200ms response times. Inference on Azure Malaysia West eliminates the round-trip to Singapore or US API endpoints. For customer-facing agents, that round-trip is often the difference between "feels instant" and "feels sluggish."

3. Predictable Costs

API pricing changes without notice. Reserved capacity helps but doesn't eliminate the exposure. Self-hosted infrastructure has a fixed monthly cost that finance can budget — and that you can optimize over time with quantization and better utilization.

Production Deployment Checklist

Hardware Sizing

Model SizeMin GPURecommended GPUAzure VM SKU
7-8B1x A10 24GB1x A100 80GBNVadsA10_v5 / NC24ads_A100_v4
13-14B1x A100 80GB1x A100 80GBNC24ads_A100_v4
30-34B1x A100 80GB2x A100 80GBNC48ads_A100_v4
70B (INT4)2x A100 80GB4x A100 80GBNC96ads_A100_v4
250-300B MoE (INT4)4x A100 80GB4x-8x A100 80GBNC96ads_A100_v4 / ND96amsr_A100_v4 (US/Europe only)
100B+ dense4x A100 80GB8x A100 80GBND96amsr_A100_v4 (US/Europe only)

High Availability

# Health check endpoint (vLLM and SGLang both expose it)
curl http://localhost:8000/health
upstream llm_backend {
    server 10.0.1.10:8000 weight=1;
    server 10.0.1.11:8000 weight=1;
    server 10.0.1.12:8000 weight=1;
}

server {
    listen 443 ssl;
    location /v1/ {
        proxy_pass http://llm_backend;
        proxy_read_timeout 300s;
    }
}

Monitoring

vLLM exposes a Prometheus endpoint at /metrics — scrape it into Azure Monitor managed Prometheus and track:

  • Tokens per second (throughput)
  • Time to first token (latency)
  • GPU utilization (efficiency)
  • Queue depth (capacity planning)
  • Error rate (reliability)
  • Cost per 1M tokens (unit economics)

Security

  • TLS termination at the load balancer or App Gateway, not the inference server
  • API key authentication (vLLM supports --api-key; SGLang supports --api-key)
  • Network isolation (NSG rules limiting access to known subnets — never expose 8000 to the internet)
  • Model artifact security (private endpoints or encrypted storage for model weights)
  • Prompt injection defense — self-hosted models lack the safety layers of Azure OpenAI. Implement input validation and output filtering at the application layer

Five Pitfalls That Sink Self-Hosted Inference

  1. Benchmarking single-stream speed instead of throughput. A demo that generates 60 tokens/second tells you nothing about production capacity. Load-test with realistic concurrency — vLLM ships benchmark_serving.py for exactly this.
  2. Defaulting to maximum context length. KV cache memory grows linearly with context. Setting --max-model-len 131072 on a large model can consume more VRAM than the weights themselves. Set it to what your workload actually needs.
  3. Shipping an open endpoint. vLLM and SGLang ship without authentication or rate limiting by default. I've seen inference servers indexed by Shodan within days. Use --api-key, NSG restrictions, and TLS termination — all three.
  4. Treating quantization as free. INT4/AWQ quality regressions are task-dependent — often negligible for summarization, noticeable for math and code. Evaluate on your own domain data before committing to a quantization level.
  5. Discovering GPU quota at deploy time. Azure NC/ND families default to zero cores on most subscriptions, and quota approval can take days. Request increases two to three weeks before your target launch date.

Conclusion

Self-hosting LLM inference in 2026 is no longer an experiment — it's an architecture decision with a calculable break-even point. Below roughly 1B tokens/month, the API wins on simplicity and usually on cost; above a few billion tokens/month, self-hosting wins decisively — especially with reserved capacity; and data sovereignty can tip the decision at any volume in between. Start with vLLM, reach for SGLang when structured output dominates your workload, and keep llama.cpp for the edge.

Key Takeaways

  1. The break-even point is ~900M-1B tokens/month for a single-GPU VM against flagship-class API pricing at pay-as-you-go rates — below it, the API wins; above a few billion tokens/month, self-hosting saves real money, and reserved capacity turns that into 50%+ savings.
  2. vLLM is the default choice for production — best ecosystem, most features, battle-tested at scale.
  3. SGLang wins for structured output — JSON extraction, function calling, and multi-turn agent loops benefit most from RadixAttention and native constrained decoding.
  4. llama.cpp is for edge and development — not production inference at scale, but unmatched for CPU-only and on-device deployments.
  5. Data sovereignty is the hidden accelerator — for Malaysian enterprises, PDPA and BNM RMiT often justify self-hosting even below the cost break-even point.