One of the most common questions on the NVIDIA Developer Forums is: Should you use Docker Compose or Kubernetes to run NIM microservices on your DGX Spark? This guide cuts through the confusion and gives you practical steps for both setups so you can decide which fits your workload.

Why This Decision Matters

The NVIDIA DGX Spark (GB10 Grace Blackwell Superchip, 128GB unified memory) is a powerful personal AI supercomputer. Choosing the wrong orchestration layer can waste GPU cycles or complicate your deployment. Here is a practical breakdown.

Option 1: Docker Compose Setup

Best for: Single DGX Spark node, quick prototyping, dev/test workflows.

Step 1: Install NVIDIA Container Toolkit

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Step 2: Authenticate with NGC

docker login nvcr.io
# Username: $oauthtoken
# Password: YOUR_NGC_API_KEY

Step 3: Create docker-compose.yml for NIM

version: "3.8"
services:
  nim-llm:
    image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
    runtime: nvidia
    environment:
      - NIM_HTTP_API_PORT=8000
    ports:
      - "8000:8000"
    volumes:
      - ./models:/opt/nim/.cache
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Step 4: Launch the NIM service

export NGC_API_KEY=nvapi-xxxxxxxxxxxx
docker compose up -d

Step 5: Test the endpoint

curl -X POST http://localhost:8000/v1/completions   -H "Content-Type: application/json"   -d '{"model": "meta/llama-3.1-8b-instruct", "prompt": "What is DGX Spark?", "max_tokens": 100}'

Option 2: Kubernetes Setup (K3s)

Best for: Multi-node DGX Spark clusters, production workloads, GPU scheduling across nodes.

Step 1: Configure NVIDIA runtime as default

sudo nvidia-ctk runtime configure --runtime=docker --set-as-default
sudo systemctl restart docker

Step 2: Install K3s with Docker runtime

curl -sfL https://get.k3s.io | sh -s - --docker --write-kubeconfig-mode 644

Step 3: Install NVIDIA GPU Operator

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator   --namespace gpu-operator   --create-namespace   --set driver.enabled=false   --set toolkit.enabled=false

Step 4: Create NGC pull secret

kubectl create secret docker-registry ngc-secret   --docker-server=nvcr.io   --docker-username='$oauthtoken'   --docker-password=YOUR_NGC_API_KEY

Step 5: Deploy NIM as a Kubernetes Deployment

cat <<'EOF' | kubectl apply -f -
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nim-llm
spec:
  replicas: 1
  selector:
    matchLabels:
      app: nim-llm
  template:
    metadata:
      labels:
        app: nim-llm
    spec:
      imagePullSecrets:
      - name: ngc-secret
      containers:
      - name: nim-llm
        image: nvcr.io/nim/meta/llama-3.1-8b-instruct:latest
        env:
        - name: NGC_API_KEY
          value: "YOUR_NGC_API_KEY"
        ports:
        - containerPort: 8000
        resources:
          limits:
            nvidia.com/gpu: 1
EOF

Step 6: Expose the service and test

kubectl expose deployment nim-llm --type=NodePort --port=8000
kubectl get svc nim-llm

Quick Comparison

Feature	Docker Compose	Kubernetes (K3s)
Setup time	~10 minutes	~30 minutes
Multi-node support	No	Yes
GPU scheduling	Basic	Advanced
Best for	Single Spark, dev	2+ Spark nodes, prod

Verdict

If you have a single DGX Spark and want to get NIM running fast, Docker Compose wins. If you have two or more Sparks and need production-grade scheduling, go with K3s or full Kubernetes. The GPU performance of NIM containers is identical in both — the difference is only in orchestration overhead.