helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator \
  --create-namespace \
  --version v24.3.0 \
  -f values.yaml \
  --wait --timeout 10m

Multi-Instance GPU (MIG) on A100 and H100 partitions a single physical GPU into up to 7 isolated instances, each with dedicated compute, memory, and memory bandwidth. One A100 80GB can serve seven independent inference workloads simultaneously with hardware-level isolation — one tenant cannot affect another's performance.

Time-slicing for GPUs without MIG support (T4, L4, A10G) shares a single GPU across multiple pods by interleaving their execution. It does not provide memory isolation, so workloads can interfere with each other — but for development and low-priority inference, it multiplies effective GPU count by 2-4x.

The combination of MIG for production inference and time-slicing for development is what took utilization from 18% to 72% at the Fortune 500 deployment I mentioned. That utilization improvement is worth more than any spot instance discount.

FinOps Strategies That Cut GPU Spend by 60%

After deploying GPU platforms at three enterprises, these are the five strategies that consistently deliver the largest savings:

1. Spot instances for fault-tolerant training. Distributed training with checkpointing every 15 minutes tolerates spot interruptions. At 60-80% discount, spot H100s cost $2.46-$4.92/GPU-hr versus $12.29 on-demand. The training job adds 10-15% overhead for checkpointing but saves 65% on compute.

2. Right-sizing with GPU utilization data. DCGM Exporter exposes DCGM_FI_DEV_GPU_UTIL (compute utilization) and DCGM_FI_DEV_FB_USED (memory used). If a workload consistently uses 8GB VRAM on an A100 80GB, it should be on a T4 or L4. I have seen teams run XGBoost training on H100 instances because "that is what we had." Moving those workloads to T4s saved $180,000 per quarter.

3. Inference auto-scaling on queue depth. Scale GPU inference deployments based on request queue depth, not CPU utilization. A KEDA scaler watching the pending request count in your inference queue adds replicas before latency degrades and removes them during idle periods.

4. Reserved capacity for baseline, on-demand for peaks. Size your reserved instances to cover the bottom 40th percentile of your GPU demand curve. Handle the remaining 60% with on-demand or spot. This typically yields 35-45% savings versus all on-demand.

5. Caching and batching at the inference layer. Semantic caching (embedding-based similarity matching of previous requests) reduces redundant inference calls by 15-30% in typical enterprise workloads. Request batching groups multiple inference requests into a single GPU forward pass, improving throughput by 3-8x with minimal latency impact.

Explore our Cloud Infrastructure Toolkits for GPU FinOps dashboards, Grafana templates, and Kubernetes scheduling configurations.

Observability: Knowing Where Your GPU Dollars Go

GPU observability is not optional — it is the feedback loop that makes every other optimization possible. NVIDIA's DCGM (Data Center GPU Manager) Exporter feeds Prometheus metrics into Grafana dashboards covering:

• GPU utilization per pod — identifies underutilized workloads for right-sizing
• Memory allocation vs actual usage — catches overprovisioned requests
• SM (Streaming Multiprocessor) occupancy — reveals whether compute cores are actually busy or waiting on memory
• Power consumption per GPU — directly maps to electricity cost for on-prem
• Job queue wait time — tracks how long training jobs wait for GPU resources

The dashboard that changed behavior at every organization I have deployed it in is the cost attribution dashboard: GPU-hours consumed per team, per project, per model, with dollar values attached. When a data science team sees that their weekly experiment runs cost $14,000 and only 3 of 47 runs produced models worth evaluating, they start designing experiments more carefully.

Learn the full GPU infrastructure stack with our free Cloud Infrastructure course covering Kubernetes, monitoring, and cost optimization for AI workloads.

Frequently Asked Questions

Should we buy on-prem GPUs or use cloud for AI training in 2026?

The decision depends on sustained utilization and time horizon. If your team will use GPUs at 55% or higher utilization continuously for three or more years, on-prem DGX or custom-built GPU servers deliver 40-60% lower total cost of ownership compared to on-demand cloud pricing. However, most teams overestimate their utilization — the honest number for enterprise AI teams without a shared Kubernetes platform is 15-25%. Start with cloud reserved instances for 12 months to establish your real utilization baseline before committing to hardware purchases.

How do MIG and time-slicing compare for sharing GPUs across teams?

MIG provides hardware-level isolation with dedicated compute, memory, and memory bandwidth per partition — one workload cannot affect another's performance. It is available only on A100 and H100 GPUs and supports up to 7 partitions. Time-slicing shares a single GPU by interleaving execution across pods, works on any NVIDIA GPU, but provides no memory isolation. Use MIG for production inference where performance guarantees matter. Use time-slicing for development, testing, and low-priority batch jobs where occasional contention is acceptable.

What is the most cost-effective GPU for LLM inference in 2026?

For models under 13B parameters (Llama 2 7B, Mistral 7B, Phi-3), the NVIDIA L4 at $0.805/hr offers the best cost-per-token ratio at $11.29 per million tokens. For models between 13B and 70B parameters, the A100 80GB provides sufficient VRAM and bandwidth at a reasonable cost point. For 70B+ parameter models requiring multi-GPU setups, H100 SXM clusters are the only viable option for production throughput. Always benchmark on your specific model and batch size — published TFLOPS numbers do not predict real-world inference cost.

Ready to build production GPU infrastructure? Browse 320 premium cloud architecture blueprints or start with our 17 free courses covering Kubernetes, GPU scheduling, and cloud cost optimization.