Support NVIDIA MIG slices in the Kubernetes backend

[peterschmidt85]

Design doc and real-cluster results: MIG support for the Kubernetes backend

Summary

• MIG slices created by the NVIDIA GPU Operator, with either the single or the mixed strategy, are offered
  as GPUs of the parent model with the slice's memory. For example, a 1g.10gb slice of an A100 is A100:10GB.
• A run gets slices only when its GPU memory has an upper bound (gpu: A100:10GB). A run with gpu: A100 gets
  only whole GPUs. --full-offers shows everything.
• With the mixed strategy, each nvidia.com/mig-<profile> resource on a node is its own offer, next to the
  node's whole GPUs. The pod requests that resource.
• After a pod fails to schedule, only offers for the same GPU resource are skipped. Before, every offer in the
  cluster was skipped, so a failed 3g.20gb attempt also ruled out the cluster's 4g.20gb slices and whole GPUs.
• The change is limited to the Kubernetes backend: no new configuration and no model changes. The docs get a MIG
  section under the Kubernetes backend.

Out of scope

• Requesting a specific MIG profile, e.g., 3g.40gb vs 4g.40gb. Slices are matched by memory only.
• GPU metrics for slices in dstack metrics (a follow-up).
• GKE's built-in MIG (gpu-partition-size) without the GPU Operator.

Testing

On GKE (A100 40GB, GPU Operator v26.7.2, driver 595), these layouts all passed:

• single strategy next to whole-GPU nodes;
• mixed strategy with whole GPUs and slices on one node;
• MIG only.

The results are in the design doc.

---

Generated by Claude Code