Self-Healing GPU Training on Kubernetes

Self-healing GPU training means a failed accelerator no longer needs an engineer at 3 a.m. Crusoe builds its Managed Slurm on Kubernetes so that an XID 79 failure triggers node drain, replacement, and checkpoint resume without human action, and the vendor reports roughly five minutes for the platform portion of that recovery.

Self-healing GPU training explained

Self-healing GPU training is the automation of node failure detection, node replacement, and job requeue without human intervention. Crusoe, an infrastructure-as-a-service provider selling Nvidia and AMD GPU capacity through Crusoe Cloud, builds this on top of Managed Slurm, its Slurm service that runs inside the company's managed Kubernetes offering.

The claim that matters for operations is the latency of that loop. In a recorded demo, Crusoe staff reported roughly five minutes from detecting a critical hardware error to having a healthy node back in the node pool. The remaining time was the training script itself loading a model and a checkpoint, for a little under 15 minutes wall clock on that run.

Failures at this scale are treated as routine rather than exceptional. Crusoe's speakers framed GPU failure across thousands of accelerators as inevitable, and manual remediation as the part that does not scale, which is why the remediation path is a platform feature rather than a runbook step.

One boundary belongs next to that number. Crusoe reports the timing from its own demo on a two-node A100 pool, so it is vendor-reported evidence about a specific workload, not an independent measurement across every training configuration.

Why Slurm on Kubernetes instead of either alone

Slurm on Kubernetes keeps Slurm's scheduling semantics while handing node lifecycle to Kubernetes. Slurm, the open-source workload manager built for high-performance computing, gives training teams gang scheduling, topology awareness, and prologue and epilog hooks for node validation. Kubernetes gives the platform side node health, draining, autoscaling, and one observability stack.

Crusoe's implementation rests on Slinky, the SchedMD project for running Slurm on Kubernetes, wrapped by the company's own Slurm operator. That operator manages Slurm users, partitions, configuration files, and storage, so a cluster is created with one command instead of a hand-built install.

The split matters to each audience differently. A researcher gets what behaves like an ordinary Slurm cluster and reaches it over SSH. A platform team gets Slurm as one more workload in a cluster it already runs, with the same telemetry, on-call rotation, and runbooks rather than a second stack to staff.

Crusoe's speakers described this as the reason customers consolidate: an inference service starts on Kubernetes and a training team arrives wanting Slurm, or the reverse, and two parallel stacks add operational cost to both.

What the XID 79 remediation flow actually does

An XID 79 error is an Nvidia driver-reported failure indicating a GPU has fallen off the bus and is unusable. Crusoe wires that detection into an automated sequence, and the interesting part is not any single step but that each step is triggered without a ticket.

The ordered remediation steps

1. The user is notified that a critical hardware error occurred, with no action requested from them.

2. The Slurm operator marks the affected node down and cancels the running job.

3. The job process receives SIGTERM and has up to two minutes to save a checkpoint or flush logs before the node is replaced.

4. The cancelled job is automatically requeued.

5. Autoclusters checks that no pod on the node carries a label that disables automatic replacement.

6. The Kubernetes node is cordoned so it stops accepting work, then drained.

7. The unhealthy node is removed from the node pool and a replacement is drawn from spare capacity.

8. An alert and remediation record is written for the user's reference.

9. When the healthy node returns, the Slurm job starts, loads the checkpoint, and resumes training.

The two-minute SIGTERM window is the design detail that decides whether checkpointing survives a failover. A training script that ignores SIGTERM or writes a checkpoint less frequently than the drain window allows will lose more work than the platform takes to recover the node.

Crusoe calls the remediation component AutoClusters, and describes it as enabled by default when a cluster is provisioned through the one-click path.

A comparison of the two orchestration layers

The two systems in this architecture are not competing for the same job, and treating them as alternatives misses the point of the design.

Dimension	Slurm	Kubernetes
Primary role	High-performance job scheduling, gang scheduling, topology awareness	Node lifecycle, health, autoscaling, observability
User interface	sbatch, squeue, SSH to login node	kubectl, API objects, controllers
Weak spot in this design	Static partitions, manual node health work	No native gang scheduling for training collectives
Who owns it	ML researchers and training engineers	Platform and infrastructure teams
Failure behaviour	Requeues jobs, detects bad nodes with prologue hooks	Cordon, drain, and replace unhealthy nodes

Crusoe's speakers were explicit that Slurm alone can requeue a job and detect a bad node. What adds operational weight is assembling the detection, draining, and replacement logic yourself, either manually or with scripts, and Slurm is not the layer that reports why a job ran slowly.

Slurm's shortfalls listed in the talk were dynamic resource sharing, node health maintenance, and observability. Kubernetes shortfalls were the absence of Slurm's scheduling semantics. Combining them means neither team changes the tool it already knows.

Sharing GPUs between training and inference

Sharing one GPU pool between training and inference is possible when both run as Kubernetes workloads on the same nodes. In Crusoe's model, GPU nodes are Kubernetes nodes first, so a node freed by a finished Slurm job does not sit idle waiting for the next training run.

The talk described the practical version of this: when inference load rises, hardware can be reallocated toward inference pods, and when user load falls and accelerators sit unused, that is the window for a larger training run across more GPUs. Either direction happens inside one cluster rather than across two.

What this requires in practice

A single node pool shared by both workload types means the platform team sets the policy for how much capacity training may claim and how quickly it yields. The transcript did not describe preemption rules, quotas, or priorities in detail, so the governance layer remains something each organization defines.

The rebound behaviour matters for cost more than for scheduling elegance. Idle accelerators between training runs are the same cost as running ones, and reusing them for inference is the mechanism the talk offered for that problem.

Checkpointing is still the part you own

The platform can replace a node in minutes, but only training code can decide what is worth saving. The SIGTERM window in the remediation flow is the contract between the two: the job is told to stop, and it has up to two minutes to write state before the node disappears.

That places a requirement on application code rather than on the cluster. A handler that catches SIGTERM and writes a checkpoint on a schedule tighter than the drain window turns a hardware failure into a few minutes of lost compute. A job that writes checkpoints hourly will lose up to an hour regardless of how fast the replacement node arrives.

The five-minute figure covers detection through a healthy node returning to the pool. Everything after that is model loading, checkpoint loading, and whatever warmup the training script performs, which is why the same demo measured a little under 15 minutes end to end rather than five.

Crusoe's blog post on self-healing PyTorch training documents the same mechanism for readers who want the vendor's own write-up rather than a conference demo.

What the demo measured and what it did not

The demo in the talk ran on a GPU node pool of two A100 nodes, launched a PyTorch training script with sbatch, and then triggered a simulated XID 79 error through internal monitoring. GPU utilization dropped immediately, remediation began, and utilization returned to normal after recovery.

Crusoe reports roughly five minutes for detection through node replacement and a little under 15 minutes end to end. Both figures come from the vendor describing its own platform and its own demo, not from a third-party benchmark or a peer-reviewed study.

A two-node pool also tests a different thing than a thousand-node pool. Replacing one node from spare capacity is straightforward at small scale. Crusoe's speakers framed the problem as one that appears across thousands of GPUs, yet the recorded test was deliberately small, so the numbers should be read as a demonstration of the automation path rather than as fleet-scale throughput evidence.

The talk did not report how often XID 79 errors occur, how spare capacity is sized, or what happens when no replacement node is available.

Where AI engineering teams get the full walkthrough

The recorded session is the Crusoe talk on self-healing training at the AI Engineer World's Fair, delivered by Connor Guerrero, Young Jeong, and Nikhil Gupta and published through the AI Engineer channel in October 2026.

The session covers the limitations of Slurm for modern AI workloads, the reasoning behind a single Kubernetes-based stack, the full AutoClusters remediation sequence, and the demonstration of a killed GPU resuming from checkpoint. Readers who want to go deeper into the concept can watch the walkthrough of fault-tolerant training as presented at the event.

For teams evaluating this architecture, the closing argument from the talk is the part worth keeping: failures are treated as certain, so the infrastructure is designed to handle them autonomously while engineers stay on application code.

FAQ

• What is self-healing GPU training? It is a platform pattern where a failed GPU node is detected, drained, replaced, and the interrupted job is requeued and resumed from a checkpoint without human action. Crusoe implements it by running Managed Slurm on Kubernetes and wiring Nvidia XID errors into automated node remediation.

• What does XID 79 mean? XID 79 is an Nvidia driver error code indicating a GPU has fallen off the bus and is no longer usable. In Crusoe's remediation flow, that code triggers the node being marked down, the job cancelled and requeued, and the Kubernetes node cordoned, drained, and replaced.

• How long does GPU node replacement take? Crusoe reports roughly five minutes from detecting a critical hardware error to returning a healthy node to the pool. The remaining time in its demo was application-level checkpoint and model loading, bringing the recorded run to a little under 15 minutes end to end.

• Is Slurm being replaced by Kubernetes? No. In this architecture Slurm still schedules the training jobs and provides gang scheduling and topology awareness. Kubernetes supplies node lifecycle, health, autoscaling, and observability, and Slurm runs as a workload inside it.

• What is Slinky? Slinky is the SchedMD project for running Slurm on Kubernetes. Crusoe's Slurm operator builds on it to manage Slurm users, partitions, configuration, and storage, so a cluster can be created without a manual install.

• Do training jobs still need checkpoints? Yes. The remediation flow gives a job up to two minutes of SIGTERM grace to save a checkpoint or flush logs. Checkpoint frequency relative to that window determines how much compute a hardware failure actually costs.

• Can the same GPUs serve training and inference? When both run as Kubernetes workloads on one node pool, capacity can move between them. Crusoe describes reallocating hardware to inference during high load and back to training when demand drops.

• Does the vendor demo prove it works at fleet scale? The published demonstration used a two-node A100 pool. The reported timings come from Crusoe describing its own platform, so they show the automation path rather than fleet-scale performance.

• Do ML engineers need to learn Kubernetes to use it? No. Researchers connect over SSH and use sbatch as they normally would. Platform teams keep their existing Kubernetes tooling, telemetry, and runbooks because Slurm runs as one more service in the cluster.

Source video