GPU Utilization Data Pipeline Fixes That Work

85% of training time spent waiting on data is the number that opens this story. In a 2026 talk on the GPU utilization data pipeline for a Qwen3-VL-style model, Tarun Sunkaraneni showed that serial image loading from S3 left GPUs at 15% utilization, and that four pipeline fixes raised it to 90%.

Why Multimodal Training Is IO-Bound, Not GPU-Bound

Multimodal supervised fine-tuning is frequently CPU- and IO-bound because every training example needs images loaded, decoded, resized, and normalized before the GPU does any math. In the GPU utilization data pipeline described by Tarun Sunkaraneni of Amazon AGI at the AI Engineer World's Fair 2026, the baseline spent 85% of its time waiting on data acquisition and only 15% actually training.

Text-only training streams tokens that are cheap to prepare. Vision-language training is different. Each sample involves fetching a JPEG from object storage, decoding it with a library such as Pillow, resizing and normalizing pixel arrays, and tokenizing the conversation text. Audio and video add loading and cropping work of their own.

The stack in this case trained a Qwen3-VL-style model on chat, VQA, and multi-turn VLM conversations stored as JSONL files with relatively linked images in Amazon S3. The data generator fans out to Megatron data-parallel ranks, which collate batches and run the forward and backward passes. Megatron handles weight synchronization across ranks; the pipeline's job is to keep those ranks fed.

What the 15% Baseline Looked Like

The baseline processed every example serially: fetch each image from S3, decode it, resize it, normalize it, then tokenize the text. With a trainer batch size of 256 examples, all of that image work happened one sample at a time before the GPUs could run.

The measured asymmetry was stark. Training a batch took roughly 15 to 20 seconds, while fetching and preparing the data for that batch took more than 100 seconds. A profiling flame graph showed most time in S3 loads, with a long tail in Pillow image decoding. The resulting wait-time ratio, defined as data prep and transfer time divided by total pipeline time, sat at about 85%.

The lesson generalizes: before optimizing kernels, profile whether the training loop is even blocked on the GPU. Expensive accelerators starving on IO are a common and measurable failure mode in multimodal runs.

Choosing Concurrency: asyncio, Threads, or Processes

Adding concurrency was the first fix, and the right kind depends on the bottleneck. Fetching images from S3 is network IO, so the pipeline used asyncio. Image decoding and text tokenization are CPU-bound, so they moved to Ray actors, which behave like processes but ship APIs for communicating between workers. Samples are independent, which makes the workload embarrassingly parallel.

Samples are only as fast as the slowest link, so the concurrency mechanism has to match the bottleneck rather than reuse whatever worked last time.

The three options behave differently under CPython's Global Interpreter Lock (GIL):

Mechanism	Best suited for	Key constraint
asyncio	IO-bound work such as S3 fetches or database queries	Single thread; CPU work blocks the loop
Threads	CPU work that releases the GIL, e.g. tokenizer or Pillow transforms	Not obvious which native calls release the GIL
Processes (Ray actors)	CPU-bound work needing true parallelism and isolation	Heavier; adds inter-process communication cost

The GIL caveat matters in practice. Threads help only when native extensions release the lock, and as of Python 3.14 the free-threaded build is officially supported but is not the default interpreter most training clusters run. This concurrency step cut the per-batch fetch time to around 40 seconds, a roughly 25% speedup, and the wait ratio began to fall.

Prefetching: Keep the Producer Ahead of the Consumer

Even with concurrent workers, the pipeline still waited for the trainer to request each batch before starting work. The second fix was a background thread that fills a queue ahead of time, so the producer continuously outpaces the consumer and data is available on demand.

With prefetching in place, per-batch data fetching and processing time dropped to nearly zero, because examples were fetched and transformed well before the trainer asked. This effectively eliminated both the data-load and transform bottlenecks.

Prefetching has two documented costs. Checkpointing and resumption become more intricate, since in-flight queue contents have to be accounted for, and there is a cold-start period while the queue fills before training begins. Both are manageable, but they should be designed for rather than discovered mid-run.

Zero-Copy Transfer With the Ray Object Store

After the first two fixes, a third bottleneck surfaced: transport. Multimodal tensors are large, often megabytes per sample, and by default every sample returned from a Ray worker is passed by value. Each sample was pickled on the worker, copied to the coordinator, and re-pickled to the trainer, once per sample per step.

The fix was Ray's object store. Each sample carries an object reference, a pointer to the data, instead of the data itself. The driver holds only the pointer, tensors never round-trip through the head node, and Megatron data-parallel ranks fetch tensors directly from the store as needed. This also removed an extraneous put-object call that the prefetcher design had introduced.

The broader principle: every parallelism choice adds communication costs, and those costs have to be planned for, not discovered in production.

Why Scale Flipped the Verdict on Two Flags

At production scale the same pipeline regressed. Adding four times the data-parallel ranks, about 200 data streams, and a 4x batch size pushed the wait ratio back up, because the data generator and worker pool were co-located on one node and saturated its network interface card.

Ray by default prefers to schedule workers on the same node as the driver, which makes it easy to saturate one machine before using another. The team enabled two settings: spread scheduling, which distributes workers across nodes, and zero-copy retrieval, which lets ranks read results lazily and just in time.

The instructive part is the interaction. Neither flag produced a measurable win in the small-scale experiment; together they delivered an additional 50% throughput at scale. The speaker's conclusion is worth quoting directly: the same knob can give opposite verdicts depending on the bottleneck, so ablate configurations across your scaling ladders rather than trusting small-scale results.

The final pipeline was concurrent, eager, and data-access aware. Overall utilization moved from a baseline wait ratio of 85% down to under 10%, with GPU utilization rising from 15% to 90%.

Four Fixes, In Order

The full progression from the talk, in the order it was applied:

Concurrency. asyncio workers for S3 fetches, Ray actor processes for decoding and tokenization, matched to the actual bottleneck. About 25% faster data prep per batch.
Prefetching. A background producer thread keeps a queue ahead of the trainer, cutting per-batch data wait to near zero at the cost of more complex checkpointing.
Zero-copy object store. Object references replace value copies between processes, eliminating per-sample pickling and mem-copy overhead.
Scale-aware scheduling. Spread scheduling plus zero-copy retrieval, tuned at production scale, added another 50% throughput.

Each fix exposed the next bottleneck: first throughput, then wait time, then memory, then retrieval cost. The diagnostic sequence is to establish whether you are CPU-bound or GPU-bound, and IO-bound or compute-bound, before touching the model or kernels.

FAQ

Why is GPU utilization low during multimodal training?

Because each example requires CPU-heavy preprocessing: loading, decoding, resizing, and normalizing images before the GPU runs. If this happens serially, the GPUs spend most of each step waiting, which is what drove the 15% baseline in this pipeline.

When should I use asyncio versus processes for a training data pipeline?

Use asyncio for IO-bound work such as object-storage fetches and network requests. Use processes, for example Ray actors, for CPU-bound work such as image decoding and tokenization, where true multi-core parallelism is needed. Threads help only for operations that release the GIL.

What does the Ray object store change in a training pipeline?

It replaces per-sample value copies with object references, so large image tensors are not pickled and copied between worker, coordinator, and trainer. Data-parallel ranks read tensors directly from the store, removing redundant mem-copies.

Why did spread scheduling only help at large scale?

At small scale, co-locating workers and the driver keeps transfer latency low and nothing saturates. At production scale, co-location bottlenecks a single node's network card, so spreading workers across nodes plus zero-copy retrieval became the winning configuration.

Who presented this GPU utilization data pipeline work?

Tarun Sunkaraneni of Amazon AGI presented it at the AI Engineer World's Fair 2026 in San Francisco, in a talk on Ray actors, vision tokens, and the GIL. The figures cited here are from that talk.

Turn Your Own Talks Into Articles

A 17-minute conference talk like this one holds a full engineering narrative: a baseline, four bottlenecks, and the measurements that prove each fix. That structure is exactly what written search-friendly articles are made of, yet most of it stays locked inside a video file.

If you have talks, interviews, or walkthroughs sitting on YouTube, Skala Blog turns them into structured written articles. Paste the video URL, transcribe it, and generate an article that keeps the numbers, the causal sequence, and the caveats intact.

Source video