How to Increase GPU Throughput During Training in MATLAB

Low GPU utilization occurs when the GPU idles while waiting for CPU data preparation, small batch sizes, or excessive CPU-GPU memory transfers. Apply the techniques below to maximize samples processed per second.

1. Increase Mini-Batch Size

Small batches cause kernel launch overhead and underutilize parallel CUDA cores. Double the mini-batch size until GPU VRAM reaches 80% to 90% capacity.

opts = trainingOptions('adam', ...
    'MiniBatchSize', 256, ... % Increase from 32/64 to 128/256/512
    'ExecutionEnvironment', 'gpu');

2. Enable Background Data Prefetching (DispatchInBackground)

Load and augment the next batch in CPU worker threads while the GPU trains on the current batch.

opts = trainingOptions('adam', ...
    'ExecutionEnvironment', 'gpu', ...
    'DispatchInBackground', true, ... % Asynchronous CPU prefetching
    'MiniBatchSize', 256);

3. Use Half-Precision (FP16) on Tensor Core GPUs

Modern NVIDIA GPUs (RTX series, V100, A100, H100) run significantly faster with 16-bit floating point precision and use half the VRAM.

% For trainNetwork / trainnet (R2023b+)
opts = trainingOptions('adam', ...
    'ExecutionEnvironment', 'gpu', ...
    'Precision', 'half'); % Enables FP16 acceleration

4. Disable Real-Time Training Plots and Heavy Callbacks

Drawing plots on every iteration pauses the execution pipeline and forces data transfers back to the CPU.

opts = trainingOptions('adam', ...
    'Plots', 'none', ...       % Turn off GUI rendering
    'Verbose', true, ...
    'VerboseFrequency', 50);   % Print metrics only every 50 iterations

5. Custom Training Loop: Use minibatchqueue with GPU Placement

If you write custom training loops with dlarray and dlgradient, use minibatchqueue with automatic background conversion to gpuArray.

% Configure high-throughput queue
mbq = minibatchqueue(ds, ...
    'MiniBatchSize', 256, ...
    'MiniBatchFcn', @preprocessBatch, ...
    'MiniBatchFormat', {'SSCB', 'BC'}, ...
    'DispatchInBackground', true, ...
    'OutputEnvironment', 'gpu'); % Streams directly to GPU memory

% Fast execution loop
while hasdata(mbq)
    [dlX, dlY] = next(mbq);
    [loss, gradients] = dlfeval(@modelGradients, net, dlX, dlY);
    [net, trailingAvg, trailingAvgSq] = adamupdate(net, gradients, ...
        trailingAvg, trailingAvgSq, iteration, learnRate);
end

6. Scale to Multiple GPUs

If your machine has multiple GPUs, distribute training across all devices with zero code changes.

opts = trainingOptions('adam', ...
    'ExecutionEnvironment', 'multi-gpu', ...
    'MiniBatchSize', 512); % Scales batch across all cards

Optimization Checklist