SmartNIC offload for large-model training

Overlapping ZeRO communication with GPU and CPU computation

ZeRO-style training repeatedly gathers parameters from other GPUs and offloads state to host memory, so the GPU often waits on data movement. This work puts a SmartNIC in that data path: it prefetches and buffers parameters from the network and host memory and schedules their delivery to the GPU, overlapping communication with GPU and CPU computation and reducing duplicate parameter gathers. Paper (ACM DL)

SmartNIC–GPU–CPU co-design Inside a training node, a SmartNIC prefetches and buffers parameters from the network and host memory and schedules them to the GPU, overlapping communication with computation. Training node GPU forward / backward CPU + host memory offloaded state SmartNIC prefetch · buffer · schedule Network other nodes parameters state
>1.6×
projected training throughput, 100B-parameter model
ICS 2025
ACM International Conference on Supercomputing

References

2025

  1. ICS
    SmartNIC-GPU-CPU Heterogeneous System for Large Machine Learning Model with Software-Hardware Codesign
    SmartNIC-GPU-CPU Heterogeneous System for Large Machine Learning Model with Software-Hardware Codesign
    Anqi Guo, Yuchen Hao, Xiteng Yao, Shining Yang, Jianyu Huang, Tong Geng, and Martin Herbordt
    In Proceedings of the 39th ACM International Conference on Supercomputing (ICS), 2025