Overlapping ZeRO communication with GPU and CPU computation
ZeRO-style training repeatedly gathers parameters from other GPUs and offloads state to host memory, so the GPU often waits on data movement. This work puts a SmartNIC in that data path: it prefetches and buffers parameters from the network and host memory and schedules their delivery to the GPU, overlapping communication with GPU and CPU computation and reducing duplicate parameter gathers. Paper (ACM DL)
>1.6×
projected training throughput, 100B-parameter model
A SmartNIC-based design for ZeRO training that uses parameter prefetching, buffering, and dynamic scheduling to overlap GPU/CPU computation with communication. Hardware-calibrated cycle-accurate simulation projects over 1.6× training throughput for a 100B-parameter model versus GPU-CPU offload with ordinary NICs.
@inproceedings{guo2025sgc,title={{SmartNIC-GPU-CPU} Heterogeneous System for Large Machine Learning Model with Software-Hardware Codesign},author={Guo, Anqi and Hao, Yuchen and Yao, Xiteng and Yang, Shining and Huang, Jianyu and Geng, Tong and Herbordt, Martin},booktitle={Proceedings of the 39th ACM International Conference on Supercomputing (ICS)},publisher={ACM},pages={837--852},year={2025},doi={10.1145/3721145.3729514}}