Fusing computation and placing state inside the network fabric
Distributed LLM training and inference push data through the network at every layer. This project asks which work can move into the switch that the data already passes through, and when doing so removes more critical-path time than it adds.
SmartFuse-LLM: fusing computation into collectives
With tensor and sequence parallelism, every layer boundary does a reduce-scatter, a small residual add and RMSNorm on each GPU, and then an all-gather. SmartFuse-LLM moves the add and RMSNorm into an FPGA switch datapath, so the switch returns normalized activations directly and one of the two network round trips disappears. Paper (ACM DL)
50.5%
projected lower latency per layer boundary
1
network round trip instead of 2
PoolStateLens: evaluating an on-switch memory pool for training state
Fully sharded training spreads parameters and optimizer state across GPUs, so memory capacity grows only by adding GPUs. A switch-attached memory pool, proposed in prior work, can hold that state instead: the switch multicasts parameters to GPUs, reduces gradients back, and an optimizer beside the pool applies the update. PoolStateLens is a measurement-grounded model, built on GB200 NVL72 measurements and ASTRA-sim checks, that asks how fast such a pool must be to keep up with FSDP. PDF
~7.9 TB/s
aggregate raw state bandwidth to match FSDP (modeled, controlled 7B, 32-GPU case)
GB200
NVL72 measurements ground the model
Next: stateful programmable switches
Ongoing work implements FPGA forward and backward datapaths and designs intermediate-state placement and workload-aware offload policies for LLM training and inference. This line of work, the core of my dissertation, builds on my SmartNIC offload for training work.
SmartFuse-LLM is an FPGA-based switch datapath that fuses collective communication, residual addition, and RMSNorm at tensor-parallel boundaries, removing a network round trip and GPU-side normalization. GPU profiling and RTL co-simulation project 50.5% lower per-boundary latency.
@inproceedings{yao2026smartfusellm,title={{SmartFuse-LLM}: Fusing Computation into Collectives for Efficient Tensor-Parallel {LLM} Systems},author={Yao, Xiteng and Xue, Bangjie and Skjellum, Anthony and Herbordt, Martin},booktitle={Proceedings of the 16th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART)},publisher={ACM},pages={27--35},year={2026},doi={10.1145/3814576.3814596}}
A switch-attached memory pool can hold LLM training state (parameters, master weights, optimizer moments) so capacity grows independently of GPU count. PoolStateLens is a measurement-grounded analytical method that compares this design with FSDP over a full training iteration, using GB200/NVLink-5 measurements and ASTRA-sim checks. Pooling can reduce the GPU memory needed for training state, but matching FSDP speed requires about 7.9 TB/s of aggregate raw state bandwidth in a controlled 7B, 32-GPU case, under the model’s assumptions.
@inproceedings{yao2026poolstatelens,title={{PoolStateLens}: Measurement-Grounded Evaluation of an On-Switch Memory Pool for {LLM} Training},author={Yao, Xiteng and Herbordt, Martin},booktitle={2026 IEEE High Performance Extreme Computing Conference (HPEC)},year={2026},}