Programmable switches for distributed LLM systems

Fusing computation and placing state inside the network fabric

Distributed LLM training and inference push data through the network at every layer. This project asks which work can move into the switch that the data already passes through, and when doing so removes more critical-path time than it adds.

SmartFuse-LLM: fusing computation into collectives

With tensor and sequence parallelism, every layer boundary does a reduce-scatter, a small residual add and RMSNorm on each GPU, and then an all-gather. SmartFuse-LLM moves the add and RMSNorm into an FPGA switch datapath, so the switch returns normalized activations directly and one of the two network round trips disappears. Paper (ACM DL)

SmartFuse-LLM Four GPUs send partial results to a programmable switch, which reduces them, applies the residual add and RMSNorm, and broadcasts the normalized result back: one network round trip per boundary instead of two. GPU 0 GPU 1 GPU 2 GPU 3 Programmable switch (FPGA datapath) Reduce Add residual RMSNorm Broadcast Partial results go in, normalized activations come out: one round trip instead of two
50.5%
projected lower latency per layer boundary
1
network round trip instead of 2

PoolStateLens: evaluating an on-switch memory pool for training state

Fully sharded training spreads parameters and optimizer state across GPUs, so memory capacity grows only by adding GPUs. A switch-attached memory pool, proposed in prior work, can hold that state instead: the switch multicasts parameters to GPUs, reduces gradients back, and an optimizer beside the pool applies the update. PoolStateLens is a measurement-grounded model, built on GB200 NVL72 measurements and ASTRA-sim checks, that asks how fast such a pool must be to keep up with FSDP. PDF

PoolStateLens GPUs keep only their active layer. An NVLink switch multicasts parameters from a switch-attached memory pool and reduces gradients back into it, where an optimizer updates parameters and optimizer state. GPU 0 active layer only GPU 1 active layer only GPU … active layer only GPU N active layer only NVLink switch fabric · multicast parameters, reduce gradients Switch-attached memory pool Parameters Optimizer state master weights, moments Adam parameters gradients
~7.9 TB/s
aggregate raw state bandwidth to match FSDP (modeled, controlled 7B, 32-GPU case)
GB200
NVL72 measurements ground the model

Next: stateful programmable switches

Ongoing work implements FPGA forward and backward datapaths and designs intermediate-state placement and workload-aware offload policies for LLM training and inference. This line of work, the core of my dissertation, builds on my SmartNIC offload for training work.

References

2026

  1. SmartFuse-LLM: Fusing Computation into Collectives for Efficient Tensor-Parallel LLM Systems
    Xiteng Yao, Bangjie Xue, Anthony Skjellum, and Martin Herbordt
    In Proceedings of the 16th International Symposium on Highly Efficient Accelerators and Reconfigurable Technologies (HEART), 2026
  2. PoolStateLens: Measurement-Grounded Evaluation of an On-Switch Memory Pool for LLM Training
    Xiteng Yao and Martin Herbordt
    In 2026 IEEE High Performance Extreme Computing Conference (HPEC), 2026