LLM performance modeling and AI for performance engineering
Predicting LLM inference performance, and measuring what helps code-optimizing agents
Two questions run through this work: can we predict how an LLM deployment will perform before running it, and when an LLM agent or a learned search optimizes code, what actually helps it?
Built during my AWS internship. KernelSight-LM extends the Vidur serving simulator and combines models of GPU kernel latency, communication, and host overhead in a discrete-event scheduler that replays batching and KV-cache behavior, predicting end-to-end latency and throughput for hardware/software co-design and design-space exploration. Paper (arXiv)
2.7%
median throughput error, target-measured tier
6
model families evaluated
ACBench: what helps LLM agents optimize code
ACBench turns merged performance pull requests from six production Go codebases into 14 tasks. An agent proposes patches over several rounds; each patch is applied, tested, and benchmarked against the human fix. Because a retry gives the model both new information and another chance, ACBench compares error-guided and generic retry at an equal proposal budget to separate the two. PDF · Code
RL-guided GPU compiler optimization on NVIDIA Blackwell
A runtime reward tells a reinforcement-learning search whether a GPU kernel got faster, but not why. This work studies RL-guided optimization of Triton kernels on NVIDIA Blackwell, looking at how changes carry through the compiler to the final machine code, to understand when and why an optimization helps. PDF
KernelSight-LM is a GPU-kernel-aware profiling and simulation toolchain that combines kernel-latency, communication, and host-overhead models in a discrete-event scheduler to predict end-to-end LLM inference latency and throughput. Across six model families it reaches 2.7% median throughput-prediction error in the target-measured setting.
@article{yao2026kernelsight,title={{KernelSight-LM}: A Kernel-Level {LLM} Inference Simulator},author={Yao, Xiteng and Kim, Taeho and Pei, Hengzhi and Liu, Xinle and Ulrich, Kyle and Lausen, Leonard and Khetan, Ashish and Song, Xiang and Karypis, George and Herbordt, Martin},journal={arXiv preprint arXiv:2606.28565},year={2026},}
ACBench is a benchmark of 14 code-optimization tasks drawn from merged pull requests in six production Go codebases. It separates the value of feedback (application errors, benchmark summaries, CPU profiles) from the value of simply getting more attempts, and tracks when a run can actually use that feedback.
@inproceedings{yao2026acbench,title={{ACBench}: A Benchmark Suite for Auditing Retry and Profiling Feedback in {LLM}-Driven Code Optimization},author={Yao, Xiteng and Chu, Ziyuan and Tai, Feng and Xue, Bangjie and Hu, Yigong and Herbordt, Martin},booktitle={2026 IEEE High Performance Extreme Computing Conference (HPEC)},year={2026},}
Case studies of reinforcement-learning-guided GPU compiler optimization on NVIDIA Blackwell, combining runtime measurements with evidence from each compilation stage to understand when and why an optimization helps.
@inproceedings{yao2026rlgpucompiler,title={Toward {RL}-Guided {GPU} Compiler Optimization: Case Studies on {Nvidia} {Blackwell} {GPUs}},author={Yao, Xiteng and Herbordt, Martin},booktitle={2026 IEEE High Performance Extreme Computing Conference (HPEC)},year={2026},}