LLM performance modeling and AI for performance engineering

Predicting LLM inference performance, and measuring what helps code-optimizing agents

Two questions run through this work: can we predict how an LLM deployment will perform before running it, and when an LLM agent or a learned search optimizes code, what actually helps it?

KernelSight-LM: predicting LLM inference performance

Built during my AWS internship. KernelSight-LM extends the Vidur serving simulator and combines models of GPU kernel latency, communication, and host overhead in a discrete-event scheduler that replays batching and KV-cache behavior, predicting end-to-end latency and throughput for hardware/software co-design and design-space exploration. Paper (arXiv)

KernelSight-LM Kernel-latency, communication and host-overhead models feed a discrete-event scheduler that predicts end-to-end LLM inference latency and throughput. Kernel latency model profiled GPU kernels Communication model collectives Host overhead model launch, scheduling Discrete-event scheduler batching, KV cache Predicted TTFT · TPOT · tput
2.7%
median throughput error, target-measured tier
6
model families evaluated

ACBench: what helps LLM agents optimize code

ACBench turns merged performance pull requests from six production Go codebases into 14 tasks. An agent proposes patches over several rounds; each patch is applied, tested, and benchmarked against the human fix. Because a retry gives the model both new information and another chance, ACBench compares error-guided and generic retry at an equal proposal budget to separate the two. PDF · Code

ACBench A task from a real merged pull request goes to an LLM agent, whose patch is applied, tested and benchmarked against the human patch. A feedback policy decides whether to retry and what information to send back. Real PR task production Go code LLM agent proposes patch Apply + test benchmark Score vs. human patch Feedback policy retry · error message · profile result retry with feedback What helps: the extra attempt, or the information?

RL-guided GPU compiler optimization on NVIDIA Blackwell

A runtime reward tells a reinforcement-learning search whether a GPU kernel got faster, but not why. This work studies RL-guided optimization of Triton kernels on NVIDIA Blackwell, looking at how changes carry through the compiler to the final machine code, to understand when and why an optimization helps. PDF

RL-guided GPU compiler optimization An RL-guided search proposes changes to a Triton kernel's compilation, from compiler passes to the final machine code; each change is traced through the compiler stages and timed on an NVIDIA GB200. Triton TTGIR LLVM IR PTX SASS GB200 Trace each change through the compiler RL-guided search proposes changes compiler passes machine code runtime

References

2026

  1. arXiv
    KernelSight-LM: A Kernel-Level LLM Inference Simulator
    KernelSight-LM: A Kernel-Level LLM Inference Simulator
    Xiteng Yao, Taeho Kim, Hengzhi Pei, Xinle Liu, Kyle Ulrich, Leonard Lausen, Ashish Khetan, Xiang Song, George Karypis, and Martin Herbordt
    arXiv preprint arXiv:2606.28565, 2026
  2. ACBench: A Benchmark Suite for Auditing Retry and Profiling Feedback in LLM-Driven Code Optimization
    Xiteng Yao*, Ziyuan Chu*, Feng Tai*, Bangjie Xue, Yigong Hu, and Martin Herbordt
    In 2026 IEEE High Performance Extreme Computing Conference (HPEC), 2026
  3. Toward RL-Guided GPU Compiler Optimization: Case Studies on Nvidia Blackwell GPUs
    Xiteng Yao and Martin Herbordt
    In 2026 IEEE High Performance Extreme Computing Conference (HPEC), 2026