[Perf] Tune configs for triton block fp8 gemm H100/H200 #23748
Conversation
Signed-off-by: mgoin <mgoin64@gmail.com>
mgoin
requested review from
robertgshaw2-redhat,
tlrmchlsmth and
yewentao256
as code owners
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces performance tuning configurations for the FP8 block-wise GEMM kernel on H100/H200 GPUs, specifically for matrix shapes found in the DeepSeek-V3 model. It includes a new benchmark script to compare the performance of the w8a8-block-fp8 kernel against the standard bfloat16 GEMM, along with numerous new and updated JSON files containing the tuned kernel parameters. The changes appear to be correct and well-aligned with the goal of improving performance. The new benchmark script is well-implemented, and the configuration files are consistent with the output of a tuning process. I have not identified any issues of high or critical severity.
yewentao256
left a comment
yewentao256
left a comment
There was a problem hiding this comment.
LGTM, thanks for the work!
mgoin
changed the title
…#23748) Signed-off-by: mgoin <mgoin64@gmail.com>
…#23748) Signed-off-by: mgoin <mgoin64@gmail.com>
…#23748) Signed-off-by: mgoin <mgoin64@gmail.com>
…#23748) Signed-off-by: mgoin <mgoin64@gmail.com>
3 participants
Purpose
Retune the triton fp8 block dense gemm configs for modern triton. Also adds a simple benchmark script that doesn't tune.
Mostly improves performance for smaller M, but crucially gives improvement for
N=576,K=7168Test Plan
Test Result
H100
H200
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.