@haofrank thanks! any chance you or ur ai assistant can add the abiltty for only a couple of fwd passes. doing the whole time would create an large trace

@Oseltamivir was working on some of this too with outputting to github action artfacts or side repo and even have an perfetto relay for inferencemax (tho put on hold to work on evals)

https://github.com/Oseltamivir/profiler-storage

https://oseltamivir.github.io/profiler-storage/?src=https%3A%2F%2Fraw.githubusercontent.com%2FOseltamivir%2Fprofiler-storage%2F9fcdc64038c4f1e297e2ad629b25d4fa77ae975d%2Fprofiles%2F9313de6f1e694aa65314a8f83ab0e4983643b401%2Fh200%2Fsglang%2Fdsr1_1k1k_fp8_tp8_ep1_conc64%2Ftrace.json.gz&title=dsr1_1k1k_fp8_tp8_ep1_conc64

Thanks @haofrank, we already have a profiling branch at https://github.com/InferenceMAX/InferenceMAX/tree/profiling, where we tried to do automatic profiler analysis of the trace in order to give InferenceMAX another axis of information: How to improve.

However, it is currently stale as only hopper had usable information (kernel input dims, shapes, etc).

Nevertheless, having the ability to run the profiler may be useful. We will consider merging the profiler choice with the perfetto relay into main.

@haofrank If you would like to contribute to the repo, we would appreciate it if you could help update the branch at https://github.com/InferenceMAX/InferenceMAX/tree/profiling and PR. Else, I will probably do that this coming weekend.

Hi @Oseltamivir, thanks for the note! I’d be happy to help with this.

From a quick look, it seems the profiling branch already supports enabling profiling via the PROFILE env variable, but there are also quite a few other diffs in the branch. I’m not sure yet whether some of them were experimental or intended to be merged.

I’ll need a bit of time to go through and understand the intent. If I’m not able to make meaningful progress in time, it probably makes sense for you to proceed it.

Hi @haofrank, yep, that branch is very different in its intentions. If you could help:

1. Update the branch with main(container images, etc)
2. Add the profiling option to benchmark (Like this PR)
3. Profile decode & prefill
4. Enable a way to run with profiler by adding a profiler arg with .github/workflows/e2e-tests.yml and utils/matrix_logic/generate_sweep_configs.py
5. Enable artifact and relay, like how https://github.com/InferenceMAX/InferenceMAX/actions/runs/20731533868 has a perfetto relay link
6. Remove the python file that analyzes the results (original intent of profiling that went stale)

We value open source contributions and can definitly merge it in. The profiling branch works so you can assume env vars/relays are working.

However, I agree it might be difficult as there are quite a few parts involved. Let me know what you think, I'll work on this next week Wednesday if you're unable to make progress