• update to the latest TRTLLM 1.2 release container
• fine-tune choice of parallelism in nvidia master (go to TP only for most points)
• Enable latest optimizations offered by trtllm

For most of the tests we switch to the TRTLLM backend for best performance.

As in the non-mtp fp8 agg code, we use Piecewise Cuda Graphs (https://nvidia.github.io/TensorRT-LLM/features/torch_compile_and_piecewise_cuda_graph.html) which enables some components to execute thorugh cuda graphs while other components are run eagerly, to gain benefit with lower overhead. We prepare the yaml configuration as per the documentation including a "capture_num_tokens" list based partly on MAX_NUM_TOKENS. Though we still exclude a few narrow-concurrency scenarios for performance reasons, we are working to improve this and will update this config once that is done.

For some of the higher-concurrency points we use data-parallel attention, through the DEEPGEMM MOE backend. This backend requires a few different optimizations vs TRTLLM as can be seen in lines 33-43. Particularly the flag ENABLE_CONFIGURABLE_MOE enables DEEPGEMM to use the MOE backend from the latest 1.3 code tree for its improved communication performance.