Without aiter tkw1
VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ROCM_USE_AITER=0 VLLM_ROCM_USE_AITER_MOE=0 VLLM_ROCM_USE_AITER_RMSNORM=0 VLLM_ROCM_USE_AITER_LINEAR=0 SAFETENSORS_FAST_GPU=1 python benchmarks/benchmark_throughput.py --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 --dataset-name random --input-len 1000 --output-len 1000 -tp 8 --max-model-len 8192 --enforce-eager
Throughput: 6.47 requests/s, 13159.90 total tokens/s, 6468.73 output tokens/s

With aiter tkw1
VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1  VLLM_ROCM_USE_AITER_RMSNORM=0 VLLM_ROCM_USE_AITER_LINEAR=0 SAFETENSORS_FAST_GPU=1 python benchmarks/benchmark_throughput.py --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 --dataset-name random --input-len 1000 --output-len 1000 -tp 8 --max-model-len 8192 --enforce-eager
Throughput: 7.94 requests/s, 16143.27 total tokens/s, 7937.78 output tokens/s

Without aiter tkw1
VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ROCM_USE_AITER=0 VLLM_ROCM_USE_AITER_MOE=0 VLLM_ROCM_USE_AITER_RMSNORM=0 VLLM_ROCM_USE_AITER_LINEAR=0 python -m vllm.entrypoints.openai.api_server --max-model-len 30000 --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 -tp 8 --enforce-eager
============ Serving Benchmark Result ============
Successful requests:                     160
Benchmark duration (s):                  150.38
Total input tokens:                      160000
Total generated tokens:                  160000
Request throughput (req/s):              1.06
Output token throughput (tok/s):         1063.96
Total Token throughput (tok/s):          2127.93
---------------Time to First Token----------------
Mean TTFT (ms):                          268.25
Median TTFT (ms):                        153.78
P99 TTFT (ms):                           1199.93
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          29.79
Median TPOT (ms):                        29.78
P99 TPOT (ms):                           30.24
---------------Inter-token Latency----------------
Mean ITL (ms):                           29.79
Median ITL (ms):                         29.34
P99 ITL (ms):                            51.17
==================================================

With aiter tkw1
VLLM_USE_V1=1 VLLM_WORKER_MULTIPROC_METHOD=spawn VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1  VLLM_ROCM_USE_AITER_RMSNORM=0 VLLM_ROCM_USE_AITER_LINEAR=0 python -m vllm.entrypoints.openai.api_server --max-model-len 30000 --model meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8 -tp 8 --enforce-eager
============ Serving Benchmark Result ============
Successful requests:                     160
Benchmark duration (s):                  117.88
Total input tokens:                      160000
Total generated tokens:                  160000
Request throughput (req/s):              1.36
Output token throughput (tok/s):         1357.26
Total Token throughput (tok/s):          2714.52
---------------Time to First Token----------------
Mean TTFT (ms):                          191.74
Median TTFT (ms):                        138.49
P99 TTFT (ms):                           783.90
-----Time per Output Token (excl. 1st token)------
Mean TPOT (ms):                          23.37
Median TPOT (ms):                        23.41
P99 TPOT (ms):                           23.60
---------------Inter-token Latency----------------
Mean ITL (ms):                           23.37
Median ITL (ms):                         23.07
P99 ITL (ms):                            44.99
==================================================

meta-llama/Llama-4-Maverick-17B-128E-Instruct-FP8

Without aiter tkw1
Prompt: 'The color of the sky is blue but sometimes it can also be', Generated text: ' red, orange or grey. What is the reason behind the different colors of the sky? _ Tiwari Academy Discussion\nThe color of the sky is blue but sometimes it can also be red, orange or grey. What is the reason behind the different colors of the sky?\nThe color of the sky is primarily determined by the scattering of sunlight by the Earth_s atmosphere. The most common color we see is blue, and this is due to a phenomenon called Rayleigh scattering. Here_s why the sky appears blue and how other colors can manifest under different conditions:\n1. Blue Sky (Rayleigh Scattering):\n_ During the daytime when the sun'
Prompt: 'The capital of France is', Generated text: ' Paris. It is a major European city and a global center for art, fashion, and culture. Paris is known for its iconic landmarks such as the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum. The city is also famous for its cuisine, fashion, and romantic atmosphere. Paris is a popular tourist destination and is often referred to as the "City of Light" due to its role in the Enlightenment and its many famous intellectuals and artists. \nParis, the capital of France, is a city steeped in history, art, and culture. It is one of the most visited cities in the world,'
Prompt: 'What is batch inference?', Generated text: ' - Azure Machine Learning | Microsoft Learn Skip to main content \nWhat is batch inference?\nBatch inference, or batch scoring, is the process of generating predictions on a batch of observations. Batch inference or batch scoring is a common pattern for machine learning (ML) models in production environments. The batch inference process can be run on a recurring schedule or on-demand.\nBatch inference is a key component of an end-to-end ML solution. An end-to-end ML solution typically requires:\n0. Data preparation and preprocessing\n1. Model training\n2. Model evaluation\n3. Model deployment\n4. Batch inference\n5. Monitoring and retraining\n'

With aiter tkw1
Prompt: 'The color of the sky is blue but sometimes it can also be', Generated text: ' red, orange, or violet. What is the reason behind the different colors of the sky? _ Tiwari Academy Discussion\nThe color of the sky is blue but sometimes it can also be red, orange, or violet. What is the reason behind the different colors of the sky?\nThe color of the sky is primarily determined by the scattering of sunlight by the Earth_s atmosphere. The most common color we see is blue because blue light is scattered more than other colors by the molecules and small particles in the atmosphere. However, during sunrise and sunset, the sky can appear red, orange, or violet due to the following reasons:\n1'
Prompt: 'The capital of France is', Generated text: ' Paris. It is a major European city and a global center for art, fashion, and culture. Paris is known for its iconic landmarks such as the Eiffel Tower, Notre-Dame Cathedral, and the Louvre Museum. The city is also famous for its cuisine, fashion, and romantic atmosphere. Paris is a popular tourist destination and is often referred to as the _City of Light_ due to its role in the Enlightenment and its many famous intellectuals and artists. The city is divided into 20 arrondissements, or districts, each with its own unique character and charm. Paris is a must-visit destination for anyone interested'
Prompt: 'What is batch inference?', Generated text: " - Azure Machine Learning | Microsoft Learn Skip to main content \nWhat is batch inference?\nBatch inference, or batch scoring, is the process of generating predictions on a batch of observations. Batch inference or batch scoring is a common pattern for models that are trained offline. Batch inference can be used for both tabular data and unstructured data like images or text.\nBatch inference is typically used for offline scoring where the response time isn't critical. For example, a model that predicts energy demand for a utility company can be used to make predictions every hour, as the demand forecast is required only once per hour. In contrast, online inference is used for real-time scoring"