👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @czhu-cohere.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

Related Documentation

No published documentation to review for changes on this repository.
Write your first living document

^{How did I do? Any feedback?}  

Hey, thanks for the PR I just tried with the following scheme
{
"architectures": [
"LlamaForCausalLM"
],
"attention_bias": false,
"attention_dropout": 0.0,
"bos_token_id": 128000,
"eos_token_id": [
128001,
128008,
128009
],
"head_dim": 128,
"hidden_act": "silu",
"hidden_size": 4096,
"initializer_range": 0.02,
"intermediate_size": 14336,
"max_position_embeddings": 131072,
"mlp_bias": false,
"model_type": "llama",
"num_attention_heads": 32,
"num_hidden_layers": 32,
"num_key_value_heads": 8,
"pretraining_tp": 1,
"quantization_config": {
"config_groups": {
"group_0": {
"format": "int-quantized",
"input_activations": {
"actorder": null,
"block_structure": null,
"dynamic": true,
"group_size": null,
"num_bits": 8,
"observer": null,
"observer_kwargs": {},
"strategy": "token",
"symmetric": true,
"type": "int"
},
"output_activations": null,
"targets": [
"Linear"
],
"weights": {
"actorder": null,
"block_structure": null,
"dynamic": false,
"group_size": 128,
"num_bits": 4,
"observer": "minmax",
"observer_kwargs": {},
"strategy": "group",
"symmetric": true,
"type": "int"
}
}
},
"format": "int-quantized",
"global_compression_ratio": null,
"ignore": [
"lm_head"
],
"kv_cache_scheme": null,
"quant_method": "compressed-tensors",
"quantization_status": "compressed",
"sparsity_config": {},
"transform_config": {},
"version": "0.11.0"
},
"rms_norm_eps": 1e-05,
"rope_scaling": {
"factor": 8.0,
"high_freq_factor": 4.0,
"low_freq_factor": 1.0,
"original_max_position_embeddings": 8192,
"rope_type": "llama3"
},
"rope_theta": 500000.0,
"tie_word_embeddings": false,
"torch_dtype": "bfloat16",
"transformers_version": "4.55.2",
"use_cache": true,
"vocab_size": 128256
}

and got a key error

(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/v1/worker/gpu_worker.py", line 213, in load_model
(EngineCore_DP0 pid=30408) self.model_runner.load_model(eep_scale_up=eep_scale_up)
(EngineCore_DP0 pid=30408) ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/v1/worker/gpu_model_runner.py", line 2211, in load_model
(EngineCore_DP0 pid=30408) self.model = model_loader.load_model(
(EngineCore_DP0 pid=30408) ~~~~~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=30408) vllm_config=self.vllm_config, model_config=self.model_config)
(EngineCore_DP0 pid=30408) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/model_loader/base_loader.py", line 49, in load_model
(EngineCore_DP0 pid=30408) self.load_weights(model, model_config)
(EngineCore_DP0 pid=30408) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/model_loader/default_loader.py", line 263, in load_weights
(EngineCore_DP0 pid=30408) loaded_weights = model.load_weights(
(EngineCore_DP0 pid=30408) self.get_all_weights(model_config, model))
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/models/llama.py", line 607, in load_weights
(EngineCore_DP0 pid=30408) return loader.load_weights(
(EngineCore_DP0 pid=30408) ~~~~~~~~~~~~~~~~~~~^
(EngineCore_DP0 pid=30408) self.maybe_remap_mistral(name, loaded_weight)
(EngineCore_DP0 pid=30408) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=30408) for name, loaded_weight in weights)
(EngineCore_DP0 pid=30408) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/models/utils.py", line 291, in load_weights
(EngineCore_DP0 pid=30408) autoloaded_weights = set(self._load_module("", self.module, weights))
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/models/utils.py", line 249, in _load_module
(EngineCore_DP0 pid=30408) yield from self._load_module(prefix,
(EngineCore_DP0 pid=30408) child_modules[child_prefix],
(EngineCore_DP0 pid=30408) child_weights)
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/models/utils.py", line 222, in _load_module
(EngineCore_DP0 pid=30408) loaded_params = module_load_weights(weights)
(EngineCore_DP0 pid=30408) File "/teamspace/studios/this_studio/vllm/vllm/model_executor/models/llama.py", line 467, in load_weights
(EngineCore_DP0 pid=30408) param = params_dict[name]
(EngineCore_DP0 pid=30408) ~~~~~~~~~~~^^^^^^
(EngineCore_DP0 pid=30408) KeyError: 'layers.0.mlp.down_proj.weight'

any tips on what one would need to change to make it work for this w4a8 quantization scheme?