tests: fix cuda model tests. - #14464
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
| BaseModelTesterConfig, | ||
| MemoryTesterMixin, | ||
| ModelTesterMixin, | ||
| TorchCompileTesterMixin, |
There was a problem hiding this comment.
Not common to run compilation tests for autoencoders.
There was a problem hiding this comment.
I think we should consider having compilation tests for autoencoders in general (although maybe there's a good reason not to that I'm not aware of).
There was a problem hiding this comment.
Encoders and decoders are typically run only once and hence it doesn't make much sense to use compile and any potential latency improvements we might get from them. We can consider case by case and see if they deliver (i.e., a compile decode) a significant speedup but that is a separate PR. The exception here is if a decode (and encode) uses attention backends like Flex, then we must compile; otherwise it will be terribly slow and might even lead to OOMs.
Same reason why we don't typically compile the text encoder (or anything other than the denoiser for that matter).
I hope this makes sense. Thanks for asking the question.
| # the runtime base image only ships the libraries. | ||
| RUN apt install -y bash \ | ||
| build-essential \ | ||
| cuda-cudart-dev-12-9 \ |
There was a problem hiding this comment.
For running AOT tests in our CI:
https://github.com/huggingface/diffusers/actions/runs/31722536141
|
PEFT related failures are reported in huggingface/peft#3548 |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_kl_kvae_video.py |
|
✅ |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_kl_minimax_h3.py |
|
✅ |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_kl_minimax_h3_audio.py |
|
✅ |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_vidtok.py |
|
✅ |
|
/diffusers-bot pytest tests/models/transformers/test_models_transformer_cosmos3.py |
|
❌ |
|
/diffusers-bot pytest tests/models/transformers/test_models_transformer_z_image.py |
|
✅ |
dg845
left a comment
dg845
left a comment
There was a problem hiding this comment.
Thanks for the PR! I think we should also consider fixing other failing tests in the modified model test files. If I test locally with a CUDA device, I get the following test failures:
test_model_parallelism- Minimax H3 video VAE:
TestAutoencoderKLMiniMaxH3.test_model_parallelism - Cosmos 3 transformer:
TestCosmos3OmniTransformerModel.test_model_parallelism - Z-Image transformer:
TestZImageTransformer.test_model_parallelism
- Minimax H3 video VAE:
|
@dg845 I have replied to / addressed your comments. The parallelism failures target more than one GPUs, so, I will fix it in a follow-up. |
|
/diffusers-bot pytest tests/models/autoencoders/test_models_autoencoder_kl_kvae_video.py tests/models/autoencoders/test_models_autoencoder_kl_minimax_h3.py tests/models/autoencoders/test_models_autoencoder_kl_minimax_h3_audio.py tests/models/autoencoders/test_models_autoencoder_vidtok.py tests/models/transformers/test_models_transformer_cosmos3.py tests/models/transformers/test_models_transformer_z_image.py |
|
❌ |
|
The failing tests are related to AOT which are being fixed in this PR. The problem is that the Docker container the runner used was overwritten using the Dockerfile from |
3 participants
Fix: https://github.com/huggingface/diffusers/actions/runs/31675311029/job/94368495140 and https://github.com/huggingface/diffusers/actions/runs/31675311029/job/94368495111.
I have individually verified that the test files being changed in this PR are now running successfully on our CI (as seen in the bot invocations below).