fix: missing AutoencoderKL lora adapter by beniz · Pull Request #9807 · huggingface/diffusers

beniz · 2024-10-30T07:43:20Z

What does this PR do?

This PR fixes the missing lora adapter with VAE (AutoencoderKL class).
Discussion is here: #9771
Related reports:
GaParmar/img2img-turbo#64
radames/Real-Time-Latent-Consistency-Model#38

Who can review?

cc @sayakpaul

sayakpaul

Thank you! Just a single comment.

src/diffusers/models/autoencoders/autoencoder_kl.py

tests/models/autoencoders/test_models_vae.py

HuggingFaceDocBuilderDev · 2024-11-06T01:50:01Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

yiyixuxu · 2024-11-06T02:19:12Z

can you run make style?

yiyixuxu · 2024-11-08T07:33:33Z

the failing tests are related, can we look into them?

I think we may need to add a @require_peft_backend similar to

diffusers/tests/models/unets/test_models_unet_2d_condition.py

Line 1081 in 5588725

def test_lora(self):

sayakpaul

Sorry I had actually forgot to submit my reviews.

sayakpaul · 2024-11-02T14:54:13Z

tests/models/autoencoders/test_models_vae.py


 from ..test_modeling_common import ModelTesterMixin, UNetTesterMixin
-
+from peft import LoraConfig


This needs to be guarded like this:

diffusers/tests/models/unets/test_models_unet_2d_condition.py

Line 56 in 13e8fde

if is_peft_available():

sayakpaul · 2024-11-02T14:54:37Z

tests/models/autoencoders/test_models_vae.py


        self.assertTrue(torch_all_close(output_slice, expected_output_slice, rtol=1e-2))

+    def test_lora_adapter(self):


This needs to be decorated with:

diffusers/tests/models/unets/test_models_unet_2d_condition.py

Line 1080 in 13e8fde

@require_peft_backend

@beniz seems like this was not resolved?

@sayakpaul ah apologies, may have forgot to push to repo. Done. Thanks for your vigilance.

beniz · 2024-11-11T14:32:15Z

the failing tests are related, can we look into them?

I think we may need to add a @require_peft_backend similar to

diffusers/tests/models/unets/test_models_unet_2d_condition.py

Line 1081 in 5588725

def test_lora(self):

All done @yiyixuxu I believe, let me know if anything else remains.

sayakpaul · 2024-11-29T02:14:10Z

@beniz thanks! Possible to fix the quality issues by running make style && make quality?

beniz · 2024-11-29T10:59:19Z

@beniz thanks! Possible to fix the quality issues by running make style && make quality?

I've fix a missing dependency. FYI running make style && qualilty fails early on other unrelated files, so I've been looking at the underlying ruff calls to apply them to the relevant files.

sayakpaul · 2024-11-29T11:28:21Z

@beniz I pushed the quality fixes directly to your branch, which I hope is okay. If not, please let me know, I will revert immediately

beniz · 2024-11-29T11:39:54Z

@beniz I pushed the quality fixes directly to your branch, which I hope is okay. If not, please let me know, I will revert immediately

Much appreciated, thank you.

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

…h bnb components (#9840) * allow device placement when using bnb quantization. * warning. * tests * fixes * docs. * require accelerate version. * remove print. * revert to() * tests * fixes * fix: missing AutoencoderKL lora adapter (#9807) * fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fixes * fix condition test * updates * updates * remove is_offloaded. * fixes * better * empty --------- Co-authored-by: Emmanuel Benazera <emmanuel.benazera@jolibrain.com>

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

…h bnb components (#9840) * allow device placement when using bnb quantization. * warning. * tests * fixes * docs. * require accelerate version. * remove print. * revert to() * tests * fixes * fix: missing AutoencoderKL lora adapter (#9807) * fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com> * fixes * fix condition test * updates * updates * remove is_offloaded. * fixes * better * empty --------- Co-authored-by: Emmanuel Benazera <emmanuel.benazera@jolibrain.com>

sayakpaul reviewed Oct 30, 2024

View reviewed changes

src/diffusers/models/autoencoders/autoencoder_kl.py Show resolved Hide resolved

beniz mentioned this pull request Oct 30, 2024

chore: diffusers update jolibrain/joliGEN#719

Merged

beniz force-pushed the fix-vae-lora branch from c4f243d to f4c8d6c Compare October 30, 2024 08:15

sayakpaul reviewed Oct 30, 2024

View reviewed changes

tests/models/autoencoders/test_models_vae.py Outdated Show resolved Hide resolved

beniz force-pushed the fix-vae-lora branch 4 times, most recently from 09cd44d to 9fb4880 Compare November 2, 2024 11:42

beniz force-pushed the fix-vae-lora branch from 9fb4880 to ca918e4 Compare November 6, 2024 13:31

sayakpaul reviewed Nov 8, 2024

View reviewed changes

beniz force-pushed the fix-vae-lora branch from 591af15 to ca918e4 Compare November 8, 2024 16:08

beniz force-pushed the fix-vae-lora branch from c7b60f5 to 76b2277 Compare November 28, 2024 21:18

fix: missing AutoencoderKL lora adapter

7025526

beniz force-pushed the fix-vae-lora branch from da99538 to 7025526 Compare November 29, 2024 08:26

sayakpaul added 2 commits November 29, 2024 16:47

Merge branch 'main' into fix-vae-lora

1a5e634

fix

ecda350

sayakpaul approved these changes Nov 29, 2024

View reviewed changes

sayakpaul requested a review from yiyixuxu November 29, 2024 11:48

sayakpaul added the close-to-merge label Nov 29, 2024

yiyixuxu merged commit 963ffca into huggingface:main Dec 3, 2024

sayakpaul added a commit that referenced this pull request Dec 3, 2024

fix: missing AutoencoderKL lora adapter (#9807)

1fe8a79

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

lawrence-cj pushed a commit to lawrence-cj/diffusers that referenced this pull request Dec 4, 2024

fix: missing AutoencoderKL lora adapter (huggingface#9807)

f379ac2

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>

sayakpaul added a commit that referenced this pull request Dec 23, 2024

fix: missing AutoencoderKL lora adapter (#9807)

3dd8aa4

* fix: missing AutoencoderKL lora adapter * fix --------- Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>


		from ..test_modeling_common import ModelTesterMixin, UNetTesterMixin

		from peft import LoraConfig


		self.assertTrue(torch_all_close(output_slice, expected_output_slice, rtol=1e-2))

		def test_lora_adapter(self):

Conversation

beniz commented Oct 30, 2024

What does this PR do?

Who can review?

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

HuggingFaceDocBuilderDev commented Nov 6, 2024

Uh oh!

yiyixuxu commented Nov 6, 2024

Uh oh!

yiyixuxu commented Nov 8, 2024

Uh oh!

sayakpaul left a comment

Choose a reason for hiding this comment

Uh oh!

sayakpaul Nov 2, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaul Nov 2, 2024

Choose a reason for hiding this comment

Uh oh!

sayakpaul Nov 28, 2024

Choose a reason for hiding this comment

Uh oh!

beniz Nov 28, 2024

Choose a reason for hiding this comment

Uh oh!

beniz commented Nov 11, 2024

Uh oh!

sayakpaul commented Nov 29, 2024

Uh oh!

beniz commented Nov 29, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sayakpaul commented Nov 29, 2024

Uh oh!

beniz commented Nov 29, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

beniz commented Nov 29, 2024 •

edited

Loading