Skip to content

fix bugs for sd3 controlnet training#9489

Merged
yiyixuxu merged 3 commits intohuggingface:mainfrom
xduzhangjiayu:fix_sd3_training
Sep 23, 2024
Merged

fix bugs for sd3 controlnet training#9489
yiyixuxu merged 3 commits intohuggingface:mainfrom
xduzhangjiayu:fix_sd3_training

Conversation

@xduzhangjiayu
Copy link
Contributor

@xduzhangjiayu xduzhangjiayu commented Sep 21, 2024

What does this PR do?

This PR is for fix some bugs for sd3 controlnet training

  1. fix issue for multi-gpu training (examples\controlnet\train_controlnet_sd3.py)
  2. fix issue for enabling --gradient_checkpointing in training process (src\diffusers\models\controlnet_sd3.py)

Who can review?

@yiyixuxu @sayakpaul @asomoza

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Sep 21, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

sayakpaul
sayakpaul approved these changes Sep 21, 2024
View reviewed changes
Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, looks good to me.

In a future PR, I will add a few more tests related to gradient checkpointing.

@sayakpaul sayakpaul requested a review from yiyixuxu September 21, 2024 02:06
@sayakpaul sayakpaul mentioned this pull request Sep 23, 2024
Merged
@xduzhangjiayu
Copy link
Contributor Author

xduzhangjiayu commented Sep 23, 2024

Hi, any comment on this PR ? @yiyixuxu

yiyixuxu
yiyixuxu approved these changes Sep 23, 2024
View reviewed changes
Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@yiyixuxu yiyixuxu merged commit bab1778 into huggingface:main Sep 23, 2024
leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request Oct 11, 2024
@xduzhangjiayu @sayakpaul
@yiyixuxu @leisuzz
fix bugs for sd3 controlnet training (huggingface#9489)
ccc1a62 
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
@tobiasfshr
Copy link

tobiasfshr commented Oct 11, 2024

Hi, I have a question on 1: Is having unused parameters expected behavior? The following parameters seem to be unused:

transformer_blocks.11.ff_context.net.2.bias, transformer_blocks.11.ff_context.net.2.weight, transformer_blocks.11.ff_context.net.0.proj.bias, transformer_blocks.11.ff_context.net.0.proj.weight, transformer_blocks.11.attn.to_add_out.bias, transformer_blocks.11.attn.to_add_out.weight

@Daryu-Fan
Copy link

Daryu-Fan commented Nov 1, 2024

sformer_blo

Hi, I meet the same error, do you solve it now?

@xduzhangjiayu
Copy link
Contributor Author

xduzhangjiayu commented Nov 1, 2024 •
edited
Loading

Hi, I have never met this error, could you please describe in detail, thanks!

@Daryu-Fan
Copy link

Daryu-Fan commented Nov 1, 2024

Hi, I have never met this error, could you please describe in detail, thanks!

error
Hi, I didn't change the code in diffusers, but the weight parameters in controlnets seem not be used to compute the final loss, but obviously, these weights have been initialized.

@xduzhangjiayu
Copy link
Contributor Author

xduzhangjiayu commented Nov 1, 2024

Hi, I have never met this error, could you please describe in detail, thanks!

error Hi, I didn't change the code in diffusers, but the weight parameters in controlnets seem not be used to compute the final loss, but obviously, these weights have been initialized.

Seems maybe I have met before, but I'm not sure, could you please contact me with email: xduzhangjiayu@163.com for further discussion ?

sayakpaul added a commit that referenced this pull request Dec 23, 2024
@xduzhangjiayu @sayakpaul @yiyixuxu
fix bugs for sd3 controlnet training (#9489)
c4e77f1 
Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Co-authored-by: YiYi Xu <yixu310@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiyixuxu yiyixuxu yiyixuxu approved these changes

@sayakpaul sayakpaul sayakpaul approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@xduzhangjiayu @HuggingFaceDocBuilderDev @tobiasfshr @Daryu-Fan @yiyixuxu @sayakpaul