ControlNet from_single_file when already converted#9978
Merged
DN6 merged 2 commits intohuggingface:mainfrom Nov 22, 2024
Merged
ControlNet from_single_file when already converted#9978DN6 merged 2 commits intohuggingface:mainfrom
DN6 merged 2 commits intohuggingface:mainfrom
Conversation
Contributor
|
nice! did a quick test and seems fine. |
Collaborator
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
DN6
approved these changes
Nov 22, 2024
Collaborator
|
Thanks @hlky. Would it be possible to do a follow up for the ControlNetXL small version? |
Contributor
Author
|
Small, mid and large/full versions of ControlNet XL should all be covered, I've tested a couple checkpoints for each. I haven't found any ControlNet XL checkpoints that aren't already in Diffusers format but I'll add non-Diffusers keys to check in a follow up, and I'll check for something specific to detect small version - currently small is selected if known keys in medium and large versions aren't detected. import torch
from diffusers import ControlNetModel
from huggingface_hub import hf_hub_download
small = 'diffusers/controlnet-canny-sdxl-1.0-small'
mid = 'diffusers/controlnet-canny-sdxl-1.0-mid'
large = 'diffusers/controlnet-canny-sdxl-1.0'
for size in {small, mid, large}:
local_path = hf_hub_download(repo_id=size, filename='diffusion_pytorch_model.fp16.safetensors')
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
small = 'diffusers_xl_canny_small.safetensors'
mid = 'diffusers_xl_canny_mid.safetensors'
large = 'diffusers_xl_canny_full.safetensors'
for size in {small, mid, large}:
local_path = hf_hub_download(repo_id="lllyasviel/sd_control_collection", filename=size)
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
# xl large/full
local_path = hf_hub_download(repo_id="Laxhar/noob_openpose", filename="openpose_pre.safetensors")
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__)
# v1
local_path = hf_hub_download(repo_id="lllyasviel/sd-controlnet-canny", filename="diffusion_pytorch_model.safetensors")
cn = ControlNetModel.from_single_file(local_path, torch_dtype=torch.float16)
print(cn.__class__) |
lawrence-cj
pushed a commit
to lawrence-cj/diffusers
that referenced
this pull request
Nov 26, 2024
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
sayakpaul
pushed a commit
that referenced
this pull request
Dec 23, 2024
Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
5 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
This PR fixes #9976
ControlNet
from_single_fileis broken when used with a checkpoint that has already been converted:diffusers/src/diffusers/loaders/single_file_utils.py
Line 457 in f6f7afa
diffusers/src/diffusers/loaders/single_file_utils.py
Line 65 in f6f7afa
The state dict is incorrectly being detected as
v1Stable Diffusion because the checked keys forcontrolnetare for non-Diffusers format.We add to checked keys for
controlnetand addcontrolnet_xl_large,controlnet_xl_midandcontrolnet_xl_smallas the final types withcontrolnet_xlbeing used internally as a key to check for XL vs v1.controlnet_xlkeys are for Diffusers type checkpoints and need equivalents for non-converted checkpoints.When fixed, or if we provide the correct config:
convert_controlnet_checkpointcalled as part offrom_single_fileexpects non-Diffusers type, the conversion function runs without error and returns an unusable state dict.convert_controlnet_checkpointis modified to return the checkpoint if it's already been converted.Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@sayakpaul @vladmandic