Skip to content

Several fixes to Flux ControlNet pipelines#9472

Merged
yiyixuxu merged 4 commits intohuggingface:mainfrom
vladmandic:main
Sep 20, 2024
Merged

Several fixes to Flux ControlNet pipelines#9472
yiyixuxu merged 4 commits intohuggingface:mainfrom
vladmandic:main

Conversation

@vladmandic
Copy link
Contributor

@vladmandic vladmandic commented Sep 19, 2024

Adds several fixes to Flux ControlNet pipelines

  • Add txt2img<->img2img<->inpaint automappings
  • Correct image encoding dtype as VAE may not be the same as main model (typically its not when using quantized model)
  • Add guidance handling to main txt pipeline (now inline with img2img and inpaint that already had this)

Closes #9398
Closes #9370

CC: @yiyixuxu @sayakpaul

@vladmandic vladmandic changed the title vladmandic Adds several fixes to Flux ControlNet pipelines Sep 19, 2024
@vladmandic vladmandic changed the title Adds several fixes to Flux ControlNet pipelines Several fixes to Flux ControlNet pipelines Sep 19, 2024
yiyixuxu
yiyixuxu approved these changes Sep 19, 2024
View reviewed changes
Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks!

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Sep 19, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@vladmandic
Copy link
Contributor Author

vladmandic commented Sep 19, 2024

updated pipelines list to be sorted due to failing check.

@yiyixuxu
Copy link
Collaborator

yiyixuxu commented Sep 20, 2024

@vladmandic
we have 2 commands that you can call to automatically format the code:make style and make quality - otherwise the CI wouldn't pass
just for future reference! I fixed the style already, will merge once the tests pass:)

@yiyixuxu yiyixuxu merged commit 14a1b86 into huggingface:main Sep 20, 2024
@hervenivon
Copy link

hervenivon commented Sep 20, 2024

Facing issues when using 'FluxMultiControlNetModel'.

A call to a pipeline leads to the following error: 'FluxMultiControlNetModel' object has no attribute 'config'

@hervenivon
Copy link

hervenivon commented Sep 23, 2024

Facing issues when using 'FluxMultiControlNetModel'.

A call to a pipeline leads to the following error: 'FluxMultiControlNetModel' object has no attribute 'config'

For those stumbling upon this message.

A simple set of the config property does the trick:

base_model = 'black-forest-labs/FLUX.1-dev'
controlnet_model_union = 'InstantX/FLUX.1-dev-Controlnet-Union'

controlnet_union = FluxControlNetModel.from_pretrained(controlnet_model_union, torch_dtype=torch.bfloat16)
controlnet = FluxMultiControlNetModel([controlnet_union]) # we always recommend loading via FluxMultiControlNetModel

# Trick
controlnet.config = controlnet_union.config

pipe = FluxControlNetPipeline.from_pretrained(base_model, controlnet=controlnet, torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = 'A bohemian-style female travel blogger with sun-kissed skin and messy beach waves.'
control_image_depth = load_image("https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union/resolve/main/images/depth.jpg")
control_mode_depth = 2

control_image_canny = load_image("https://huggingface.co/InstantX/FLUX.1-dev-Controlnet-Union/resolve/main/images/canny.jpg")
control_mode_canny = 0

width, height = control_image.size

image = pipe(
    prompt, 
    control_image=[control_image_depth, control_image_canny],
    control_mode=[control_mode_depth, control_mode_canny],
    width=width,
    height=height,
    controlnet_conditioning_scale=[0.2, 0.4],
    num_inference_steps=24, 
    guidance_scale=3.5,
    generator=torch.manual_seed(42),
).images[0]

@yiyixuxu yiyixuxu mentioned this pull request Sep 23, 2024
Merged
leisuzz pushed a commit to leisuzz/diffusers that referenced this pull request Oct 11, 2024
@vladmandic @yiyixuxu @leisuzz
Several fixes to Flux ControlNet pipelines (huggingface#9472)
e716cb2 
* fix flux controlnet pipelines

---------

Co-authored-by: yiyixuxu <yixu310@gmail.com>
sayakpaul pushed a commit that referenced this pull request Dec 23, 2024
@vladmandic @yiyixuxu @sayakpaul
Several fixes to Flux ControlNet pipelines (#9472)
e1ea942 
* fix flux controlnet pipelines

---------

Co-authored-by: yiyixuxu <yixu310@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiyixuxu yiyixuxu yiyixuxu approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

4 participants

@vladmandic @HuggingFaceDocBuilderDev @yiyixuxu @hervenivon