Add dynamic_shifting to SD3#10236
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
since there's not an integrated way in the sd35 pipeline to calculate to expand on @hlky example, with the suggested implementation this is what would be needed: from diffusers import FlowMatchEulerDiscreteScheduler
scheduler_config = {
'num_train_timesteps': 1000,
'shift': 3.0,
'use_dynamic_shifting': True, # <- note this
'base_shift': 0.5,
'max_shift': 1.15,
'base_image_seq_len': 256,
'max_image_seq_len': 4096,
'invert_sigmas': False,
}
scheduler = FlowMatchEulerDiscreteScheduler.from_config(scheduler_config)
pipe = StableDiffusion3Pipeline.from_pretrained(
...
scheduler=scheduler,
}
mu = calculate_shift(...) # your own shift function
image = pipe(prompt, ..., mu=mu).images[0]To calculate the shift you could use a strategy similar to what Flux does: def calculate_shift(
image_seq_len,
base_seq_len: int = 256,
max_seq_len: int = 4096,
base_shift: float = 0.5,
max_shift: float = 1.16,
):
m = (max_shift - base_shift) / (max_seq_len - base_seq_len)
b = base_shift - m * base_seq_len
mu = image_seq_len * m + b
return muBut there might be better strategies (maybe logarithmic?), that also raises the question if As a side note maybe a more robust solution would be to have a |
What does this PR do?
Adds
dynamic_shiftingto SD3.We can pass a custom
muvalueWhen not provided
muis calculated the same as in Flux.Euler
Img2Img
Inpaint
Heun
As expected Heun is unaffected by
use_dynamic_shift.Img2Img
Inpaint
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@yiyixuxu @cubiq