save_pretrained for serializing config.#11603
Merged
yiyixuxu merged 4 commits intomodular-refactorfrom Jun 19, 2025
Merged
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
Member
Author
|
@DN6 I pushed an initial version of the Provided you have implemented a diffusers-cli custom_blocksAnd then it should spit out: config.json
requirements.txt
block.py (should already be present)This is a rough proposal, happy to iterate further. |
Collaborator
|
thanks @sayakpaul, ok if I merge in first? I think erializes the config.json is sufficient for now |
Member
Author
|
@yiyixuxu let's go. |
yiyixuxu
added a commit
that referenced
this pull request
Jun 20, 2025
* add componentspec and configspec * up * up * move methods to blocks * Modular Diffusers Guiders (#11311) * cfg; slg; pag; sdxl without controlnet * support sdxl controlnet * support controlnet union * update * update * cfg zero* * use unwrap_module for torch compiled modules * remove guider kwargs * remove commented code * remove old guider * fix slg bug * remove debug print * autoguidance * smoothed energy guidance * add note about seg * tangential cfg * cfg plus plus * support cfgpp in ddim * apply review suggestions * refactor * rename enable/disable * remove cfg++ for now * rename do_classifier_free_guidance->prepare_unconditional_embeds * remove unused * [modular diffusers] introducing ModularLoader (#11462) * cfg; slg; pag; sdxl without controlnet --------- Co-authored-by: Aryan <aryan@huggingface.co> * make loader optional * remove lora step and ip-adapter step -> no longer needed * rename pipeline -> components, data -> block_state * seperate controlnet step into input + denoise * refactor controlnet union * reefactor pipeline/block states so that it can dynamically accept kwargs * remove controlnet union denoise step, refactor & reuse controlnet denoisee step to accept aditional contrlnet kwargs * allow input_fields as input & update message * update input formating, consider kwarggs_type inputs with no name, e/g *_controlnet_kwargs * refactor the denoiseestep using LoopSequential! also add a new file for denoise step * change warning to debug * fix get_execusion blocks with loopsequential * fix auto denoise so all tests pass * update imports on guiders * remove modular reelated change from pipelines folder * made a modular_pipelines folder! * update __init__ * add notes * add block state will also make sure modifed intermediates_inputs will be updated * move block mappings to its own file * make inputs truly immutable, remove the output logic in sequential pipeline, and update so that intermediates_outputs are only new variables * decode block, if skip decoding do not need to update latent * fix imports * fix import * fix more * remove the output step * make generator intermediates (it is mutable) * after_denoise -> decoders * add a to-do for guider cconfig mixin * refactor component spec: replace create/create_from_pretrained/create_from_config to just create and load method * refactor modular loader: 1. load only load (pretrained components only if not specific names) 2. update acceept create spec 3. move the updte _componeent_spec logic outside register_components to each method that create/update the component: __init__/update/load * update components manager * up * [WIP] Modular Diffusers support custom code/pipeline blocks (#11539) * update * update * remove the duplicated components_manager file I forgot to deletee * fix import in block mapping * add a to-do for modular loader * prepare_latents_img2img pipeline method -> function, maybe do the same for others? * update input for loop blocks, do not need to include intermediate * solve merge conflict: manually add back the remote code change to modular_pipeline * add node_utils * modular node! * add * refator based on dhruv's feedbacks * update doc format for kwargs_type * up * updatee modular_pipeline.from_pretrained, modular_repo ->pretrained_model_name_or_path * save_pretrained for serializing config. (#11603) * save_pretrained for serializing config. * remove pushtohub * diffusers-cli rough --------- Co-authored-by: YiYi Xu <yixu310@gmail.com> --------- Co-authored-by: Aryan <aryan@huggingface.co> Co-authored-by: Dhruv Nair <dhruv.nair@gmail.com> Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What does this PR do?
Follow-up of #11539.
Test code:
Currently, it serializes the
config.json. I think we agreed to manually save the block module that implements the custom block. It looks like so:diffusers-cli custom_blockscan also do the work. More details:#11603 (comment)
More about the CLI stuff
Currently, for custom blocks, we need the block implementation module (say
block.py), and theconfig.json, and an optionalrequirements.txt. @DN6 suggested a good idea over chat.We let the user implement
block.py(or any other module name) inside a folder. We then provide a CLI tool (diffusers-cli init-block ...) that would be run from the folder and spit out the following:So, the
config.jsonwill be created by the CLI command itself.There will be some reasonable defaults to serialize a desired
config.jsonfile but we will also expose certain CLI args to let the users take fine-grained control (would come in handy in case there are multiple block implementations and scripts).