Skip to content

[Community Pipeline] Add some feature for regional prompting pipeline#9874

Merged
sayakpaul merged 12 commits intohuggingface:mainfrom
cjkangme:regional-prompter
Nov 28, 2024
Merged

[Community Pipeline] Add some feature for regional prompting pipeline#9874
sayakpaul merged 12 commits intohuggingface:mainfrom
cjkangme:regional-prompter

Conversation

@cjkangme
Copy link
Contributor

@cjkangme cjkangme commented Nov 6, 2024

What does this PR do?

Hi.
I am happily using a pipeline that implements Regional Prompting in the diffusers community pipeline.
While using the community pipeline, I created a PR as I thought it would be good to add the following:

  • Fix negative prompt processing logic (minor error)
  • Add Use base prompt logic
  • Fix initialization errors occurring in the latest version

Using Example

I tried to imitate this example

2d animation style ADDBASE
masterpiece, high quality ADDCOMM
blue sky BREAK
green hair twintail BREAK
book shelf BREAK
messy desk BREAK
orange++ dress and sofa

mode: rows
div: 1,2,1,1;2,4,6
base_ratio: 0.3

output

Before submitting

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@sayakpaul
Copy link
Member

sayakpaul commented Nov 7, 2024

Thanks for your PR! Do you maybe also want to update the README of regional prompting?

@cjkangme
Copy link
Contributor Author

cjkangme commented Nov 8, 2024

@sayakpaul
sure, I have made updates regarding the use_base option

sayakpaul
sayakpaul reviewed Nov 8, 2024
View reviewed changes
examples/community/README.md
### Optional Parameters

- `save_mask`: In `Prompt` mode, choose whether to output the generated mask along with the image. The default is `False`.
- `base_ratio`: Used with `ADDBASE`. Sets the ratio of the base prompt; if base ratio is set to 0.2, then resulting images will consist of `20%*BASE_PROMPT + 80%*REGION_PROMPT`
Copy link
Member

@sayakpaul sayakpaul Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could maybe also specify the reference that introduced this technique?

Copy link
Contributor Author

@cjkangme cjkangme Nov 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This technique was originally implemented in hako-mikan's regional-prompter, but not in the current community pipeline.
So, I don't think it's necessary to specify the reference again.

@sayakpaul
Copy link
Member

sayakpaul commented Nov 8, 2024

@cjkangme could you run make style && make quality and push your changes?

@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Nov 8, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@cjkangme
Copy link
Contributor Author

cjkangme commented Nov 8, 2024

@sayakpaul
I accidentally ran make style in the root directory and it modified some wrong files. So I reverted it 😅😅
nothing changed about the regional prompting

@cjkangme
Copy link
Contributor Author

cjkangme commented Nov 13, 2024

@sayakpaul

Hi
I've reapplied the make style, can you confirm?

sayakpaul
sayakpaul approved these changes Nov 28, 2024
View reviewed changes
Copy link
Member

@sayakpaul sayakpaul left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you!

@sayakpaul sayakpaul merged commit 69c83d6 into huggingface:main Nov 28, 2024
@vladmandic
Copy link
Contributor

vladmandic commented Dec 14, 2024

was this pr tested? i'm getting runtime error since the merge:

│ /home/vlado/.cache/huggingface/modules/diffusers_modules/git/regional_prompting_stable_diffusion.py:462 in promptsmaker                                                                                                                                                                                                                                                                                                          │
│                                                                                                                                                                                                                                                                                                                                                                                                                                  │
│   461 │   │   │   add = add.strip() + " "                                                                                                                                                                                                                                                                                                                                                                                        │
│ ❱ 462 │   │   prompts = [p.strip() for p in prompt.split(KBRK)]                                                                                                                                                                                                                                                                                                                                                                  │
│   463 │   │   out_p.append([add + p for i, p in enumerate(prompts)])                                                                                                                                                                                                                                                                                                                                                             │
╰──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────╯
AttributeError: 'list' object has no attribute 'split'

@vladmandic vladmandic mentioned this pull request Dec 14, 2024
Closed
2 tasks
@cjkangme cjkangme mentioned this pull request Dec 17, 2024
Merged
6 tasks
@cjkangme
Copy link
Contributor Author

cjkangme commented Dec 17, 2024

@vladmandic

Hi
I apologize for the issue.
There was a typo in my code that caused an error when prompts is a str type and negative_prompts is a list type

I've fixed the affected line and created PR #10251

sayakpaul added a commit that referenced this pull request Dec 23, 2024
@cjkangme @sayakpaul
[Community Pipeline] Add some feature for regional prompting pipeline (…
82cd61d 
…#9874)

* [Fix] fix bugs of  regional_prompting pipeline

* [Feat] add base prompt feature

* [Fix] fix __init__ pipeline error

* [Fix] delete unused args

* [Fix] improve string handling

* [Docs] docs to use_base in regional_prompting

* make style

---------

Co-authored-by: Sayak Paul <spsayakpaul@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sayakpaul sayakpaul sayakpaul approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@cjkangme @sayakpaul @HuggingFaceDocBuilderDev @vladmandic