Allow image resolutions multiple of 8 instead of 64 in SVD pipeline#6646

Merged

hlky merged 2 commits intohuggingface:mainfrom

mlfarinha:svd-other-resolutions

Dec 13, 2024

Contributor

mlfarinha commented Jan 20, 2024

Hello! This is my first time making a PR to diffusers so I apologise if I have missed something!

This PR implements the same behaviour from unet_2d_condition.py and unet_3d_condition.py to allow generating images with height/width that are not multiples of 64 while still being multiples of 8 in unet_spatio_temporal_condition.py. To achieve this I changed the forward method of the UNetSpatioTemporalConditionModel upblocks UpBlockSpatioTemporal and CrossAttnUpBlockSpatioTemporal in unet_3d_blocks.py. This PR is related to the closed issue #255

Code to reproduce the error:

import torch

from diffusers import StableVideoDiffusionPipeline
from diffusers.utils import load_image

pipe = StableVideoDiffusionPipeline.from_pretrained(
    "stabilityai/stable-video-diffusion-img2vid-xt", torch_dtype=torch.float16, variant="fp16"
)
pipe.enable_model_cpu_offload()

# Load the conditioning image
image = load_image("https://huggingface.co/datasets/huggingface/documentation-images/resolve/main/diffusers/svd/rocket.png")
image = image.resize((640, 480))

generator = torch.manual_seed(42)
frames = pipe(image, height=480, width=640, decode_chunk_size=8, generator=generator).frames[0]

Error:

RuntimeError: Sizes of tensors must match except in dimension 1. Expected size 16 but got size 15 for tensor number 1 in the list.

@patrickvonplaten and @sayakpaul


          allow resolutions not multiple of 64 in SVD

99fb2b4

sayakpaul requested review from DN6 and patil-suraj

January 20, 2024 02:06

Contributor

patrickvonplaten commented Jan 23, 2024

@DN6 @patil-suraj can you check here?

DN6 self-assigned this

Contributor

github-actions bot commented Feb 19, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions bot added the stale label

Member

sayakpaul commented Feb 19, 2024

@DN6 could you give this a look?

github-actions bot removed the stale label

Contributor

github-actions bot commented Mar 27, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions bot added the stale label

yiyixuxu removed the stale label

Collaborator

DN6 commented Mar 28, 2024

HI @mlfarinha sorry I missed this. PR looks good to me. Could we resolve the conflicts and we can merge.

Contributor

github-actions bot commented Apr 21, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions bot added the stale label

DN6 removed the stale label

Contributor

github-actions bot commented May 16, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions bot added the stale label

yiyixuxu removed the stale label

Contributor

github-actions bot commented Sep 14, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions bot added the stale label

yiyixuxu removed the stale label

Collaborator

yiyixuxu commented Sep 17, 2024

gentle pin @mlfarinha

Contributor

github-actions bot commented Oct 12, 2024

This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the contributing guidelines are likely to be ignored.

github-actions bot added the stale label

hlky mentioned this pull request

unet/unet_3d_blocks/.py - dimension size mismatche #10178

Closed


          Merge remote-tracking branch 'upstream/main' into svd-other-resolutions

4042f8e

HuggingFaceDocBuilderDev commented Dec 12, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

hlky approved these changes

View reviewed changes

hlky removed the stale label

hlky added the close-to-merge label

DN6 approved these changes

View reviewed changes

hlky merged commit 6bd30ba into huggingface:main

12 checks passed

sayakpaul pushed a commit that referenced this pull request


          Allow image resolutions multiple of 8 instead of 64 in SVD pipeline (#…

e34fb67

…6646)

allow resolutions not multiple of 64 in SVD

Co-authored-by: Miguel Farinha <mignha@CSL15958.local>
Co-authored-by: hlky <hlky@hlky.ac>

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels