Skip to content

[core] LTX Video 0.9.1#10330

Merged
a-r-r-o-w merged 19 commits intomainfrom
ltxv-0.9.1-integration
Dec 23, 2024
Merged

[core] LTX Video 0.9.1#10330
a-r-r-o-w merged 19 commits intomainfrom
ltxv-0.9.1-integration

Conversation

@a-r-r-o-w
Copy link
Contributor

@a-r-r-o-w a-r-r-o-w commented Dec 21, 2024

To run conversion:

 python3 scripts/convert_ltx_to_diffusers.py --transformer_ckpt_path /raid/aryan/ltx-new/ltx-video-2b-v0.9.1.safetensors --vae_ckpt_path /raid/aryan/ltx-new/ltx-video-2b-v0.9.1.safetensors --output_path /raid/aryan/ltx-diffusers --dtype bf16 --version 0.9.1 --text_encoder_cache_dir /raid/.cache/huggingface/ --save_pipeline

(I've verified that the conversion for v0.9.0 still works after the current modifications to the script)

Inference after conversion:

import torch
from diffusers import LTXPipeline
from diffusers.utils import export_to_video

pipe = LTXPipeline.from_pretrained("/raid/aryan/ltx-diffusers", torch_dtype=torch.bfloat16)
pipe.to("cuda")

prompt = "A woman with long brown hair and light skin smiles at another woman with long blonde hair. The woman with brown hair wears a black jacket and has a small, barely noticeable mole on her right cheek. The camera angle is a close-up, focused on the woman with brown hair's face. The lighting is warm and natural, likely from the setting sun, casting a soft glow on the scene. The scene appears to be real-life footage"
negative_prompt = "worst quality, inconsistent motion, blurry, jittery, distorted"

video = pipe(
    prompt=prompt,
    negative_prompt=negative_prompt,
    width=704,
    height=480,
    num_frames=161,
    num_inference_steps=50,
    decode_timestep=0.05,
    generator=torch.Generator(device="cuda").manual_seed(0),
).frames[0]
export_to_video(video, "output.mp4", fps=24)

Output on prompts from the model page:

ltxv-091-output-downscaled.mp4

Will open weights PR to the official repository soon.

cc @yoavhacohen @SapirW

@a-r-r-o-w a-r-r-o-w requested a review from yiyixuxu December 21, 2024 04:25
@HuggingFaceDocBuilderDev
Copy link

HuggingFaceDocBuilderDev commented Dec 21, 2024

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

@a-r-r-o-w a-r-r-o-w mentioned this pull request Dec 21, 2024
Closed
@a-r-r-o-w a-r-r-o-w added the roadmap Add to current release roadmap label Dec 21, 2024
yiyixuxu
yiyixuxu reviewed Dec 21, 2024
View reviewed changes
Copy link
Collaborator

@yiyixuxu yiyixuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

that's fast! thanks!

@nitinmukesh nitinmukesh mentioned this pull request Dec 21, 2024
Closed
@tin2tin
Copy link

tin2tin commented Dec 22, 2024

Are the 0.9.1 diffusers weights up on HuggingFace?

@a-r-r-o-w
Copy link
Contributor Author

a-r-r-o-w commented Dec 22, 2024

@tin2tin We're still working with their team on how to host the weights. It might take some time :(

Until then, the model is available here as well: https://huggingface.co/a-r-r-o-w/LTX-Video-0.9.1-diffusers, mostly because I need this weight format for finetuning. Once we have it hosted officially, those can be used instead

@nitinmukesh
Copy link

nitinmukesh commented Dec 22, 2024

Until then, the model is available here as well: https://huggingface.co/a-r-r-o-w/LTX-Video-0.9.1-diffusers, mostly because I need this weight format for finetuning. Once we have it hosted officially, those can be used instead

Thank you for sharing.

@a-r-r-o-w
Copy link
Contributor Author

a-r-r-o-w commented Dec 22, 2024

cc @DN6 for single-file related support

@a-r-r-o-w a-r-r-o-w requested a review from DN6 December 22, 2024 12:14
@tin2tin
Copy link

tin2tin commented Dec 23, 2024 •
edited
Loading

Wow, this 0.9.1 (with prompt input) delivers very good quality video, and very fast!
First try:

-932937092_A_woman_with_long_brown_hair_and_light_.mp4

I guess img2vid is not implemented yet - I'm getting this error - sorry, if this is just premature to test this:

Python311\site-packages\diffusers\models\autoencoders\autoencoder_kl_ltx.py", line 431, in forward
    timestep=temb.flatten(),
             ^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'flatten'

@a-r-r-o-w
Copy link
Contributor Author

a-r-r-o-w commented Dec 23, 2024

Oh, taking a look... This PR should worked for I2V as well

@a-r-r-o-w
Copy link
Contributor Author

a-r-r-o-w commented Dec 23, 2024

@tin2tin Could you try again? Thanks for catching!

@tin2tin
Copy link

tin2tin commented Dec 23, 2024

@a-r-r-o-w That was quick! Yes, it is fixed now! Thank you!

1017385988_Photo_of_Photo_of_A_woman_with_long_bro.mp4

@a-r-r-o-w a-r-r-o-w requested a review from DN6 December 23, 2024 13:30
@a-r-r-o-w a-r-r-o-w merged commit 4b55713 into main Dec 23, 2024
15 checks passed
@a-r-r-o-w a-r-r-o-w deleted the ltxv-0.9.1-integration branch December 23, 2024 14:21
@scarbain
Copy link

scarbain commented Dec 23, 2024

@tin2tin We're still working with their team on how to host the weights. It might take some time :(

Until then, the model is available here as well: https://huggingface.co/a-r-r-o-w/LTX-Video-0.9.1-diffusers, mostly because I need this weight format for finetuning. Once we have it hosted officially, those can be used instead

Hi @a-r-r-o-w, thanks for this PR merged ! Any script for finetuning or LoRA i2v available somewhere ? :)

@a-r-r-o-w
Copy link
Contributor Author

a-r-r-o-w commented Dec 23, 2024

@scarbain There's one for T2V here: https://github.com/a-r-r-o-w/finetrainers. I2V will be supported soon!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@yiyixuxu yiyixuxu yiyixuxu approved these changes

@DN6 DN6 Awaiting requested review from DN6

Assignees

No one assigned

Labels

roadmap Add to current release roadmap

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

7 participants

@a-r-r-o-w @HuggingFaceDocBuilderDev @tin2tin @nitinmukesh @scarbain @DN6 @yiyixuxu