Skip to content

Add SkyReels-V2 pipelines #11518

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 48 commits into
base: main
Choose a base branch
from
+4,724 −0
Draft

Add SkyReels-V2 pipelines #11518

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
48 commits
Select commit Hold shift + click to select a range
4dd739f
Add SkyReels-V2 pipelines for text-to-video, image-to-video, and diff…
tolgacangoz May 7, 2025
899f41c
Merge branch 'main' into skyreels-v2
tolgacangoz May 7, 2025
607b5ba
up
tolgacangoz May 7, 2025
3ccf201
second draft
tolgacangoz May 8, 2025
959ca1f
Merge branch 'main' into skyreels-v2
tolgacangoz May 8, 2025
37ca14f
up
tolgacangoz May 8, 2025
d80b505
Merge branch 'main' into skyreels-v2
tolgacangoz May 8, 2025
95d0621
3rd draft
tolgacangoz May 8, 2025
6f8a945
4th draft
tolgacangoz May 9, 2025
e781084
upup
tolgacangoz May 9, 2025
4806660
style
tolgacangoz May 9, 2025
0986e81
up
tolgacangoz May 9, 2025
6a300f5
up
tolgacangoz May 9, 2025
45e1680
fix fn name
tolgacangoz May 9, 2025
c8a0c14
update import structure for SkyReelsV2
tolgacangoz May 9, 2025
47306b6
add SkyreelsV2 pipeline classes with backend requirements
tolgacangoz May 9, 2025
c5b8da9
up
tolgacangoz May 10, 2025
5835eaa
up
tolgacangoz May 10, 2025
9d2880e
add draft transformer_skyreels_v2.py with a custom WanModel and atten…
tolgacangoz May 10, 2025
2c0586e
up
tolgacangoz May 10, 2025
52590ea
split i2v and t2v pipes for diffusion forcing
tolgacangoz May 10, 2025
c50fcad
Refactors SkyReelsV2 attention and normalizations
tolgacangoz May 10, 2025
d0c71fd
Add SkyReelsV2 image and time-text embeddings
tolgacangoz May 11, 2025
f318efa
up
tolgacangoz May 11, 2025
9688a82
Refactors the `SkyReelsV2Transformer3DModel` by removing unused metho…
tolgacangoz May 11, 2025
825c2c1
Refactors `SkyReelsV2TransformerBlock` to integrate its `forward()` m…
tolgacangoz May 12, 2025
d848500
Refactors `SkyReelsV2AttnProcessor2_0` to enhance the `forward()` met…
tolgacangoz May 12, 2025
2f5a4e2
Refactors `SkyReelsV2Transformer3DModel` to enhance the `forward()` m…
tolgacangoz May 12, 2025
e5870dd
Refactors `SkyReelsV2Transformer3DModel` to improve the `forward()` m…
tolgacangoz May 12, 2025
d54e3e1
Refactors `SkyReelsV2Transformer3DModel` forward pass
tolgacangoz May 13, 2025
10d7480
Add DF inference template.
tolgacangoz May 13, 2025
fc68bf3
style
tolgacangoz May 13, 2025
1cb6a9e
Refactor `SkyReelsV2DiffusionForcingPipeline` to remove image process…
tolgacangoz May 13, 2025
ded93bc
Enhance `SkyReelsV2DiffusionForcingImageToVideoPipeline` by refining …
tolgacangoz May 13, 2025
c9483b2
Remove unused dtype handling in `SkyReelsV2DiffusionForcingPipeline` …
tolgacangoz May 13, 2025
f7fed01
up
tolgacangoz May 13, 2025
0e7b21d
up
tolgacangoz May 13, 2025
b3698d7
Update references
tolgacangoz May 13, 2025
7e0f0f5
Add `generate_timestep_matrix` method to `SkyReelsV2DiffusionForcingP…
tolgacangoz May 14, 2025
47080c2
Merge branch 'main' into skyreels-v2
tolgacangoz May 14, 2025
8c23208
Remove training-related code
tolgacangoz May 14, 2025
1f8e268
Add gradient checkpointing support in `SkyReelsV2Transformer3DModel` …
tolgacangoz May 14, 2025
d853521
Refactor `SkyReelsV2TransformerBlock` and remove unused `Head` class.…
tolgacangoz May 14, 2025
2b79584
Remove unused parameter `y` and associated documentation from `SkyRee…
tolgacangoz May 14, 2025
600ced3
Update context length calculation in `SkyReelsV2AttnProcessor2_0` to …
tolgacangoz May 14, 2025
586fe56
Fix comparison logic in `SkyReelsV2AttnProcessor2_0` to correctly det…
tolgacangoz May 14, 2025
afcaf6e
Remove unused `flex_attention` variable from `transformer_skyreels_v2…
tolgacangoz May 14, 2025
465df8c
Updates SkyReelsV2 pipeline defaults and docs
tolgacangoz May 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 12 additions & 0 deletions src/diffusers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -198,6 +198,7 @@
"SD3ControlNetModel",
"SD3MultiControlNetModel",
"SD3Transformer2DModel",
"SkyReelsV2Transformer3DModel",
"SparseControlNetModel",
"StableAudioDiTModel",
"StableCascadeUNet",
Expand Down Expand Up @@ -272,6 +273,7 @@
"FlowMatchEulerDiscreteScheduler",
"FlowMatchHeunDiscreteScheduler",
"FlowMatchLCMScheduler",
"FlowMatchUniPCMultistepScheduler",
"HeunDiscreteScheduler",
"IPNDMScheduler",
"KarrasVeScheduler",
Expand Down Expand Up @@ -445,6 +447,10 @@
"SemanticStableDiffusionPipeline",
"ShapEImg2ImgPipeline",
"ShapEPipeline",
"SkyreelsV2DiffusionForcingImageToVideoPipeline",
"SkyreelsV2DiffusionForcingPipeline",
"SkyreelsV2ImageToVideoPipeline",
"SkyreelsV2Pipeline",
"StableAudioPipeline",
"StableAudioProjectionModel",
"StableCascadeCombinedPipeline",
Expand Down Expand Up @@ -803,6 +809,7 @@
SD3ControlNetModel,
SD3MultiControlNetModel,
SD3Transformer2DModel,
SkyReelsV2Transformer3DModel,
SparseControlNetModel,
StableAudioDiTModel,
T2IAdapter,
Expand Down Expand Up @@ -875,6 +882,7 @@
FlowMatchEulerDiscreteScheduler,
FlowMatchHeunDiscreteScheduler,
FlowMatchLCMScheduler,
FlowMatchUniPCMultistepScheduler,
HeunDiscreteScheduler,
IPNDMScheduler,
KarrasVeScheduler,
Expand Down Expand Up @@ -1029,6 +1037,10 @@
SemanticStableDiffusionPipeline,
ShapEImg2ImgPipeline,
ShapEPipeline,
SkyreelsV2DiffusionForcingImageToVideoPipeline,
SkyreelsV2DiffusionForcingPipeline,
SkyreelsV2ImageToVideoPipeline,
SkyreelsV2Pipeline,
StableAudioPipeline,
StableAudioProjectionModel,
StableCascadeCombinedPipeline,
Expand Down
2 changes: 2 additions & 0 deletions src/diffusers/models/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -87,6 +87,7 @@
_import_structure["transformers.transformer_mochi"] = ["MochiTransformer3DModel"]
_import_structure["transformers.transformer_omnigen"] = ["OmniGenTransformer2DModel"]
_import_structure["transformers.transformer_sd3"] = ["SD3Transformer2DModel"]
_import_structure["transformers.transformer_skyreels_v2"] = ["SkyReelsV2Transformer3DModel"]
_import_structure["transformers.transformer_temporal"] = ["TransformerTemporalModel"]
_import_structure["transformers.transformer_wan"] = ["WanTransformer3DModel"]
_import_structure["unets.unet_1d"] = ["UNet1DModel"]
Expand Down Expand Up @@ -173,6 +174,7 @@
PriorTransformer,
SanaTransformer2DModel,
SD3Transformer2DModel,
SkyReelsV2Transformer3DModel,
StableAudioDiTModel,
T5FilmDecoder,
Transformer2DModel,
Expand Down
1 change: 1 addition & 0 deletions src/diffusers/models/transformers/__init__.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,5 +30,6 @@
from .transformer_mochi import MochiTransformer3DModel
from .transformer_omnigen import OmniGenTransformer2DModel
from .transformer_sd3 import SD3Transformer2DModel
from .transformer_skyreels_v2 import SkyReelsV2Transformer3DModel
from .transformer_temporal import TransformerTemporalModel
from .transformer_wan import WanTransformer3DModel
Loading