Skip to content

Modify the implementation of retrieve_timesteps in CogView4-Control. #11125

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 56 commits into from
Mar 23, 2025
Merged
Changes from all commits
Commits
Show all changes
56 commits
Select commit Hold shift + click to select a range
a97fca2
1
zRzRzRzRzRzRzR Feb 17, 2025
c30ca7a
change to channel 1
zRzRzRzRzRzRzR Feb 17, 2025
5c25cd2
cogview4 control training
zRzRzRzRzRzRzR Feb 18, 2025
44bfd4c
add CacheMixin
zRzRzRzRzRzRzR Feb 18, 2025
a9f448e
1
zRzRzRzRzRzRzR Feb 18, 2025
2cbdf35
remove initial_input_channels change for val
zRzRzRzRzRzRzR Feb 18, 2025
df83bf2
1
zRzRzRzRzRzRzR Feb 18, 2025
8bba67a
update
zRzRzRzRzRzRzR Feb 18, 2025
b9d864b
use 3.5
zRzRzRzRzRzRzR Feb 18, 2025
5d2e994
new loss
zRzRzRzRzRzRzR Feb 18, 2025
ebeb1e4
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Feb 19, 2025
95e8504
1
zRzRzRzRzRzRzR Feb 19, 2025
940c23b
Merge branch 'cogview4_control' of https://github.com/zRzRzRzRzRzRzR/…
zRzRzRzRzRzRzR Feb 19, 2025
7a68a3e
use imagetoken
zRzRzRzRzRzRzR Feb 19, 2025
2a81772
for megatron convert
zRzRzRzRzRzRzR Feb 19, 2025
1d91a24
1
zRzRzRzRzRzRzR Feb 19, 2025
dff4b29
train con and uc
zRzRzRzRzRzRzR Feb 19, 2025
050b97c
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Feb 19, 2025
b007be0
2
zRzRzRzRzRzRzR Feb 19, 2025
25f4e4b
remove guidance_scale
zRzRzRzRzRzRzR Feb 20, 2025
7ffecbc
Update pipeline_cogview4_control.py
zRzRzRzRzRzRzR Feb 21, 2025
b4e11e7
fix
zRzRzRzRzRzRzR Feb 21, 2025
efa0f41
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Feb 21, 2025
f55e3cc
use cogview4 pipeline with timestep
zRzRzRzRzRzRzR Feb 21, 2025
9410e46
Merge branch 'cogview4_control' of https://github.com/zRzRzRzRzRzRzR/…
zRzRzRzRzRzRzR Feb 21, 2025
29b0c81
update shift_factor
zRzRzRzRzRzRzR Feb 24, 2025
52d4ebf
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Feb 25, 2025
65b3719
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Feb 26, 2025
90830ed
remove the uncond
zRzRzRzRzRzRzR Feb 26, 2025
71f9235
add max length
zRzRzRzRzRzRzR Feb 26, 2025
19d7d27
change convert and use GLMModel instead of GLMForCasualLM
zRzRzRzRzRzRzR Feb 27, 2025
fe6287a
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Feb 27, 2025
2f74c4e
fix
zRzRzRzRzRzRzR Feb 27, 2025
264060e
[cogview4] Add attention mask support to transformer model
OleehyO Feb 28, 2025
9a10ceb
[fix] Add attention mask for padded token
OleehyO Mar 4, 2025
b6e10e7
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Mar 5, 2025
692e5cc
update
zRzRzRzRzRzRzR Mar 5, 2025
fc3830c
remove padding type
zRzRzRzRzRzRzR Mar 6, 2025
98a2417
Update train_control_cogview4.py
zRzRzRzRzRzRzR Mar 6, 2025
c774f45
resolve conflicts with #10981
zRzRzRzRzRzRzR Mar 12, 2025
687faa4
Merge branch 'main' into cogview4_control
zRzRzRzRzRzRzR Mar 12, 2025
8abca19
add control convert
zRzRzRzRzRzRzR Mar 12, 2025
cbfeb0b
Merge branch 'cogview4_control' of https://github.com/zRzRzRzRzRzRzR/…
zRzRzRzRzRzRzR Mar 12, 2025
347dd17
use control format
zRzRzRzRzRzRzR Mar 13, 2025
775bb8c
fix
zRzRzRzRzRzRzR Mar 13, 2025
985baa9
add missing import
zRzRzRzRzRzRzR Mar 13, 2025
c2a1985
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Mar 15, 2025
88abb39
update with cogview4 formate
zRzRzRzRzRzRzR Mar 15, 2025
3e3387e
make style
yiyixuxu Mar 15, 2025
ddb31d3
Merge branch 'huggingface:main' into cogview4_control
zRzRzRzRzRzRzR Mar 20, 2025
64637ef
Update pipeline_cogview4_control.py
zRzRzRzRzRzRzR Mar 20, 2025
4174736
Update pipeline_cogview4_control.py
zRzRzRzRzRzRzR Mar 21, 2025
8cdd36f
remove
zRzRzRzRzRzRzR Mar 21, 2025
785e230
Update pipeline_cogview4_control.py
zRzRzRzRzRzRzR Mar 22, 2025
07ef22e
put back
zRzRzRzRzRzRzR Mar 23, 2025
1e93e98
Apply style fixes
github-actions[bot] Mar 23, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 17 additions & 10 deletions src/diffusers/pipelines/cogview4/pipeline_cogview4_control.py
Original file line number Diff line number Diff line change
Expand Up @@ -68,7 +68,7 @@ def calculate_shift(
return mu


# Copied from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion.retrieve_timesteps
# Copied from diffusers.pipelines.cogview4.pipeline_cogview4.retrieve_timesteps
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just the change here is correct @zRzRzRzRzRzRzR

def retrieve_timesteps(
scheduler,
num_inference_steps: Optional[int] = None,
Expand Down Expand Up @@ -100,10 +100,19 @@ def retrieve_timesteps(
`Tuple[torch.Tensor, int]`: A tuple where the first element is the timestep schedule from the scheduler and the
second element is the number of inference steps.
"""
accepts_timesteps = "timesteps" in set(inspect.signature(scheduler.set_timesteps).parameters.keys())
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update the # Copied from comment to point to CogView4 pipeline file

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is that how I understand it?

accepts_sigmas = "sigmas" in set(inspect.signature(scheduler.set_timesteps).parameters.keys())

if timesteps is not None and sigmas is not None:
raise ValueError("Only one of `timesteps` or `sigmas` can be passed. Please choose one to set custom values")
if timesteps is not None:
accepts_timesteps = "timesteps" in set(inspect.signature(scheduler.set_timesteps).parameters.keys())
if not accepts_timesteps and not accepts_sigmas:
raise ValueError(
f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
f" timestep or sigma schedules. Please check whether you are using the correct scheduler."
)
scheduler.set_timesteps(timesteps=timesteps, sigmas=sigmas, device=device, **kwargs)
timesteps = scheduler.timesteps
num_inference_steps = len(timesteps)
elif timesteps is not None and sigmas is None:
if not accepts_timesteps:
raise ValueError(
f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
Expand All @@ -112,9 +121,8 @@ def retrieve_timesteps(
scheduler.set_timesteps(timesteps=timesteps, device=device, **kwargs)
timesteps = scheduler.timesteps
num_inference_steps = len(timesteps)
elif sigmas is not None:
accept_sigmas = "sigmas" in set(inspect.signature(scheduler.set_timesteps).parameters.keys())
if not accept_sigmas:
elif timesteps is None and sigmas is not None:
if not accepts_sigmas:
raise ValueError(
f"The current scheduler class {scheduler.__class__}'s `set_timesteps` does not support custom"
f" sigmas schedules. Please check whether you are using the correct scheduler."
Expand Down Expand Up @@ -515,8 +523,8 @@ def __call__(
The output format of the generate image. Choose between
[PIL](https://pillow.readthedocs.io/en/stable/): `PIL.Image.Image` or `np.array`.
return_dict (`bool`, *optional*, defaults to `True`):
Whether or not to return a [`~pipelines.stable_diffusion_xl.StableDiffusionXLPipelineOutput`] instead
of a plain tuple.
Whether or not to return a [`~pipelines.pipeline_CogView4.CogView4PipelineOutput`] instead of a plain
tuple.
attention_kwargs (`dict`, *optional*):
A kwargs dictionary that if specified is passed along to the `AttentionProcessor` as defined under
`self.processor` in
Expand All @@ -532,7 +540,6 @@ def __call__(
`._callback_tensor_inputs` attribute of your pipeline class.
max_sequence_length (`int`, defaults to `224`):
Maximum sequence length in encoded prompt. Can be set to other values but may lead to poorer results.

Examples:

Returns:
Expand Down