Wan Pipeline scaling fix, type hint warning, multi generator fix

[hlky]

What does this PR do?

Output changed slightly with #10998 due to where scaling was applied, it has to be applied before retrieve_latents. We may want to consider refactoring to allow returning from encode before posterior/DiagonalGaussianDistribution.

original

output.16.mp4

10998

output.17.mp4

PR

output.19.mp4

Type hint for Image encoder updated to CLIPVisionModelWithProjection as there's a warning.

In multi-generator path latents is overwritten by latent_condition: latents = latent_condition = torch.cat(latent_condition).

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

[a-r-r-o-w]

LGTM but:

• Usually we do the latent mean/std related scaling in the pipeline. Will let YiYi comment on that
• It seems the authors made the change to CLIPVisionModel instead of CLIPVisionModelWithProjection: https://github.com/huggingface/diffusers/pull/10975/files. Is that change incorrect? I'm noticing differences in output after that PR but I don't know which is right (sorry it's too late for me and I'm almost asleep to check myself today)

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[hlky]

It's still in the pipeline, no? It was previously inside vae encode, moved out to be in the pipeline like usual but this model does an unusual thing of applying scaling before posterior/DiagonalGaussianDistribution.

The output changes from #10975 are due to image_embeds.hidden_states[-2] instead of image_embeds.hidden_states[-1]. The config on the Hub uses CLIPVisionModelWithProjection so the recent comprehensive type checking logs a warning. The checkpoint contains visual_projection which makes it CLIPVisionModelWithProjection, although there's no difference between the two as far as this model is concerned, it's not using image_embeds (from CLIPVisionModelWithProjection).

[yiyixuxu]

why this change? the authors just changed to CLIPVisionModel https://github.com/huggingface/diffusers/pull/10975/files

[yiyixuxu]

@a-r-r-o-w the change of output is probably from the encode_image https://github.com/huggingface/diffusers/pull/10975/files#r1985808256

[yiyixuxu]

The output changes from #10975 are due to image_embeds.hidden_states[-2] instead of image_embeds.hidden_states[-1]. The config on the Hub uses CLIPVisionModelWithProjection so the recent comprehensive type checking logs a warning. The checkpoint contains visual_projection which makes it CLIPVisionModelWithProjection, although there's no difference between the two as far as this model is concerned

so since the change was made by authors, I think we need to check with them. maybe leave that out from this PR if we want to get merged quickly?

[yiyixuxu]

t's still in the pipeline, no? It was previously inside vae encode, moved out to be in the pipeline like usual but this model does an unusual thing of applying scaling before posterior/DiagonalGaussianDistribution.

indeed it would be ideal if we can keep the retrive_latent consistent but ok if it's the case

[hlky]

Can move this out of retrieve_latents if you'd prefer, it would affect this section more

diffusers/src/diffusers/pipelines/wan/pipeline_wan_i2v.py

Lines 413 to 420 in e461b61

	if isinstance(generator, list):
	latent_condition = [
	retrieve_latents(self.vae.encode(video_condition), latents_mean, latents_std, g) for g in generator
	]
	latent_condition = torch.cat(latent_condition)
	else:
	latent_condition = retrieve_latents(self.vae.encode(video_condition), latents_mean, latents_std, generator)
	latent_condition = latent_condition.repeat(batch_size, 1, 1, 1, 1)

[hlky]

so since the change was made by authors, I think we need to check with them. maybe leave that out from this PR if we want to get merged quickly?

removed in e461b61
IMO it should be CLIPVisionModelWithProjection to match the checkpoint and avoid the type checking warning, but I will let authors comment.

[hlky]

Merging now to restore outputs, will follow-up with further refactoring.