lora_scale has no effect when loading with Flux

[cshowley]

Describe the bug

According to loading loras for inference an argument cross_attention_kwargs={"scale": 0.5} can be added to a pipeline() call to vary the impact of a LORA on image generation. As the FluxPipeline class doesn't support this argument I followed the guide here to embed the text prompt with a LORA scaling parameter. However the image remained unchanged with a fixed seed+prompt and a variable lora_scale. I checked the embedding values for different values of lora_scale and saw they did not change either. Does Flux in diffusers not support LORA scaling or am I missing something?

Reproduction

from diffusers import FluxPipeline
import torch
from PIL import Image


model_path="black-forest-labs/FLUX.1-dev"
lora_path="CiroN2022/toy-face"
weight_name="toy_face_sdxl.safetensors"
device = 'cuda'
seed = torch.manual_seed(0)

pipeline = FluxPipeline.from_pretrained(
    model_path=model_path,
    torch_dtype=torch.bfloat16,
    use_safetensors=True,
).to(device)

pipeline.load_lora_weights(
    lora_path, 
    weight_name=weight_name
)

prompt = "toy_face of a hacker with a hoodie"
lora_scale = 0.5
prompt_embeds, pooled_prompt_embeds, _ = pipeline.encode_prompt(
    prompt=prompt,
    prompt_2=None,
    lora_scale=lora_scale,
)

image = pipeline(
    prompt_embeds=prompt_embeds,
    pooled_prompt_embeds=pooled_prompt_embeds,
    num_inference_steps=10,
    guidance_scale=5,
    generator=seed,
).images[0]

image.show()

Logs

No response

System Info

• 🤗 Diffusers version: 0.30.3
• Platform: Linux-6.5.0-26-generic-x86_64-with-glibc2.31
• Running on Google Colab?: No
• Python version: 3.10.15
• PyTorch version (GPU?): 2.3.1+cu121 (True)
• Flax version (CPU?/GPU?/TPU?): not installed (NA)
• Jax version: not installed
• JaxLib version: not installed
• Huggingface_hub version: 0.23.4
• Transformers version: 4.44.0
• Accelerate version: 0.33.0
• PEFT version: 0.12.0
• Bitsandbytes version: not installed
• Safetensors version: 0.4.3
• xFormers version: not installed
• Accelerator: NVIDIA A100 80GB PCIe, 81920 MiB
• Using GPU in script?: yes
• Using distributed or parallel set-up in script?: no

Who can help?

No response

[asomoza]

Hi, I never use that method, can you test with this?

pipeline.load_lora_weights(lora_path, weight_name=weight_name, adapter_name="toy")
pipe.set_adapters("toy", 0.5)

And yeah, the Flux pipeline doesn't have cross_attention_kwargs and you're using it directly when encoding the prompt, if the lora didn't train the text encoders (most don't), you won't see any difference.

[cshowley]

Your suggestion pipe.set_adapters("toy", 0.5) is not showing any change unfortunately.

In this guide I see the following code block:

pipe = ... # create pipeline
pipe.load_lora_weights(..., adapter_name="my_adapter")
scales = {
    "text_encoder": 0.5,
    "text_encoder_2": 0.5,  # only usable if pipe has a 2nd text encoder
    "unet": {
        "down": 0.9,  # all transformers in the down-part will use scale 0.9
        # "mid"  # in this example "mid" is not given, therefore all transformers in the mid part will use the default scale 1.0
        "up": {
            "block_0": 0.6,  # all 3 transformers in the 0th block in the up-part will use scale 0.6
            "block_1": [0.4, 0.8, 1.0],  # the 3 transformers in the 1st block in the up-part will use scales 0.4, 0.8 and 1.0 respectively
        }
    }
}
pipe.set_adapters("my_adapter", scales)

which says to pass a dictionary in the the .set_adapters() call. If I pass 0.5 like you said does that apply that weighting to all elements of the LORA?

[a-r-r-o-w]

Hey. I see that you're trying to load CiroN2022/toy-face as your lora. This is however a SDXL lora and is not compatible with Flux-dev. So this is basically ignored and no matter what you set as scale, it is going to ignore it.

On another note (for the diffusers devs), I would assume that there would be a warning when trying to load a lora with incompatible keys (maybe a list of all keys that were incompatible after our A1111/Kohya converters tried and couldn't find a match). Is this not the case? @yiyixuxu

[asomoza]

oh, I didn't even check the lora, IMO we should throw a warning if someone tries to load a different arch lora, at least the times I by mistake loaded SD 1.5 loras with SDXL I got the incorrect keys error.

@cshowley

which says to pass a dictionary in the the .set_adapters() call. If I pass 0.5 like you said does that apply that weighting to all elements of the LORA?

yes, if you pass a scalar instead of a dictionary, it applies the scale to all the layers.

[cshowley]

That's a good point, thanks for catching that. I swapped in a new one from here:

        """load lora"""
            pipeline.load_lora_weights(
                "XLabs-AI/flux-lora-collection", weight_name="anime_lora.safetensors", adapter_name="anime_lora"
            )

and tried applying the scale factor again as suggested, but still no change in the output:

                pipeline.set_adapters('anime_lora', 0.5)
                image = pipeline(
                    prompt="corgi living in a house of sushi, anime",
                    num_inference_steps=10,
                    guidance_scale=5,
                    generator=torch.manual_seed(0),
                ).images[0]

[bghira]

i think it's joint_attention_kwargs for flux/sd3

[asomoza]

I don't really use that much Flux so I had to do some tests.

Actually the set_adapters doesn't work with Flux, I wonder if there's people using loras with flux and diffusers, without it, you pretty much can't use multiple loras.

So right now it works like this:

import torch

from diffusers import FluxPipeline


pipe = FluxPipeline.from_pretrained("black-forest-labs/FLUX.1-dev", torch_dtype=torch.bfloat16).to("cuda")
pipe.load_lora_weights(
    "TheLastBen/The_Hound", weight_name="sandor_clegane_single_layer.safetensors"
)

joint_attention_kwargs = {"scale": 1.0}

prompt = "sandor clegane drinking in a pub"

image = pipe(
    prompt=prompt,
    num_inference_steps=30,
    width=1024,
    generator=torch.Generator("cpu").manual_seed(42),
    height=1024,
    joint_attention_kwargs=joint_attention_kwargs,
).images[0]

Doing this:

pipe.load_lora_weights(
    "TheLastBen/The_Hound", weight_name="sandor_clegane_single_layer.safetensors", adapter_nane="test_lora"
)
pipe.set_adapters("test_lora", 1.0)

Disables the lora for some reason but I can't look into it right now.

@sayakpaul for awareness

[bghira]

yes, but as you've noted, i can not set the scale: https://github.com/bghira/discord-tron-client/blob/master/discord_tron_client/classes/image_manipulation/pipeline_runners/flux.py#L43

[sayakpaul]

Interesting. Will take a look.

We do check for the validity of set_adapters() here, though:

diffusers/tests/lora/utils.py

Line 896 in 1c6ede9

def test_simple_inference_with_text_denoiser_multi_adapter(self):

I will look into it to try to make it more robust.

[cshowley]

I verified that including joint_attention_kwargs = {"scale": 1.0} with different values for scale in the pipeline call modulates the effect of the lora as expected, with scale=0 resulting in an image that flux would generate with no lora at all.

If anyone has any insight into how to upload multiple loras and independently modify their weights that would be useful, but otherwise my primary issue is fixed.

[sayakpaul]

Appreciate the investigation, @cshowley! I am looking into the other point now. Thanks for flagging!

[sayakpaul]

@asomoza this is what we had to do:

pipe.load_lora_weights(
    "TheLastBen/The_Hound", weight_name="sandor_clegane_single_layer.safetensors", adapter_nane="test_lora"
)
- pipe.set_adapters("test_lora", 1.0)
+ pipe.set_adapters("default_0", 1.0) 

test_lora is not the name of the adapter as we didn't specify anything in load_lora_weights(). default_0 is the default name.

I will work on catching this as an error and improve the testing suite.

[sayakpaul]

#9535

[asomoza]

oh, I had a typo: adapter_nane. Well, it helps to make the API more robust ^^

[sayakpaul]

#9535 should solve it. Closing this issue.