[docs] Memory optims #11385
[docs] Memory optims #11385
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
AutoencoderKLWan and AsymmetricAutoencoderKL does not support tiling or slicing (asymmetric have just unused flags), it most likely should be mentioned. |
|
Thanks for the feedback on the memory doc! I also updated the inference speed doc, so please feel free to check it out and leave some feedback ❤️ |
docs/source/en/optimization/fp16.md
Outdated
| prompt = "slice of delicious New York-style cheesecake topped with berries, mint, chocolate crumble" | ||
| pipeline(prompt, num_inference_steps=50).images[0] |
There was a problem hiding this comment.
I think we should also have an advanced guide on accelerating inference that make use of other techniques like SAGE, multi-GPU inference (context-parallel), etc. @a-r-r-o-w WDYT?
There was a problem hiding this comment.
Happy to add in a separate PR!
There was a problem hiding this comment.
SGTM. Not sure how we plan to show context parallel + Sage because the implementation is based on PR in finetrainers. Do we want to showcase that in the docs?
Or do you want me to take just the relevant parts and create a minimal attention processor to showcase the example? I can send over a rough writeup to Steven for whichever we choose to do
Note that the latter will only yield a small amount of speedup compared to using the current finetrainers implementation (because it shards the sequence dim early on instead of just for the attention, which is not possible to showcase in only-diffusers example because it either involves rewriting the forward method OR adds the overhead of explaining model hooks to reader and making that part of the example)
There was a problem hiding this comment.
Hmm got it. Sorry for being unclear.
If it's not too much would it be possible to send over a write-up covering both cases? And we can then further brainstorm how to best disseminate the information to the readers? But really no rush here. Do it whenever you get time.
docs/source/en/optimization/fp16.md
Outdated
| </div> | ||
| ## Distilled models | ||
|
|
||
| Another option for accelerating inference is to use a smaller distilled model if it's available. During distillation, many of the UNet's residual and attention blocks are discarded to reduce model size and improve latency. A distilled model is faster and uses less memory without compromising quality compared to a full-sized model. |
There was a problem hiding this comment.
@stevhliu I thought we were removing these sections?
There was a problem hiding this comment.
My bad, thought I already had! Should be gone now :)
|
|
||
| The `offload_type` parameter can be set to `block_level` or `leaf_level`. | ||
|
|
||
| - `block_level` offloads groups of layers based on the `num_blocks_per_group` parameter. For example, if `num_blocks_per_group=2` on a model with 40 layers, 2 layers are onloaded and offloaded at a time (2o total onloads/offloads). This drastically reduces memory requirements. |
There was a problem hiding this comment.
| - `block_level` offloads groups of layers based on the `num_blocks_per_group` parameter. For example, if `num_blocks_per_group=2` on a model with 40 layers, 2 layers are onloaded and offloaded at a time (2o total onloads/offloads). This drastically reduces memory requirements. | |
| - `block_level` offloads groups of layers based on the `num_blocks_per_group` parameter. For example, if `num_blocks_per_group=2` on a model with 40 layers, 2 layers are onloaded and offloaded at a time (20 total onloads/offloads). This drastically reduces memory requirements. |
Refactors the memory optimization docs and combines it with working with big models (distributed setups).
Let me know if I'm missing anything!