[Attention] MLA move o_proj q_proj into cuda-graph region

[LucasWilkinson]

With: #14770 we are no longer materializing the absorbed matrices so we can move q_proj and o_proj into the cuda-graph region lowering CPU overhead:

Accuracy

VLLM_USE_V1=0 lm_eval --model vllm --model_args pretrained=deepseek-ai/DeepSeek-V2-Lite-Chat,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.9,trust_remote_code=True,max_model_len=16384 --task gsm8k --num_fewshot 5  --batch_size auto

vllm (pretrained=deepseek-ai/DeepSeek-V2-Lite-Chat,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.9,trust_remote_code=True,max_model_len=16384), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.6679|±  |0.0130|
|     |       |strict-match    |     5|exact_match|↑  |0.6581|±  |0.0131|

VLLM_USE_V1=1 lm_eval --model vllm --model_args pretrained=deepseek-ai/DeepSeek-V2-Lite-Chat,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.9,trust_remote_code=True,max_model_len=1638
4 --task gsm8k --num_fewshot 5  --batch_size auto

vllm (pretrained=deepseek-ai/DeepSeek-V2-Lite-Chat,tensor_parallel_size=2,dtype=auto,gpu_memory_utilization=0.9,trust_remote_code=True,max_model_len=16384), gen_kwargs: (None), limit: None, num_fewshot: 5, batch_size: auto
|Tasks|Version|     Filter     |n-shot|  Metric   |   |Value |   |Stderr|
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|gsm8k|      3|flexible-extract|     5|exact_match|↑  |0.6657|±  |0.0130|
|     |       |strict-match    |     5|exact_match|↑  |0.6573|±  |0.0131|

CPU implementation confirmed working with: #17494

Benchmark Results (DeepSeek-R1, 8xH200)

This PR

  input_tokens output_tokens  vllm_output_toks/s
0         1000          2000         1183.526109
1         5000          1000          917.148781
2        10000           500          475.100371
3        30000           100           36.287921
4     sharegpt      sharegpt         1415.561950

Main

  input_tokens output_tokens  vllm_output_toks/s
0         1000          2000         1113.702931
1         5000          1000          857.432545
2        10000           500          463.061531
3        30000           100           35.905324
4     sharegpt      sharegpt         1379.961186

[github-actions]

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

[simon-mo]

nice!