Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[core] add sleep and wake up endpoint and v1 support #12987

Merged
merged 24 commits into from
Feb 20, 2025
Merged

[core] add sleep and wake up endpoint and v1 support #12987

merged 24 commits into from
Feb 20, 2025

Conversation

youkaichao
Copy link
Member

@youkaichao youkaichao commented Feb 9, 2025 •
edited
Loading

expose /sleep and /wake_up endpoint, and also support it in v1

@youkaichao
add sleep and wake up endpoint
3a7ecb4 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add tests
4fbb39e 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add ack
e74dd9a 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@github-actions GitHub Actions
Copy link

github-actions bot commented Feb 9, 2025

👋 Hi! Thank you for contributing to the vLLM project.

💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels.

Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run fastcheck CI which starts running only a small and essential subset of CI tests to quickly catch errors. You can run other CI tests on top of those by going to your fastcheck build on Buildkite UI (linked in the PR checks section) and unblock them. If you do not have permission to unblock, ping simon-mo or khluu to add you in our Buildkite org.

Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging.

To run CI, PR reviewers can either: Add ready label to the PR or enable auto-merge.

🚀

@youkaichao
add status code check
79ec652 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix socket
13f4bdf 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
use send_request_and_get_response
fa86994 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix
ef145d9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
fix output
042c959 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@njhill
Copy link
Member

njhill commented Feb 9, 2025

@youkaichao I wonder if we could merge #12918 first which simplifies the v1 request handling a bit? (it should be ready now)

@mergify Mergify
Copy link

mergify bot commented Feb 10, 2025

This pull request has merge conflicts that must be resolved before it can be
merged. Please rebase the PR, @youkaichao.

https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork

@mergify mergify bot added the needs-rebase label Feb 10, 2025
@njhill njhill mentioned this pull request Feb 11, 2025
Closed
@cennn
Merge branch 'main' of https://github.com/vllm-project/vllm into slee…
dbf1756 
…p-youkaichao

Signed-off-by: cennn <2523403608@qq.com>
@mergify mergify bot removed the needs-rebase label Feb 12, 2025
cennn and others added 6 commits February 12, 2025 14:54
@cennn
support VLLM_USE_V1 && fix typo WAKE_UP
4010671 
Signed-off-by: cennn <2523403608@qq.com>
@youkaichao
Merge branch 'main' into sleep_apiserver
1f808d5
@youkaichao
add comments
283507d 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
minimize diff
e6d618d 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
minimize diff
7511795 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
aa23022 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao changed the title [core] add sleep and wake up endpoint [core] add sleep and wake up endpoint and v1 support Feb 13, 2025
@youkaichao
add timing
f7bcffa 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
aaba2f4 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
4b8bd93 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
add v1 test
b48f686 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao mentioned this pull request Feb 15, 2025
Merged
cennn and others added 4 commits February 19, 2025 23:41
@cennn
Merge branch 'main' of https://github.com/vllm-project/vllm into slee…
e23031f 
…p-youkaichao

Signed-off-by: cennn <2523403608@qq.com>
@cennn
rm time.sleep(30)
83813d8 
Signed-off-by: cennn <2523403608@qq.com>
@youkaichao
improve tests
c795115 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao
improve tests
1657de0 
Signed-off-by: youkaichao <youkaichao@gmail.com>
youkaichao
youkaichao commented Feb 20, 2025
View reviewed changes
vllm/entrypoints/openai/serving_transcription.py
@@ -295,6 +295,7 @@ async def create_transcription(
# TODO(rob): figure out a way to pipe streaming in.
# Non-streaming response.
try:
assert result_generator is not None
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this is to pass the linter.

@youkaichao youkaichao marked this pull request as ready for review February 20, 2025 02:21
@youkaichao youkaichao requested a review from njhill February 20, 2025 02:27
njhill
njhill approved these changes Feb 20, 2025
View reviewed changes
Copy link
Member

@njhill njhill left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @youkaichao LGTM

vllm/entrypoints/openai/api_server.py Outdated
await engine_client(raw_request).sleep(int(level))
# when we return a response, the sleep command
# is sent but does not finish yet.
# TODO: we should wait for the sleep to finish and then return
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure if it's worth updating this comment since it does now wait in V1?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated in e8416d9 now.

@youkaichao
update comments
e8416d9 
Signed-off-by: youkaichao <youkaichao@gmail.com>
@youkaichao youkaichao merged commit ba81163 into vllm-project:main Feb 20, 2025
7 of 9 checks passed
@youkaichao youkaichao deleted the sleep_apiserver branch February 20, 2025 04:41
xjpang pushed a commit to xjpang/vllm that referenced this pull request Feb 20, 2025
@youkaichao @cennn
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
46b8c77 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
kerthcet pushed a commit to kerthcet/vllm that referenced this pull request Feb 21, 2025
@youkaichao @cennn @kerthcet
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
ae2e97a 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
michaelrglass pushed a commit to michaelrglass/vllm that referenced this pull request Feb 21, 2025
@youkaichao @cennn @michaelrglass
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
ea7560a 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Signed-off-by: Michael Glass <mrglass@us.ibm.com>
Akshat-Tripathi pushed a commit to krai/vllm that referenced this pull request Mar 3, 2025
@youkaichao @cennn @Akshat-Tripathi
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
5cde704 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
@sivanantha321 sivanantha321 mentioned this pull request Mar 4, 2025
Merged
9 tasks
lk-chen pushed a commit to lk-chen/vllm that referenced this pull request Mar 5, 2025
@youkaichao @cennn @lk-chen
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
0584715 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Signed-off-by: Linkun Chen <github@lkchen.net>
Said-Akbar pushed a commit to Said-Akbar/vllm-rocm that referenced this pull request Mar 7, 2025
@youkaichao @cennn @Said-Akbar
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
5ed06bb 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Signed-off-by: saeediy <saidakbarp@gmail.com>
qscqesze pushed a commit to ZZBoom/vllm that referenced this pull request Mar 13, 2025
@youkaichao @cennn @qscqesze
[core] add sleep and wake up endpoint and v1 support (vllm-project#12987
e359543 
)

Signed-off-by: youkaichao <youkaichao@gmail.com>
Signed-off-by: cennn <2523403608@qq.com>
Co-authored-by: cennn <2523403608@qq.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@comaniac comaniac comaniac approved these changes

@njhill njhill njhill approved these changes

@DarkLight1337 DarkLight1337 Awaiting requested review from DarkLight1337 DarkLight1337 is a code owner

@robertgshaw2-redhat robertgshaw2-redhat Awaiting requested review from robertgshaw2-redhat robertgshaw2-redhat is a code owner

@simon-mo simon-mo Awaiting requested review from simon-mo simon-mo is a code owner

@WoosukKwon WoosukKwon Awaiting requested review from WoosukKwon WoosukKwon is a code owner

@ywang96 ywang96 Awaiting requested review from ywang96 ywang96 is a code owner

@alexm-redhat alexm-redhat Awaiting requested review from alexm-redhat alexm-redhat is a code owner

Assignees
No one assigned
Labels
frontend v1
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@youkaichao @njhill @comaniac @cennn