Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[data] async flat_map #50329

Closed
schmidt-ai opened this issue Feb 7, 2025 · 3 comments
Closed

[data] async flat_map #50329

schmidt-ai opened this issue Feb 7, 2025 · 3 comments
Labels
data Ray Data-related issues enhancement Request for new feature and/or capability good-first-issue Great starter issue for someone just starting to contribute to Ray triage Needs triage (eg: priority, bug/not-bug, and owning component)

Comments

@schmidt-ai
Copy link

Description

Similar to #46129, the idea is to support AsyncGenerators in Dataset.flat_map:

async def my_async_generator(...) -> AsyncGenerator[dict]:
    for x in ...:
        yield await ...

dataset = ...
dataset = dataset.flat_map(my_async_generator, ...)

(or the OOP equivalent)

Use case

Example: I have a list of zarr array files. For each array, I want to asynchronously fetch its chunks, and I can't just load the entire array due to memory constraints.
@schmidt-ai schmidt-ai added enhancement Request for new feature and/or capability triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Feb 7, 2025
@richardliaw richardliaw added good-first-issue Great starter issue for someone just starting to contribute to Ray data Ray Data-related issues labels Feb 8, 2025
@whiz-Tuhin
Copy link

@schmidt-ai I'd like to take this up.

@Drice1999
Copy link
Contributor

Hi @whiz-Tuhin , I noticed there hasn't been any progress on this issue for a while. Are you still working on it? If not, I'd be happy to take it up and work on a solution. Let me know what you think.

@Drice1999 Drice1999 mentioned this issue Mar 8, 2025
Merged
8 tasks
richardliaw pushed a commit that referenced this issue Mar 25, 2025
@Drice1999
[Data] Support async callable classes in flat_map() (#51180)
3f44633 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
This PR adds async generator support to `flat_map`.
The implementation is similar to how #46129 handled async callable
classes for map_batches(), changes include:
* Generalize the logic in `_generate_transform_fn_for_async_flat_map` so
it can process both batches and rows
* Add test case for async `flat_map`
<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number
#50329 
<!-- For example: "Closes #1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Drice1999 <chenxh267@gmail.com>
dentiny pushed a commit to dentiny/ray that referenced this issue Mar 25, 2025
@Drice1999 @dentiny
[Data] Support async callable classes in flat_map() (ray-project#51180)
076afcf 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
This PR adds async generator support to `flat_map`.
The implementation is similar to how ray-project#46129 handled async callable
classes for map_batches(), changes include:
* Generalize the logic in `_generate_transform_fn_for_async_flat_map` so
it can process both batches and rows
* Add test case for async `flat_map`
<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number
ray-project#50329 
<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Drice1999 <chenxh267@gmail.com>
@richardliaw richardliaw changed the title [<Ray component: Data>] async flat_map [data] async flat_map Mar 26, 2025
@richardliaw
Copy link
Contributor

richardliaw commented Mar 26, 2025 •
edited
Loading

Closed by #51180

dhakshin32 pushed a commit to dhakshin32/ray that referenced this issue Mar 27, 2025
@Drice1999
[Data] Support async callable classes in flat_map() (ray-project#51180)
597fcd5 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
This PR adds async generator support to `flat_map`.
The implementation is similar to how ray-project#46129 handled async callable
classes for map_batches(), changes include:
* Generalize the logic in `_generate_transform_fn_for_async_flat_map` so
it can process both batches and rows
* Add test case for async `flat_map`
<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number
ray-project#50329
<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Drice1999 <chenxh267@gmail.com>
Signed-off-by: Dhakshin Suriakannu <d_suriakannu@apple.com>
d-miketa pushed a commit to d-miketa/ray that referenced this issue Mar 28, 2025
@Drice1999 @d-miketa
[Data] Support async callable classes in flat_map() (ray-project#51180)
2dbb716 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
This PR adds async generator support to `flat_map`.
The implementation is similar to how ray-project#46129 handled async callable
classes for map_batches(), changes include:
* Generalize the logic in `_generate_transform_fn_for_async_flat_map` so
it can process both batches and rows
* Add test case for async `flat_map`
<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number
ray-project#50329 
<!-- For example: "Closes ray-project#1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Drice1999 <chenxh267@gmail.com>
srinathk10 pushed a commit that referenced this issue Mar 28, 2025
@Drice1999 @srinathk10
[Data] Support async callable classes in flat_map() (#51180)
61128c0 
<!-- Thank you for your contribution! Please review
https://github.com/ray-project/ray/blob/master/CONTRIBUTING.rst before
opening a pull request. -->

<!-- Please add a reviewer to the assignee section when you create a PR.
If you don't have the access to it, we will shortly find a reviewer and
assign them to your PR. -->

## Why are these changes needed?
This PR adds async generator support to `flat_map`.
The implementation is similar to how #46129 handled async callable
classes for map_batches(), changes include:
* Generalize the logic in `_generate_transform_fn_for_async_flat_map` so
it can process both batches and rows
* Add test case for async `flat_map`
<!-- Please give a short summary of the change and the problem this
solves. -->

## Related issue number
#50329
<!-- For example: "Closes #1234" -->

## Checks

- [x] I've signed off every commit(by using the -s flag, i.e., `git
commit -s`) in this PR.
- [x] I've run `scripts/format.sh` to lint the changes in this PR.
- [ ] I've included any doc changes needed for
https://docs.ray.io/en/master/.
- [ ] I've added any new APIs to the API Reference. For example, if I
added a
method in Tune, I've added it in `doc/source/tune/api/` under the
           corresponding `.rst` file.
- [ ] I've made sure the tests are passing. Note that there might be a
few flaky tests, see the recent failures at https://flakey-tests.ray.io/
- Testing Strategy
   - [x] Unit tests
   - [ ] Release tests
   - [ ] This PR is not tested :(

---------

Signed-off-by: Drice1999 <chenxh267@gmail.com>
Signed-off-by: Srinath Krishnamachari <srinath.krishnamachari@anyscale.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
data Ray Data-related issues enhancement Request for new feature and/or capability good-first-issue Great starter issue for someone just starting to contribute to Ray triage Needs triage (eg: priority, bug/not-bug, and owning component)
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@richardliaw @whiz-Tuhin @Drice1999 @schmidt-ai