Skip to content

Handle missing FSMap kwargs in FSStore #814

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 3 commits into from
Closed

Handle missing FSMap kwargs in FSStore #814

wants to merge 3 commits into from

Conversation

tasansal
Copy link
Contributor

This PR closes #775

fsspec.FSMap created and used by zarr.storage.FSStore does not receive the check and create kwargs.
Fixed it by exposing check and create kwargs to zarr.storage.FSStore.
Defaults are False, just like FSMap.
Is this OK, or should we default them to None so it takes the default directly from fsspec.FSMap ?

TODO:

  • Add unit tests and/or doctests in docstrings
  • Add docstrings and API docs for any new/modified user-facing classes and functions
  • New/modified features documented in docs/tutorial.rst
  • Changes documented in docs/release.rst
  • GitHub Actions have all passed
  • Test coverage is 100% (Codecov passes)

@martindurant
Copy link
Member

LGTM.

@codecov
Copy link

codecov bot commented Aug 23, 2021

Codecov Report

Merging #814 (b824b2c) into master (4b03dfe) will not change coverage.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##           master     #814   +/-   ##
=======================================
  Coverage   99.94%   99.94%           
=======================================
  Files          31       31           
  Lines       10606    10606           
=======================================
  Hits        10600    10600           
  Misses          6        6           
Impacted Files Coverage Δ
zarr/storage.py 100.00% <100.00%> (ø)

Copy link
Contributor

@rabernat rabernat left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just a question about why this is needed.

Comment on lines +1081 to +1082
check=False,
create=False,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Help me understand why these can't be passed through via **storage_options. I read #775, but here it look like any extra keywords will in the constructor will be passed right to fsspec.get_mapper.

Copy link
Contributor Author

@tasansal tasansal Aug 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@rabernat

Do I need to add this change to the release notes as a bug fix, or does someone else handle that before release?

Copy link
Contributor Author

@tasansal tasansal Aug 23, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe they changed the behavior, let me try it one more time.

Looking at the get_mapper code again,

All **storage_options get passed to url_to_fs function: (link)

The check and create are actually in the FSMap call (link)

@tasansal
Copy link
Contributor Author

tasansal commented Aug 23, 2021

@rabernat @martindurant

Digging deeper, I found some issues in fsspec + s3fs combination.

Created an fsspec issue here: fsspec/filesystem_spec#733

s3fs related conversation here.
fsspec/s3fs#401 (comment)

I think we can close this PR, and handle this in fsspec if you all agree.

@joshmoore
Copy link
Member

I think we can close this PR, and handle this in fsspec if you all agree.

Thanks, @tasansal. Closing but we can re-open if need be.

@joshmoore joshmoore closed this Aug 25, 2021
@tasansal tasansal deleted the fsspec_kwargs_bugfix branch August 25, 2021 21:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Bug: Using GCS or S3, FSStore doesn't pass check and create kwargs to FSSPEC's get_mapper()
4 participants