Bug: Using GCS or S3, FSStore doesn't pass check and create kwargs to FSSPEC's get_mapper()

[tasansal]

I think this is a bug, if everyone agrees I can fix it and do a PR.

Minimal, reproducible code sample, a copy-pastable example if possible

import zarr

# Works fine with this as expected
url = '~/my_root.zarr'

# Doesn't work with:
url = r'gcs://my_bucket/my_root.zarr'
# or
url = r's3://my_bucket/my_root.zarr'

storage_options = {'key': my_key, 'secret': my_secret, 'token': my_token}  # Example for s3
store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
zarr.open(store, 'w')

Problem description

The above code fails because the FSMap created and used by FSStore did not receive the check and create kwargs (Traceback at the bottom of the post).

The check parameter is used to verify write access to remote stores.
The create function creates the root object/directory if it doesn't exist, so it can check for write access

The fsspec.mapping.get_mapper used in zarr.storage.FSStore expects explicit kwargs for check and create to send FSMap and passes storage_options aka. other kwargs to ONLY to url_to_fs (here). The FSMap call here expects the check and create kwargs, which explicitly have to be passed to get_mapper.

The issue can be circumvented by dropping the write access checking, but that may not be always the case user wants.

The ideal fix is to make zarr.storage.FSStore to expect check and create kwargs; and default to False, just like FSMap.

Version and installation information

Please provide the following:

• zarr.__version__ 2.8.1
• numcodecs.__version__ 0.7.3
• Python 3.9.5
• Linux x86_64
• installed via conda

Traceback:

Traceback (most recent call last):
  File "/home/my_user/develop/my_library/benchmarks/test.py", line 5, in <module>
    store = zarr.storage.FSStore(url, check=True, create=True, **storage_options)
  File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/zarr/storage.py", line 1052, in __init__
    self.map = fsspec.get_mapper(url, **storage_options)
  File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 224, in get_mapper
    return FSMap(root, fs, check, create, missing_exceptions=missing_exceptions)
  File "/home/my_user/miniconda3/envs/my_library/lib/python3.9/site-packages/fsspec/mapping.py", line 52, in __init__
    raise ValueError(
ValueError: Path my_bucket/my_root.zarr does not exist. Create  with the ``create=True`` keyword

[martindurant]

Yes, please fix to pass the kwargs!

[tasansal]

@martindurant, Created PR #814 to handle this issue.

[rabernat]

I am reviewing #814 and wanted to follow up with a question here.

Isn't this a bug with fsspec? The fact that this works with one FSSpec implementation (LocalFileSystem) but not others (gcsfs, s3fs) suggests a problem with those implementations. Working around that bug in zarr-python creates technical debt here.

Is there an upstream fix possible.

[tasansal]

@rabernat

Let me verify that it still works as expected on a LocalFileSystem when write-permission is not enabled.

If it works as expected, I agree, we should probably put a fix in fsspec.

[joshmoore]

Can this issue be closed now?

[martindurant]

We should probably have tests against this behaviour to be sure, but I suspect everything now works as expected.

[tasansal]

I did a PR to fix this in #814, but it was closed to be handled in fsspec. I haven't done a PR there, has anyone else fixed this?

[jhamman]

Closing as stale now that the 3.0.0 store doesn't use the same kwargs.