Write sharded repodata #161
Conversation
conda-bot
added
the
cla-signed
…hards for patching
| @@ -91,6 +92,23 @@ | |||
| repodata_version=2 which is supported in conda 24.5.0 or later. | |||
| """, | |||
| ) | |||
| @click.option( | |||
There was a problem hiding this comment.
We could replace this with an "add-only" or "no-remove" option, that would keep packages in the index even if they are not found in the filesystem.
There was a problem hiding this comment.
These two options are more about testing from a backup of the conda-forge database, they may not survive into the main branch or we could make them easier to use.
|
Now that the CEP is approved, this branch should be completed and become the way to run conda-index. |
Combining the sharded CLI into the main CLI
| show_default=True, | ||
| ) | ||
| @click.option( | ||
| "--upstream-stage", |
There was a problem hiding this comment.
The stat table has multiple rows per package and stage. If a package only exists in stage=fs or clone then its metadata is cached and a stage=index row is added. This option doesn't explain the mechanism.
| def _make_rss(channel_name, channeldata): | ||
| return rss.get_rss(channel_name, channeldata) | ||
|
|
||
|
|
There was a problem hiding this comment.
Anticipating a "no conda dependency" or "optional current_repodata" feature, we move these to another module.
| @@ -481,6 +339,9 @@ class ChannelIndex: | |||
| :param channel_url: fsspec URL where package files live. If provided, channel_root will only be used for cache and index output. | |||
| :param fs: ``MinimalFS`` instance to be used with channel_url. Wrap fsspec AbstractFileSystem with ``conda_index.index.fs.FsspecFS(fs)``. | |||
| :param base_url: Add ``base_url/<subdir>`` to repodata.json to be able to host packages separate from repodata.json | |||
| :param save_fs_state: Pass False to use cached filesystem state instead of ``os.listdir(subdir)`` | |||
| :param write_monolithic: Pass True to write large 'repodata.json' with all packages. | |||
There was a problem hiding this comment.
Is repodata.json appropriately called the "monolithic" option?
There was a problem hiding this comment.
CEP 16 doesn't say anything about this, I think it might have been beneficial to version the repodata format as well, to be clear about it, but alas the CEP has passed.
"monolithic" seems like the closest to the content of the written file, alternatively you could use the fact that it's a JSON file. So maybe "write_monolithic_json"?
| @@ -557,7 +439,7 @@ def index( | |||
| # begin non-stop "extract packages into cache"; | |||
| # extract_subdir_to_cache manages subprocesses. Keeps cores busy | |||
| # during write/patch/update channeldata steps. | |||
| def extract_subdirs_to_cache(): | |||
| def extract_subdirs_to_cache(): # is the 'prepare' step in 'index_prepared_subdir' | |||
There was a problem hiding this comment.
index_prepared_subdir was renamed, we could say that we have to load metadata into the cache before we can generate-repodata-out-of-it.
| # exactly these packages (unless they are un-indexable) will | ||
| # be in the output repodata | ||
| if self.save_fs_state: | ||
| cache.save_fs_state(subdir_path) |
There was a problem hiding this comment.
I think we use this flag OR overriding the save_fs_state() function to be a noop
There was a problem hiding this comment.
... in fact we use this flag AND we override the "get changed packages" function to return an empty list. This preserves the list of packages that belong in the index, and prevents us from trying to update the metadata cache for any of them.
| import zstandard | ||
| from conda.base.context import context | ||
|
|
||
| # BAD BAD BAD - conda internals |
|
|
||
| if self.base_url: | ||
| # per https://github.com/conda-incubator/ceps/blob/main/cep-15.md | ||
| shards_index["info"]["base_url"] = f"{self.base_url.rstrip('/')}/{subdir}/" |
| db.execute( | ||
| "ALTER TABLE index_json ADD COLUMN sha256 AS (json_extract(index_json, '$.sha256'))" | ||
| ) | ||
|
|
There was a problem hiding this comment.
Wondering when it's time to port to sqlalchemy and add alembic to conda-index 😬
There was a problem hiding this comment.
That port exists, we have a sqlalchemy version of the schema that originated in conda-index, first used for a queryable metadata database.
conda-index tries to be deliberately low-dependency. But if we add postgres support we will certainly use sqlalchemy.
|
I'm looking into a sharded repodata comparison test with what prefix-dev's splitter produces... |
|
Excellent news, thank you @dholth |
Description
What would it look like to generate sharded repodata per conda/ceps#75
Interested in seeing whether we can efficiently generate shards; how repodata patching should work; and whether we could generate shards as the primary artifact and then derive repodata.json from shards [at the same time in "processes a sequence of package names" code]
How to test
Check out this repository and https://github.com/dholth/conda-test-data
Decompress
conda_test_data/conda-forge/*/.cache/cache.db.zstDownload
conda-forge-repodata-patches-<version>.condapython3 -m conda_index --write-shards --no-write-monolithic --upstream-stage=clone --no-update-cache --patch-generator ~/miniconda3/pkgs/conda-forge-repodata-patches-20240401.20.33.07-hd8ed1ab_1.conda --output /tmp/shards ~/prog/conda-test-data/conda-forgeExamine output in
/tmp/shardsI've begun trying to apply the patches to individual shards. This is slow; should compare against applying the many patches against a whole repodata.json.
Checklist - did you ...
newsdirectory (using the template) for the next release's release notes?