From 53318b99b16e487175fda50a3d1728c6821e1b78 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 17:28:46 -0400
Subject: [PATCH 01/29] hagne description to point to components page,
zarr-python, and the implementations page
---
index.md | 12 +++---------
1 file changed, 3 insertions(+), 9 deletions(-)
diff --git a/index.md b/index.md
index 6a98119..07e0798 100644
--- a/index.md
+++ b/index.md
@@ -32,15 +32,9 @@ can be represented as a key-value store, including most commonly POSIX file
systems and cloud object storage but also zip files as well as relational and
document databases.
-See the following GitHub repositories for more information:
-
-* [Zarr Python](https://github.com/zarr-developers/zarr)
-* [Zarr Specs](https://github.com/zarr-developers/zarr-specs)
-* [Numcodecs](https://github.com/zarr-developers/numcodecs)
-* [Z5](https://github.com/constantinpape/z5)
-* [N5](https://github.com/saalfeldlab/n5)
-* [Zarr.jl](https://github.com/meggart/Zarr.jl)
-* [ndarray.scala](https://github.com/lasersonlab/ndarray.scala)
+For more details read about the various [Components of Zarr](https://zarr.dev/componenets/),
+see the canonical [Zarr-Python](https://github.com/zarr-developers/zarr-python) implementation,
+or look through [other Zarr implementations](https://zarr.dev/implementations/) for one in your preferred language.
## Applications
From d91c743adc6bfb014bc3632f39d29beb842f9175 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 17:29:11 -0400
Subject: [PATCH 02/29] add compression as another key feature of zarr
---
index.md | 1 +
1 file changed, 1 insertion(+)
diff --git a/index.md b/index.md
index 07e0798..393d293 100644
--- a/index.md
+++ b/index.md
@@ -45,6 +45,7 @@ or look through [other Zarr implementations](https://zarr.dev/implementations/)
## Features
* Chunk multi-dimensional arrays along any dimension.
+* Compress array chunks via an extensible system of compressors.
* Store arrays in memory, on disk, inside a Zip file, on S3, etc.
* Read and write arrays concurrently from multiple threads or processes.
* Organize arrays into hierarchies via annotatable groups.
From 237b67454aa47d0436ea4ae8205cf07034f3e283 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 17:56:14 -0400
Subject: [PATCH 03/29] describe abstract components
---
components/index.md | 21 +++++++++++++++++++++
1 file changed, 21 insertions(+)
create mode 100644 components/index.md
diff --git a/components/index.md b/components/index.md
new file mode 100644
index 0000000..aac56bb
--- /dev/null
+++ b/components/index.md
@@ -0,0 +1,21 @@
+---
+layout: single
+author_profile: false
+title: Zarr Components
+sidebar:
+ title: "Content"
+ nav: sidebar
+---
+
+Zarr consists of several components, both abstract and concrete. These span both the physical storage layer and the conceptual structural layer. Zarr-related projects all follow the Zarr specification (and hence data model), but otherwise may choose to implement other layers however they wish.
+
+## Abstract components
+
+**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data and metadata as byte streams via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface), using a system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) to describe the encoding and serialization steps.
+
+**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html). It consists of a heirarchical tree of groups and arrays, with optional arbitrary metadata at every node. This model is completely domain-agnostic.
+
+**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format". Most, but not all, zarr implementations will serialize to this format.
+
+## Concrete components
+
From 072d3c5db86c75e125d1298e961455bb12023fe7 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 18:19:20 -0400
Subject: [PATCH 04/29] add section on concrete components
---
components/index.md | 11 +++++++++++
1 file changed, 11 insertions(+)
diff --git a/components/index.md b/components/index.md
index aac56bb..0ad6797 100644
--- a/components/index.md
+++ b/components/index.md
@@ -11,11 +11,22 @@ Zarr consists of several components, both abstract and concrete. These span both
## Abstract components
+These abstract components together describe what type of data can be stored in zarr, and how to store it, without assuming you are working in a particular programming language, or with a particular storage system.
+
**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data and metadata as byte streams via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface), using a system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) to describe the encoding and serialization steps.
**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html). It consists of a heirarchical tree of groups and arrays, with optional arbitrary metadata at every node. This model is completely domain-agnostic.
**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format". Most, but not all, zarr implementations will serialize to this format.
+**Extensions**:
+
## Concrete components
+Concrete implementations of the abstract components can be implemented in any language. The canonical reference implementation is [Zarr-Python](https://github.com/zarr-developers/zarr-python), but there are many [other implementations](https://zarr.dev/implementations/). Zarr-Python contains reference examples of useful constructs that can be re-implemented in other languages.
+
+**Zarr-Python Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, based on a `MutableMapping`-like API. This component is concrete in the sense that it is implemented in a specific programming language, and enforces particular syntax for getting and setting values in a key-value store.
+
+**Zarr-Python Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage. These write data in the Native Zarr Format. It's expected that most users of zarr will just use one of these implementations.
+
+**Zarr-Python User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface. This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
\ No newline at end of file
From 96cbd4dadc93a56dcdd586b85dce39adade491e4 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 18:29:55 -0400
Subject: [PATCH 05/29] add heading for section on flexibility
---
components/index.md | 10 ++++++++--
1 file changed, 8 insertions(+), 2 deletions(-)
diff --git a/components/index.md b/components/index.md
index 0ad6797..edefc6e 100644
--- a/components/index.md
+++ b/components/index.md
@@ -19,7 +19,7 @@ These abstract components together describe what type of data can be stored in z
**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format". Most, but not all, zarr implementations will serialize to this format.
-**Extensions**:
+**Extensions**:
## Concrete components
@@ -29,4 +29,10 @@ Concrete implementations of the abstract components can be implemented in any la
**Zarr-Python Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage. These write data in the Native Zarr Format. It's expected that most users of zarr will just use one of these implementations.
-**Zarr-Python User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface. This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
\ No newline at end of file
+**Zarr-Python User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface. This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
+
+## Component Flexibility
+
+One of Zarr's greatest strengths is its flexibility. Here are a few interesting zarr-related projects, with descriptions of how they do or don't make use of different zarr components.
+
+TODO
\ No newline at end of file
From 750e99ce4d2f7c94feb4614313caf76ee8cdc010 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 18:33:28 -0400
Subject: [PATCH 06/29] make each sentence a new line
---
components/index.md | 29 ++++++++++++++++++++---------
1 file changed, 20 insertions(+), 9 deletions(-)
diff --git a/components/index.md b/components/index.md
index edefc6e..337c76b 100644
--- a/components/index.md
+++ b/components/index.md
@@ -7,32 +7,43 @@ sidebar:
nav: sidebar
---
-Zarr consists of several components, both abstract and concrete. These span both the physical storage layer and the conceptual structural layer. Zarr-related projects all follow the Zarr specification (and hence data model), but otherwise may choose to implement other layers however they wish.
+Zarr consists of several components, both abstract and concrete.
+These span both the physical storage layer and the conceptual structural layer.
+Zarr-related projects all follow the Zarr specification (and hence data model), but otherwise may choose to implement other layers however they wish.
## Abstract components
These abstract components together describe what type of data can be stored in zarr, and how to store it, without assuming you are working in a particular programming language, or with a particular storage system.
-**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data and metadata as byte streams via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface), using a system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) to describe the encoding and serialization steps.
+**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data and metadata as byte streams via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface).
+A system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) is used to describe the encoding and serialization steps.
-**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html). It consists of a heirarchical tree of groups and arrays, with optional arbitrary metadata at every node. This model is completely domain-agnostic.
+**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html).
+It consists of a heirarchical tree of groups and arrays, with optional arbitrary metadata at every node. This model is completely domain-agnostic.
-**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format". Most, but not all, zarr implementations will serialize to this format.
+**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format".
+Most, but not all, zarr implementations will serialize to this format.
**Extensions**:
## Concrete components
-Concrete implementations of the abstract components can be implemented in any language. The canonical reference implementation is [Zarr-Python](https://github.com/zarr-developers/zarr-python), but there are many [other implementations](https://zarr.dev/implementations/). Zarr-Python contains reference examples of useful constructs that can be re-implemented in other languages.
+Concrete implementations of the abstract components can be implemented in any language.
+The canonical reference implementation is [Zarr-Python](https://github.com/zarr-developers/zarr-python), but there are many [other implementations](https://zarr.dev/implementations/).
+Zarr-Python contains reference examples of useful constructs that can be re-implemented in other languages.
-**Zarr-Python Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, based on a `MutableMapping`-like API. This component is concrete in the sense that it is implemented in a specific programming language, and enforces particular syntax for getting and setting values in a key-value store.
+**Zarr-Python Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, based on a `MutableMapping`-like API.
+This component is concrete in the sense that it is implemented in a specific programming language, and enforces particular syntax for getting and setting values in a key-value store.
-**Zarr-Python Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage. These write data in the Native Zarr Format. It's expected that most users of zarr will just use one of these implementations.
+**Zarr-Python Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage. These write data in the Native Zarr Format.
+It's expected that most users of zarr will just use one of these implementations.
-**Zarr-Python User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface. This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
+**Zarr-Python User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface.
+This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
## Component Flexibility
-One of Zarr's greatest strengths is its flexibility. Here are a few interesting zarr-related projects, with descriptions of how they do or don't make use of different zarr components.
+One of Zarr's greatest strengths is its flexibility.
+Here are a few interesting zarr-related projects, with descriptions of how they do or don't make use of different zarr components.
TODO
\ No newline at end of file
From 4a2492739b5a61d73c3030f16be2421e70c22c99 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 18:49:22 -0400
Subject: [PATCH 07/29] add section on extensions
---
components/index.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index 337c76b..58f66fa 100644
--- a/components/index.md
+++ b/components/index.md
@@ -24,7 +24,7 @@ It consists of a heirarchical tree of groups and arrays, with optional arbitrary
**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format".
Most, but not all, zarr implementations will serialize to this format.
-**Extensions**:
+**Extensions**: Zarr provides a core set of generally-useful features, but extensions to this core are encouraged. These might take the form of domain-specific [metadata conventions](https://zarr.dev/conventions/), new codecs, or additions to the data model via [extension points](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#extension-points). These can be enforced by implementations or client libraries however they like, but generally should be opt-in.
## Concrete components
From dfd10e4b2df71a3b5662900c106f9ffd75ae5918 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 19:16:53 -0400
Subject: [PATCH 08/29] add section on TensorStore
---
components/index.md | 24 +++++++++++++++++-------
1 file changed, 17 insertions(+), 7 deletions(-)
diff --git a/components/index.md b/components/index.md
index 58f66fa..9cf2be0 100644
--- a/components/index.md
+++ b/components/index.md
@@ -32,18 +32,28 @@ Concrete implementations of the abstract components can be implemented in any la
The canonical reference implementation is [Zarr-Python](https://github.com/zarr-developers/zarr-python), but there are many [other implementations](https://zarr.dev/implementations/).
Zarr-Python contains reference examples of useful constructs that can be re-implemented in other languages.
-**Zarr-Python Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, based on a `MutableMapping`-like API.
+**Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, based on a `MutableMapping`-like API.
This component is concrete in the sense that it is implemented in a specific programming language, and enforces particular syntax for getting and setting values in a key-value store.
-**Zarr-Python Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage. These write data in the Native Zarr Format.
-It's expected that most users of zarr will just use one of these implementations.
+**Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage.
+These write data in the Native Zarr Format.
+It's expected that most users of zarr from python will just use one of these implementations.
-**Zarr-Python User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface.
+**User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface.
This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
## Component Flexibility
-One of Zarr's greatest strengths is its flexibility.
-Here are a few interesting zarr-related projects, with descriptions of how they do or don't make use of different zarr components.
+One of Zarr's greatest strengths is its flexibility, or "hackability". In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
+Here are a few interesting zarr-related projects, which selectively make use of a subset of different zarr components.
-TODO
\ No newline at end of file
+- **VirtualiZarr**
+
+- **NCZarr** is in some sense the opposite of VirtualiZarr - it
+
+- **Tensorstore** is a general storage library written in C++ that can write to Zarr (so is spec-compliant non-python store implementation) but also to other array formats such as N5.
+It can write to multiple different storage sytems, so effectively has its own set of concrete store implementations.
+Additional features are provided, notably using an Optionally-Cooperative Distributed B+Tree (OCDBT) on top of a base key-value store to implement ACID transactions.
+It still stores all data using the native Zarr Format, but versions keys at the store level.
+
+- **Icechunk**
From d6d0c135edef42d43db21f882c7ee8e967c943fc Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 19:45:54 -0400
Subject: [PATCH 09/29] add extensions, icechunk, and mongodb
---
components/index.md | 17 +++++++++++------
1 file changed, 11 insertions(+), 6 deletions(-)
diff --git a/components/index.md b/components/index.md
index 9cf2be0..c16d843 100644
--- a/components/index.md
+++ b/components/index.md
@@ -24,7 +24,7 @@ It consists of a heirarchical tree of groups and arrays, with optional arbitrary
**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format".
Most, but not all, zarr implementations will serialize to this format.
-**Extensions**: Zarr provides a core set of generally-useful features, but extensions to this core are encouraged. These might take the form of domain-specific [metadata conventions](https://zarr.dev/conventions/), new codecs, or additions to the data model via [extension points](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#extension-points). These can be enforced by implementations or client libraries however they like, but generally should be opt-in.
+**Extensions**: Zarr provides a core set of generally-useful features, but extensions to this core are encouraged. These might take the form of domain-specific [metadata conventions](https://zarr.dev/conventions/), new codecs, or additions to the data model via [extension points](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#extension-points). These can be abstract, or enforced by implementations or client libraries however they like, but generally should be opt-in.
## Concrete components
@@ -44,16 +44,21 @@ This allows user applications to use a standard zarr API to read and write from
## Component Flexibility
-One of Zarr's greatest strengths is its flexibility, or "hackability". In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
-Here are a few interesting zarr-related projects, which selectively make use of a subset of different zarr components.
+One of Zarr's greatest strengths is its flexibility, or "hackability".
+In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
+Here are a few interesting zarr-related projects, which selectively make use of a subset of different zarr components, both abstract and concrete.
+
+- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys. It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
- **VirtualiZarr**
- **NCZarr** is in some sense the opposite of VirtualiZarr - it
-- **Tensorstore** is a general storage library written in C++ that can write to Zarr (so is spec-compliant non-python store implementation) but also to other array formats such as N5.
-It can write to multiple different storage sytems, so effectively has its own set of concrete store implementations.
+- **Tensorstore** is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
+As it can write to multiple different storage sytems, it effectively has its own set of concrete store implementations.
Additional features are provided, notably using an Optionally-Cooperative Distributed B+Tree (OCDBT) on top of a base key-value store to implement ACID transactions.
It still stores all data using the native Zarr Format, but versions keys at the store level.
-- **Icechunk**
+- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format spec).
+Nevertheless the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC, so libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, which they effectively can treat as version-controlled zarr stores.
+Icechunk also integrates with VirtualiZarr, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
From 04699205aa835be00522880443225214d3d878f3 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:01:15 -0400
Subject: [PATCH 10/29] NCZarr and Lindi
---
components/index.md | 7 ++++---
1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/components/index.md b/components/index.md
index c16d843..4a57d59 100644
--- a/components/index.md
+++ b/components/index.md
@@ -50,9 +50,9 @@ Here are a few interesting zarr-related projects, which selectively make use of
- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys. It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
-- **VirtualiZarr**
+- **VirtualiZarr**
-- **NCZarr** is in some sense the opposite of VirtualiZarr - it
+- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API. Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class. [NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library. Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
- **Tensorstore** is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
As it can write to multiple different storage sytems, it effectively has its own set of concrete store implementations.
@@ -60,5 +60,6 @@ Additional features are provided, notably using an Optionally-Cooperative Distri
It still stores all data using the native Zarr Format, but versions keys at the store level.
- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format spec).
-Nevertheless the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC, so libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, which they effectively can treat as version-controlled zarr stores.
+Whilst the core icechunk client is written in rust, the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC.
+Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
Icechunk also integrates with VirtualiZarr, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
From 3ab1fbd0227a9d8ecb2492fc49f9b2186a97b652 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:06:39 -0400
Subject: [PATCH 11/29] add virtualizarr
---
components/index.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/components/index.md b/components/index.md
index 4a57d59..f848aaa 100644
--- a/components/index.md
+++ b/components/index.md
@@ -50,7 +50,9 @@ Here are a few interesting zarr-related projects, which selectively make use of
- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys. It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
-- **VirtualiZarr**
+- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk, which reside inside files stored in other formats such as netCDF.
+These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
+VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to the data using zarr-python's API without copying it.
- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API. Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class. [NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library. Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
@@ -62,4 +64,4 @@ It still stores all data using the native Zarr Format, but versions keys at the
- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format spec).
Whilst the core icechunk client is written in rust, the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC.
Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
-Icechunk also integrates with VirtualiZarr, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
+Icechunk also integrates with VirtualiZarr as a serialization format for the byte range references, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
From 1bb62381b1206e4797a0e3d2ee765ff40222fd36 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:07:50 -0400
Subject: [PATCH 12/29] format onto one sentence per line
---
components/index.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index f848aaa..91694ee 100644
--- a/components/index.md
+++ b/components/index.md
@@ -54,7 +54,9 @@ Here are a few interesting zarr-related projects, which selectively make use of
These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to the data using zarr-python's API without copying it.
-- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API. Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class. [NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library. Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
+- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
+Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class.
+[NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library. Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
- **Tensorstore** is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
As it can write to multiple different storage sytems, it effectively has its own set of concrete store implementations.
From 5a655e683e814a0661062ea8bacf2680389b7b42 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:10:01 -0400
Subject: [PATCH 13/29] virtualizarr clarifications
---
components/index.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index 91694ee..b784964 100644
--- a/components/index.md
+++ b/components/index.md
@@ -52,7 +52,7 @@ Here are a few interesting zarr-related projects, which selectively make use of
- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk, which reside inside files stored in other formats such as netCDF.
These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
-VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to the data using zarr-python's API without copying it.
+VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to non-zarr-formatted data using zarr-python's API, without duplicating the original data.
- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class.
From 13fc385b2a140d09213f9226004b176eb04fc48a Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:10:21 -0400
Subject: [PATCH 14/29] linebreak
---
components/index.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index b784964..3ad5c42 100644
--- a/components/index.md
+++ b/components/index.md
@@ -48,7 +48,8 @@ One of Zarr's greatest strengths is its flexibility, or "hackability".
In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
Here are a few interesting zarr-related projects, which selectively make use of a subset of different zarr components, both abstract and concrete.
-- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys. It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
+- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys.
+It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk, which reside inside files stored in other formats such as netCDF.
These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
From 3514d41a1d06a9acab9518df6fb4338273babdd8 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:14:02 -0400
Subject: [PATCH 15/29] don't imply metadata are serialized as byte streams
---
components/index.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index 3ad5c42..51d7377 100644
--- a/components/index.md
+++ b/components/index.md
@@ -15,7 +15,7 @@ Zarr-related projects all follow the Zarr specification (and hence data model),
These abstract components together describe what type of data can be stored in zarr, and how to store it, without assuming you are working in a particular programming language, or with a particular storage system.
-**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data and metadata as byte streams via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface).
+**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data as byte streams as well as store metadata via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface).
A system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) is used to describe the encoding and serialization steps.
**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html).
From 81791056ac9d9f73c68ba3aed066f8048516b757 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:28:32 -0400
Subject: [PATCH 16/29] add to sidebar and fix link
---
_data/navigation.yml | 2 ++
index.md | 2 +-
2 files changed, 3 insertions(+), 1 deletion(-)
diff --git a/_data/navigation.yml b/_data/navigation.yml
index 1eaf5d1..f3b7547 100644
--- a/_data/navigation.yml
+++ b/_data/navigation.yml
@@ -27,6 +27,8 @@ sidebar:
url: '/blog'
- title: "Community"
url: '/community'
+ - title: "Components"
+ url: '/components'
- title: "Conventions"
url: '/conventions'
- title: "Datasets"
diff --git a/index.md b/index.md
index 393d293..ea76ba7 100644
--- a/index.md
+++ b/index.md
@@ -32,7 +32,7 @@ can be represented as a key-value store, including most commonly POSIX file
systems and cloud object storage but also zip files as well as relational and
document databases.
-For more details read about the various [Components of Zarr](https://zarr.dev/componenets/),
+For more details read about the various [Components of Zarr](https://zarr.dev/components/),
see the canonical [Zarr-Python](https://github.com/zarr-developers/zarr-python) implementation,
or look through [other Zarr implementations](https://zarr.dev/implementations/) for one in your preferred language.
From d16deab5baa3adc696da9700caa4e0368a198e82 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:40:48 -0400
Subject: [PATCH 17/29] fix some links
---
components/index.md | 2 +-
index.md | 2 +-
2 files changed, 2 insertions(+), 2 deletions(-)
diff --git a/components/index.md b/components/index.md
index 51d7377..2b3086d 100644
--- a/components/index.md
+++ b/components/index.md
@@ -3,7 +3,7 @@ layout: single
author_profile: false
title: Zarr Components
sidebar:
- title: "Content"
+ title: "Components"
nav: sidebar
---
diff --git a/index.md b/index.md
index ea76ba7..29b39b8 100644
--- a/index.md
+++ b/index.md
@@ -32,7 +32,7 @@ can be represented as a key-value store, including most commonly POSIX file
systems and cloud object storage but also zip files as well as relational and
document databases.
-For more details read about the various [Components of Zarr](https://zarr.dev/components/),
+For more details read about the various [components of Zarr](https://zarr.dev/components/),
see the canonical [Zarr-Python](https://github.com/zarr-developers/zarr-python) implementation,
or look through [other Zarr implementations](https://zarr.dev/implementations/) for one in your preferred language.
From bda516fbcbb5fbe6f4d792a33d79f618d86ac80a Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Fri, 4 Apr 2025 20:44:36 -0400
Subject: [PATCH 18/29] redirection layer
---
components/index.md | 3 ++-
1 file changed, 2 insertions(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index 2b3086d..f8f250c 100644
--- a/components/index.md
+++ b/components/index.md
@@ -51,9 +51,10 @@ Here are a few interesting zarr-related projects, which selectively make use of
- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys.
It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
-- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk, which reside inside files stored in other formats such as netCDF.
+- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk inside "chunk manifests", which reside inside files stored in other formats such as netCDF.
These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to non-zarr-formatted data using zarr-python's API, without duplicating the original data.
+The manifests effectively act as an indirection layer between the zarr-spec-compliant key interface, and the actual location of the chunks in storage.
- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class.
From 6d73eceac0349afa5369c4abeceef4e9af2ac602 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 13:00:52 -0400
Subject: [PATCH 19/29] specification->protocol
---
components/index.md | 11 ++++++-----
1 file changed, 6 insertions(+), 5 deletions(-)
diff --git a/components/index.md b/components/index.md
index f8f250c..c785940 100644
--- a/components/index.md
+++ b/components/index.md
@@ -9,13 +9,14 @@ sidebar:
Zarr consists of several components, both abstract and concrete.
These span both the physical storage layer and the conceptual structural layer.
-Zarr-related projects all follow the Zarr specification (and hence data model), but otherwise may choose to implement other layers however they wish.
+Zarr-related projects all use the Zarr Protocol (and hence data model), described by the [Zarr Specification](), but otherwise may choose to implement other layers however they wish.
## Abstract components
These abstract components together describe what type of data can be stored in zarr, and how to store it, without assuming you are working in a particular programming language, or with a particular storage system.
-**Specification**: All zarr-related projects obey the [Zarr Specification](https://zarr-specs.readthedocs.io/), which formally describes how to serialize and de-serialize array data as byte streams as well as store metadata via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface).
+**Protocol**: All zarr-related projects use the Zarr Protocol, described in the [Zarr Specification](https://zarr-specs.readthedocs.io/), which allows transfer of chunked array data and metadata between devices (or between memory regions of the same device).
+The protocol works by serializing and de-serializing array data as byte streams and storing both this data and accompanying metadata via an [Abstract Key-Value Store Interface](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#abstract-store-interface).
A system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) is used to describe the encoding and serialization steps.
**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html).
@@ -32,11 +33,11 @@ Concrete implementations of the abstract components can be implemented in any la
The canonical reference implementation is [Zarr-Python](https://github.com/zarr-developers/zarr-python), but there are many [other implementations](https://zarr.dev/implementations/).
Zarr-Python contains reference examples of useful constructs that can be re-implemented in other languages.
-**Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, based on a `MutableMapping`-like API.
+**Abstract Base Classes**: Zarr-python's [`zarr.abc`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains abstract base classes enforcing a particular python realization of the specification's Key-Value Store interface, using a `Store` ABC, which is based on a `MutableMapping`-like API.
This component is concrete in the sense that it is implemented in a specific programming language, and enforces particular syntax for getting and setting values in a key-value store.
-**Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems, such as a local filesystem or object storage.
-These write data in the Native Zarr Format.
+**Store Implementations**: Zarr-python's [`zarr.storage`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains concrete implementations of the `Store` ABC for interacting with particular storage systems.
+The zarr-python store implementations which write to local filesystems or object storage write data in the Native Zarr Format.
It's expected that most users of zarr from python will just use one of these implementations.
**User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface.
From 180b5ba8fafb0bc43647c14e7cef5ba49a5582e2 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 13:32:04 -0400
Subject: [PATCH 20/29] organize sidebar better
---
_data/navigation.yml | 16 +++++++++-------
1 file changed, 9 insertions(+), 7 deletions(-)
diff --git a/_data/navigation.yml b/_data/navigation.yml
index f3b7547..bbc5fcb 100644
--- a/_data/navigation.yml
+++ b/_data/navigation.yml
@@ -19,7 +19,15 @@ sidebar:
url: "#sponsorship"
- title: "Videos"
url: "#videos"
- - title: Subpages
+ - title: Technical
+ children:
+ - title: "Components"
+ url: '/components'
+ - title: "Implementations"
+ url: '/implementations'
+ - title: "Specification"
+ url: https://zarr-specs.readthedocs.io/
+ - title: Community
children:
- title: "Adopters"
url: "/adopters"
@@ -27,19 +35,13 @@ sidebar:
url: '/blog'
- title: "Community"
url: '/community'
- - title: "Components"
- url: '/components'
- title: "Conventions"
url: '/conventions'
- title: "Datasets"
url: '/datasets'
- - title: "Implementations"
- url: '/implementations'
- title: "Office Hours"
url: "/office-hours"
- title: "Slides"
url: "/slides"
- - title: "Specification"
- url: https://zarr-specs.readthedocs.io/
- title: "ZEPs"
url: '/zeps'
From a1dcf26c9ad8ae7e9bdf9134afd0088d0aadcd3e Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 15:01:20 -0400
Subject: [PATCH 21/29] create separate page to describe flexibility
---
_data/navigation.yml | 8 +++++---
components/index.md | 30 +-----------------------------
flexibility/index.md | 38 ++++++++++++++++++++++++++++++++++++++
3 files changed, 44 insertions(+), 32 deletions(-)
create mode 100644 flexibility/index.md
diff --git a/_data/navigation.yml b/_data/navigation.yml
index bbc5fcb..bf1f259 100644
--- a/_data/navigation.yml
+++ b/_data/navigation.yml
@@ -23,10 +23,14 @@ sidebar:
children:
- title: "Components"
url: '/components'
+ - title: "Flexibility"
+ url: '/flexibility'
- title: "Implementations"
url: '/implementations'
- title: "Specification"
url: https://zarr-specs.readthedocs.io/
+ - title: "ZEPs"
+ url: '/zeps'
- title: Community
children:
- title: "Adopters"
@@ -42,6 +46,4 @@ sidebar:
- title: "Office Hours"
url: "/office-hours"
- title: "Slides"
- url: "/slides"
- - title: "ZEPs"
- url: '/zeps'
+ url: "/slides"
\ No newline at end of file
diff --git a/components/index.md b/components/index.md
index c785940..c3d3245 100644
--- a/components/index.md
+++ b/components/index.md
@@ -9,7 +9,7 @@ sidebar:
Zarr consists of several components, both abstract and concrete.
These span both the physical storage layer and the conceptual structural layer.
-Zarr-related projects all use the Zarr Protocol (and hence data model), described by the [Zarr Specification](), but otherwise may choose to implement other layers however they wish.
+Zarr-related projects all use the Zarr Protocol (and hence data model), described by the [Zarr Specification](https://zarr-specs.readthedocs.io/), but otherwise may choose to implement other layers however they wish.
## Abstract components
@@ -42,31 +42,3 @@ It's expected that most users of zarr from python will just use one of these imp
**User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface.
This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
-
-## Component Flexibility
-
-One of Zarr's greatest strengths is its flexibility, or "hackability".
-In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
-Here are a few interesting zarr-related projects, which selectively make use of a subset of different zarr components, both abstract and concrete.
-
-- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys.
-It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
-
-- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk inside "chunk manifests", which reside inside files stored in other formats such as netCDF.
-These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
-VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to non-zarr-formatted data using zarr-python's API, without duplicating the original data.
-The manifests effectively act as an indirection layer between the zarr-spec-compliant key interface, and the actual location of the chunks in storage.
-
-- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
-Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class.
-[NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library. Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
-
-- **Tensorstore** is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
-As it can write to multiple different storage sytems, it effectively has its own set of concrete store implementations.
-Additional features are provided, notably using an Optionally-Cooperative Distributed B+Tree (OCDBT) on top of a base key-value store to implement ACID transactions.
-It still stores all data using the native Zarr Format, but versions keys at the store level.
-
-- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format spec).
-Whilst the core icechunk client is written in rust, the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC.
-Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
-Icechunk also integrates with VirtualiZarr as a serialization format for the byte range references, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
diff --git a/flexibility/index.md b/flexibility/index.md
new file mode 100644
index 0000000..7eb0c0b
--- /dev/null
+++ b/flexibility/index.md
@@ -0,0 +1,38 @@
+---
+layout: single
+author_profile: false
+title: Zarr's Flexibility
+sidebar:
+ title: "Flexibility"
+ nav: sidebar
+---
+
+One of Zarr's greatest strengths is its flexibility, or "hackability".
+In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
+
+## Examples
+
+Here are a few zarr-related projects, which each make use of a selected subset of different zarr components to achieve interesting functionality.
+
+- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys.
+It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
+
+- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk inside "chunk manifests", which reside inside files stored in other binary formats such as netCDF.
+These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
+VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to non-zarr-formatted data using zarr-python's API, without duplicating the original data.
+The manifests effectively act as an indirection layer between the zarr-spec-compliant key interface, and the actual location of the chunks in storage.
+
+- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
+Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class.
+[NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library.
+Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
+
+- **Tensorstore** is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
+As it can write to multiple different storage sytems, it effectively has its own set of concrete store implementations.
+Additional features are provided, notably using an Optionally-Cooperative Distributed B+Tree (OCDBT) on top of a base key-value store to implement ACID transactions.
+It still stores all data using the native Zarr Format, but versions keys at the store level.
+
+- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format spec).
+Whilst the core icechunk client is written in rust, the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC.
+Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
+Icechunk also integrates with VirtualiZarr as a serialization format for the byte range references, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
From 1c05a9cdfb4f6e01bdf9b746bfc82f0577848db9 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 15:30:46 -0400
Subject: [PATCH 22/29] add types of flexibility
---
flexibility/index.md | 26 +++++++++++++++++++++-----
1 file changed, 21 insertions(+), 5 deletions(-)
diff --git a/flexibility/index.md b/flexibility/index.md
index 7eb0c0b..9b3d3b3 100644
--- a/flexibility/index.md
+++ b/flexibility/index.md
@@ -7,12 +7,27 @@ sidebar:
nav: sidebar
---
-One of Zarr's greatest strengths is its flexibility, or "hackability".
-In addition to the generality of using key-value stores as the main abstraction, individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
+One of Zarr's greatest strengths is its flexibility, or "hackability".
+
+## Types of flexibility
+
+This flexibility comes in several forms:
+- The Zarr protocol is device agnostic.
+- The Zarr data model is domain agnostic.
+- Key-value stores are an almost universal abstraction in data systems, and so can almost always be mapped to existing system interfaces.
+- The Zarr format on-disk is extremely simple.
+- Storing each chunk under a different key allows implementations to scale their IO throughput in a variety of simple ways.
+- The reference Zarr implementation is written in Python, a very hackable language, with ABCs you can use when creating new store implementations.
+- Components are seperated: the protocol, file format, standard API, ABC, and store implementations are all separate.
+- There is no requirement to use more than one zarr component - individual projects can achieve powerful functionality by intelligently using only some of the Zarr components.
+- You can define your own codecs.
+- You are free to create your own domain-specific metadata standard and enforce it upon zarr stores however you like.
+- Zarr v3 has nascent support for other extension points, including defining your own type of chunk grid, data types, and more.
## Examples
-Here are a few zarr-related projects, which each make use of a selected subset of different zarr components to achieve interesting functionality.
+Here are a few zarr-related software projects, which each make use of a selected subset of different zarr components to achieve interesting functionality.
+These particular projects are more than simply zarr implementations written in a different language (you can find a [list of implementations here](https://zarr.dev/implementations/)).
- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys.
It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
@@ -32,7 +47,8 @@ As it can write to multiple different storage sytems, it effectively has its own
Additional features are provided, notably using an Optionally-Cooperative Distributed B+Tree (OCDBT) on top of a base key-value store to implement ACID transactions.
It still stores all data using the native Zarr Format, but versions keys at the store level.
-- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format spec).
+- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format specification).
Whilst the core icechunk client is written in rust, the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC.
Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
-Icechunk also integrates with VirtualiZarr as a serialization format for the byte range references, together allowing data stored in non-zarr formats to be committed to a persistent icechunk store and read back via the zarr-python API without copying the original data.
+Icechunk also integrates with VirtualiZarr as a serialization format for byte range references.
+Together they allow data stored in non-zarr formats to be committed to a persistent icechunk store and read back later via the zarr-python API without duplicating the original data chunks.
From b487799782605258aa454e4e8c4e9f0d79565ba8 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 15:36:41 -0400
Subject: [PATCH 23/29] more links between pages
---
components/index.md | 2 ++
flexibility/index.md | 3 +++
2 files changed, 5 insertions(+)
diff --git a/components/index.md b/components/index.md
index c3d3245..c8e1fee 100644
--- a/components/index.md
+++ b/components/index.md
@@ -42,3 +42,5 @@ It's expected that most users of zarr from python will just use one of these imp
**User API**: Zarr-python's [`zarr.api`](https://zarr.readthedocs.io/en/stable/api/zarr/abc/index.html) module contains functions and classes for interacting with any concrete implementation of the `zarr.abc.Store` interface.
This allows user applications to use a standard zarr API to read and write from a variety of common storage systems.
+
+These various components allow for a huge amount of [flexibility](https://zarr.dev/flexibility/).
\ No newline at end of file
diff --git a/flexibility/index.md b/flexibility/index.md
index 9b3d3b3..17770aa 100644
--- a/flexibility/index.md
+++ b/flexibility/index.md
@@ -23,6 +23,7 @@ This flexibility comes in several forms:
- You can define your own codecs.
- You are free to create your own domain-specific metadata standard and enforce it upon zarr stores however you like.
- Zarr v3 has nascent support for other extension points, including defining your own type of chunk grid, data types, and more.
+- [Zarr Enhancement Proposals](https://zarr.dev/zeps/) (or "ZEPs") provide a mechanism for enhancing or adding to the specification in a community-standardized way.
## Examples
@@ -52,3 +53,5 @@ Whilst the core icechunk client is written in rust, the `icechunk-python` client
Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
Icechunk also integrates with VirtualiZarr as a serialization format for byte range references.
Together they allow data stored in non-zarr formats to be committed to a persistent icechunk store and read back later via the zarr-python API without duplicating the original data chunks.
+
+We also have a full list of [zarr implementations](https://zarr.dev/implementations/).
\ No newline at end of file
From 1f9613c3c6ffb2ac83f6f69ef41991ffca6abd8f Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 15:43:36 -0400
Subject: [PATCH 24/29] link to external example libraries
---
flexibility/index.md | 10 +++++-----
1 file changed, 5 insertions(+), 5 deletions(-)
diff --git a/flexibility/index.md b/flexibility/index.md
index 17770aa..d870995 100644
--- a/flexibility/index.md
+++ b/flexibility/index.md
@@ -33,24 +33,24 @@ These particular projects are more than simply zarr implementations written in a
- **MongoDBStore** is a concrete store implementation in python, which stores values in a MongoDB NoSQL database under zarr keys.
It is therefore spec-compliant, and can be interacted with via the zarr-python user API, but does not write data in the native zarr format.
-- **VirtualiZarr** provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk inside "chunk manifests", which reside inside files stored in other binary formats such as netCDF.
+- [**VirtualiZarr**](https://github.com/zarr-developers/VirtualiZarr) provides a concrete store implementation in python (the `ManifestStore`) which stores references to locations and byte ranges of chunks on disk inside "chunk manifests", which reside inside files stored in other binary formats such as netCDF.
These references are generated by "readers", which do the job of parsing the file structure and mapping the contents to the zarr data model.
VirtualiZarr therefore eschews the native zarr format but still provides spec-compliant access to non-zarr-formatted data using zarr-python's API, without duplicating the original data.
The manifests effectively act as an indirection layer between the zarr-spec-compliant key interface, and the actual location of the chunks in storage.
-- **NCZarr** and **Lindi** can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
+- [**NCZarr**](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) and [**Lindi**](https://github.com/NeurodataWithoutBorders/lindi) can both in some sense be considered as the opposite of VirtualiZarr - they allow interacting with zarr-formatted data on disk via a non-zarr API.
Lindi maps zarr's data model to the HDF data model and allows access to via the `h5py` library through the [`LindiH5pyFile`](https://github.com/NeurodataWithoutBorders/lindi/blob/b125c111880dd830f2911c1bc2084b2de94f6d71/lindi/LindiH5pyFile/LindiH5pyFile.py#L28) class.
[NCZarr](https://docs.unidata.ucar.edu/nug/current/nczarr_head.html) allows interacting with zarr-formatted data via the netcdf-c library.
Note that both libraries implement optional additional optimizations by going beyond the zarr specification and format on disk, which is not recommended.
-- **Tensorstore** is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
+- [**Tensorstore**](https://github.com/google/tensorstore) is a general storage library written in C++ that can write to the Zarr format (so is a spec-compliant non-python "native" store implementation) but also to other array formats such as N5.
As it can write to multiple different storage sytems, it effectively has its own set of concrete store implementations.
Additional features are provided, notably using an Optionally-Cooperative Distributed B+Tree (OCDBT) on top of a base key-value store to implement ACID transactions.
It still stores all data using the native Zarr Format, but versions keys at the store level.
-- **Icechunk** is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format specification).
+- [**Icechunk**](https://icechunk.io/) is a cloud-native tensor storage engine which also provides ACID transactions, but does so via indirection between a zarr-spec-compliant key-value store interface and a specialized non-zarr-native storage layout on-disk (for which Icechunk has it's own format specification).
Whilst the core icechunk client is written in rust, the `icechunk-python` client implements a concrete subclass of the zarr-python `Store` ABC.
-Therefore libraries such as xarray can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
+Therefore libraries such as [xarray](https://xarray.dev/) can use the zarr-python user API to read and write to icechunk stores, effectively treating them as version-controlled zarr stores.
Icechunk also integrates with VirtualiZarr as a serialization format for byte range references.
Together they allow data stored in non-zarr formats to be committed to a persistent icechunk store and read back later via the zarr-python API without duplicating the original data chunks.
From 22df3583973302269241bb767194609d8b9d37d2 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 15:48:30 -0400
Subject: [PATCH 25/29] add flexibility as a feature
---
flexibility/index.md | 1 +
index.md | 1 +
2 files changed, 2 insertions(+)
diff --git a/flexibility/index.md b/flexibility/index.md
index d870995..e02ece0 100644
--- a/flexibility/index.md
+++ b/flexibility/index.md
@@ -8,6 +8,7 @@ sidebar:
---
One of Zarr's greatest strengths is its flexibility, or "hackability".
+This largely comes from the separation of distinct [Zarr Components](https://zarr.dev/components/), but there are a range of other properties that make zarr flexible too.
## Types of flexibility
diff --git a/index.md b/index.md
index 29b39b8..5f444a7 100644
--- a/index.md
+++ b/index.md
@@ -49,6 +49,7 @@ or look through [other Zarr implementations](https://zarr.dev/implementations/)
* Store arrays in memory, on disk, inside a Zip file, on S3, etc.
* Read and write arrays concurrently from multiple threads or processes.
* Organize arrays into hierarchies via annotatable groups.
+* Extend easily thanks to the [flexible design](https://zarr.dev/flexibility/).
## Sponsorship
From 76cf0efc8572175afdcd1e319e327c15a68c3243 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 15:55:14 -0400
Subject: [PATCH 26/29] add more applications
---
index.md | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git a/index.md b/index.md
index 5f444a7..c479c4c 100644
--- a/index.md
+++ b/index.md
@@ -40,7 +40,9 @@ or look through [other Zarr implementations](https://zarr.dev/implementations/)
* Simple and fast serialization of NumPy-like arrays, accessible from languages including Python, C, C++, Rust, Javascript and Java
* Multi-scale n-dimensional image storage, e.g. in light and electron microscopy
-* Geospatial rasters, e.g. following the NetCDF / CF metadata conventions
+* Gridded scientific data in various domains, such as CFD or Plasma Physics
+* Geospatial rasters, e.g. following the NetCDF data model
+* Checkpointing ML model weights.
## Features
From fbdfcda9c01dcd84fc899dbf10dbafef4a27e03c Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 16:02:48 -0400
Subject: [PATCH 27/29] split the applications and the features better
---
index.md | 6 ++++--
1 file changed, 4 insertions(+), 2 deletions(-)
diff --git a/index.md b/index.md
index c479c4c..ead9e34 100644
--- a/index.md
+++ b/index.md
@@ -38,14 +38,16 @@ or look through [other Zarr implementations](https://zarr.dev/implementations/)
## Applications
-* Simple and fast serialization of NumPy-like arrays, accessible from languages including Python, C, C++, Rust, Javascript and Java
* Multi-scale n-dimensional image storage, e.g. in light and electron microscopy
+* Genomics data, e.g. for quantitative and population genetics.
* Gridded scientific data in various domains, such as CFD or Plasma Physics
-* Geospatial rasters, e.g. following the NetCDF data model
+* Geospatial rasters, e.g. following the NetCDF data model.
* Checkpointing ML model weights.
## Features
+* Serialize NumPy-like arrays in a simple and fast way.
+* Access from languages including Python, C, C++, Rust, Javascript and Java.
* Chunk multi-dimensional arrays along any dimension.
* Compress array chunks via an extensible system of compressors.
* Store arrays in memory, on disk, inside a Zip file, on S3, etc.
From 87cb05c18601374fde81ac410534e4f1fb2b3e66 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sat, 5 Apr 2025 16:09:02 -0400
Subject: [PATCH 28/29] be consistent about bullet points
---
index.md | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/index.md b/index.md
index ead9e34..10c9503 100644
--- a/index.md
+++ b/index.md
@@ -38,9 +38,9 @@ or look through [other Zarr implementations](https://zarr.dev/implementations/)
## Applications
-* Multi-scale n-dimensional image storage, e.g. in light and electron microscopy
+* Multi-scale n-dimensional image storage, e.g. in light and electron microscopy.
* Genomics data, e.g. for quantitative and population genetics.
-* Gridded scientific data in various domains, such as CFD or Plasma Physics
+* Gridded scientific data in various domains, such as CFD or Plasma Physics.
* Geospatial rasters, e.g. following the NetCDF data model.
* Checkpointing ML model weights.
From d499af0c1c16bbe6a6d474ca8a6cfc92fb6ef807 Mon Sep 17 00:00:00 2001
From: Tom Nicholas <tom@earthmover.io>
Date: Sun, 6 Apr 2025 09:23:41 -0700
Subject: [PATCH 29/29] Spelling
Co-authored-by: Davis Bennett <davis.v.bennett@gmail.com>
---
components/index.md | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/components/index.md b/components/index.md
index c8e1fee..7f8e97a 100644
--- a/components/index.md
+++ b/components/index.md
@@ -20,7 +20,7 @@ The protocol works by serializing and de-serializing array data as byte streams
A system of [Codecs](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#chunk-encoding) is used to describe the encoding and serialization steps.
**Data Model**: The specification's description of the [Stored Representation](https://zarr-specs.readthedocs.io/en/latest/v3/core/v3.0.html#stored-representation) implies a particular data model, based on the [HDF Abstract Data Model](https://support.hdfgroup.org/documentation/hdf5/latest/_h5_d_m__u_g.html).
-It consists of a heirarchical tree of groups and arrays, with optional arbitrary metadata at every node. This model is completely domain-agnostic.
+It consists of a hierarchical tree of groups and arrays, with optional arbitrary metadata at every node. This model is completely domain-agnostic.
**Format**: If the keys in the abstract key-value store interface are mapped unaltered to paths in a POSIX filesystem or prefixes in object storage, the data written to disk will follow the "Native Zarr Format".
Most, but not all, zarr implementations will serialize to this format.