Kerchunk.jl

Appearance

Kerchunk.ReferenceStore Type
julia
ReferenceStore(filename_or_dict) <: Zarr.AbstractStore

A ReferenceStore is a "fake filesystem" encoded by some key-value store dictionary, either held in memory, or read from a JSON file in the Kerchunk format.

Generally, you will only need to construct this if you have an in-memory Dict or other representation, or if you want to explicitly modify the store before constructing a ZGroup, which eagerly loads metadata.

Extended help

Implementation

The reference store has several fields:

  • mapper: The actual key-value store that file information (string of base64 bytes, [single uri], [uri, byte_offset, byte_length]) is stored in. The type here is parametrized so this may be mutable if in memory, or immutable, e.g a JSON3.Object.

  • zmetadata: The toplevel Zarr metadata, sometimes stored separately.

  • templates: Key-value store for template expansion, if URLs need to be compressed.

  • cache: Key-value store for explicitly downloaded or otherwise modified keys.

source

Correction interface ​

Kerchunk files often need corrections to the metadata.

For example, the CF-convention add_offset and scale_factor metadata fields are stored as separate variables in the source data, but should ideally be stored as a single Zarr FixedScaleOffset filter so you can get performance as close to native as possible. Some CF datasets also encode an _Unsigned metadata field, which should simply be used to edit the dtype of the Zarr array.

Kerchunk also sometimes places the compressor as the last filter, which is technically compliant with Zarr v3 but is not compliant with Zarr v2. This is corrected by moving the compressor to the compressor field of the metadata, but this has to be done before the Zarr is loaded.

This is the point of the correction interface. As more idiosyncrasies are discovered, they can be added to it.

Missing docstring.

Missing docstring for do_correction!. Check Documenter's build log for details.

Missing docstring.

Missing docstring for add_scale_offset_filter_and_set_mask!. Check Documenter's build log for details.

Missing docstring.

Missing docstring for move_compressor_from_filters!. Check Documenter's build log for details.