Get Started

Zarr.jl is a Julia package for working with chunked, compressed, N-dimensional arrays, compatible with the Zarr format used across Python, Rust, and other ecosystems.

Installation

Install Julia v1.10 or above. Zarr.jl is available through the Julia package manager. You can enter it by pressing ] in the REPL and then typing add Zarr:

(@v1.x) pkg> add Zarr

alternatively, you can also do:

import Pkg
Pkg.add("Zarr")

It is recommended to check the version of Zarr.jl you have installed with the status command:

(@v1.x) pkg> status Zarr

or:

import Pkg
Pkg.status("Zarr")

Status `~/work/Zarr.jl/Zarr.jl/docs/Project.toml`
  [0a941bbe] Zarr v0.9.6 `~/work/Zarr.jl/Zarr.jl`

Where not shown explicitly, this documentation assumes using Zarr has been evaluated in your session:

using Zarr

Creating Arrays

Use zcreate for new arrays, zzeros for zero-initialized ones, or ZArray to wrap an existing Julia array:

using Zarr

z = zcreate(Float32, 1000, 1000;
    chunks=(100, 100),
    fill_value=Float32(0),
    zarr_format=3,
    path="example_v3.zarr")

ZArray{Float32} of size 1000 x 1000

you can also wrap an existing Julia array:

using Zarr
z = ZArray(rand(Float64, 100, 100))

ZArray{Float64} of size 100 x 100

zinfo(z)

Type                : ZArray
Data type           : Float64
Shape               : (100, 100)
Chunk Shape         : (100, 100)
Order               : C
Read-Only           : false
Compressor          : Zarr.BloscCompressor(0, 5, "lz4", 1)
Filters             : nothing
Store type          : Dictionary Storage
No. bytes           : 80000
No. bytes stored    : 70102
Storage ratio       : 1.1411942597928733
Chunks initialized  : 1/1

Reading and Writing

using Zarr

z = zcreate(Float32, 1000, 1000;
    chunks=(100, 100),
    fill_value=Float32(0),
    path="example.zarr")

z[:, :] = rand(Float32, 1000, 1000)  # write entire array
z[1, :] = 1:1000                     # write a row

subset = z[1:3, 1:10]                # read a subregion

3×10 Matrix{Float32}:
 1.0       2.0       3.0       4.0       …  8.0       9.0        10.0
 0.563979  0.382422  0.408976  0.954415     0.460318  0.0325561   0.619256
 0.805412  0.345718  0.47378   0.352676     0.945457  0.115197    0.65983

Opening Existing Arrays

Zarr automatically detects v2 or v3 format on open:

z = zopen("example.zarr")

println(size(z))    # (1000, 1000)
println(eltype(z))  # Float32

(1000, 1000)
Float32

WARNING

zopen throws an ArgumentError if the path does not exist. Make sure the path points to a valid Zarr store.

Missing Values

Use fill_value and fill_as_missing together to handle missing data:

using Zarr

z = zcreate(Int64, 10, 10;
    chunks=(5, 2),
    fill_value=-1,
    fill_as_missing=true)

z[:, 1] = 1:10          # write a column
z[:, 2] .= missing      # mark a column as missing

println(eltype(z))               # Union{Int64, Missing}
println(all(ismissing, z[:, 2])) # true

Union{Missing, Int64}
true

Re-open with or without missing support:

# treat fill_value as missing
z = zopen("example.zarr", fill_as_missing=true)

ZArray{Union{Missing, Float32}} of size 1000 x 1000

or treat fill_value as a regular value

z = zopen("example.zarr")

ZArray{Float32} of size 1000 x 1000

Compression

Zarr uses Blosc (lz4, level 5) by default. Several compressors are available:

using Zarr

z = zcreate(Int32, 1000, 1000;
    chunks=(100, 100),
    compressor=Zarr.BloscCompressor(cname="zstd", clevel=3, shuffle=true),
    fill_value=Int32(0),
    path="blosc.zarr")

z[:,:] = Int32(1):Int32(1000*1000)
storageratio(z)

51.39802631578947

Resizing and Appending

using Zarr

z = zzeros(Int64, 10, 10; chunks=(5, 2), fill_value=-1)

ZArray{Int64} of size 10 x 10

grow first dimension

resize!(z, 20, 10)
z

ZArray{Int64} of size 20 x 10

appends columns

append!(z, rand(Int64, 20, 5))
z

ZArray{Int64} of size 20 x 15

append a row

append!(z, rand(Int64, 15), dims=1)
z

ZArray{Int64} of size 21 x 15

Groups

Zarr allows you to create hierarchical groups, similar to directories:

using Zarr

store = Zarr.DirectoryStore("experiment.zarr")
g = zgroup(store, "", Zarr.ZarrFormat(3))

zcreate(Float64, g, "temperature", 100, 100; chunks=(50, 50), fill_value=0.0)
zcreate(Float64, g, "precipitation", 100, 100; chunks=(50, 50), fill_value=0.0)
g

ZarrGroup at DirectoryStore("experiment.zarr") and path 
Variables: precipitation temperature

Navigate into a group to access its arrays:

temp = g["temperature"]
println(size(temp))  # (100, 100)

(100, 100)

Storage Backends

Zarr supports several storage backends out of the box:

z = zopen("example.zarr")

See Storage Backends for full details on credentials and configuration.

TIP

Ready for more? Head to the User Guide for a deeper dive.