Internals & Design

TYPE_AS_DATA::ScopedValue{Bool}

Signal to the jlconvert function that the type being read will be treated as either data or a type. This is needed to allow a custom typemap function to return an Upgrade object when the type is going to be used for reconstructing an instance. (as a type)

CommittedDatatype <: H5Datatype

Reference to a shared datatype message (stored elsewhere in a file). These are stored in the _types group and indexed.

CustomSerialization{T,S}

On-disk representation for data that is written as if it were of Julia type T, but is read as type S.

GlobalHeap

Represents an HDF5 global heap structure.

Group(file::JLDFile, name::String)
Group(file::Group, name::String)

Construct a Group in file with name name. Groups are JLD2s equivalent of folders and may be nested, so file itself may alread be a Group or a JLDFile file handle.

Example usage

jldopen("example.jld2", "w") do f
    g = Group(f, "subgroup")
    g["data"] = 42
end

jldopen("example.jld2") do f
    g = f["subgroup"]
    f["subgroup/data"] == g["data"]
end

Keyword arguments:

• est_num_entries::Int = 4
• est_link_name_len::Int = 8

Determine how much (additional) empty space should be allocated for the group description. (list of entries) This can be useful for performance when one expects to append many additional datasets after first writing the file.

Group(f::JLDFile, name::AbstractString)

Construct an empty group named name at the top level of JLDFile f.

Group(g::Group, name::AbstractString)

Construct a group named name as a child of group g.

abstract type H5Datatype

Supertype of all HDF5 datatypes.

HeaderMessage

Helper struct to read and write the first part of a header message.

mutable struct HeaderMessageIterator{IO}
HeaderMessageIterator(f::JLDFile, offset::RelOffset)

Implements an iterator over header messages.

Hmessage{IO}

Representation of a Header Message in memory. Provides getproperty access to the fields of the message. Can also be used to construct and write custom messages.

IndirectPointer

When writing data, we may need to enlarge the memory mapping, which would invalidate any memory addresses arising from the old mmap pointer. IndirectPointer holds an offset relative to the MemoryBackedIO. It defers computing a memory address until converted to a Ptr{T}, so the memory mapping can be enlarged and addresses will remain valid.

InlineUnionEl{T1,T2}(mask::UInt8, t1::T1, t2::T2)

Custom serialization struct for two member isbits union fields e.g. in other structs or arrays. To indicate that t1 is relevant the mask takes the value UInt8(0) and for t2 the mask takes the value UInt8(255).

JLDFile{T<:IO}

JLD file object.

JLDWriteSession{T}

A JLDWriteSession keeps track of references to serialized objects. If T is a Dict, h5offset maps an object ID (returned by calling objectid) to th RelOffset of the written dataset. If it is Union{}, then references are not tracked, and objects referenced multiple times are written multiple times.

MemoryBackedIO <: IO

Abstract type for IO objects that are backed by memory in such a way that one can use pointer based unsafe_load and unsafe_store! operations after ensuring that there is enough memory allocated.

It needs to provide:

• getproperty(io, :curptr) to get the current pointer
• ensureroom(io, nb) to ensure that there are at least nb bytes available
• position(io) to get the current (zero-based) position
• seek(io, pos) to set the current position (zero-based)

Message{IO}

Representation of a Message in memory. Provides getproperty access

ReadRepresentation{T,ODR}

A type encoding both the Julia type T and the on-disk (HDF5) representation ODR.

RelOffset

Represents an HDF5 relative offset. This differs from a file offset (used elsewhere) in that it is relative to the superblock base address. fileoffset and h5offset convert between RelOffsets and file offsets.

SharedDatatype <: H5Datatype

Reference to a shared datatype message (stored elsewhere in a file).

Upgrade(T)

Specify an upgrade path for serialized structs using the typemap keyword argument and rconvert.

attributes(dset::Dataset; plain::Bool=false)

Return the attributes of a dataset as an OrderedDict. If plain is set to true then the values are returned as stored in the dataset object.

behead(T)

Given a UnionAll type, recursively eliminates the where clauses

bufferpos(io::Union{BufferedReader, BufferedWriter})

Get the current position in the buffer.

construct_array(io::IO, eltype, ndims::Int)

Construct array by reading ndims dimensions from io. Assumes io has already been seeked to the correct position.

constructrr(f::JLDFile, T::DataType, dt::CompoundType, attrs::Vector{ReadAttribute},
            hard_failure::Bool=false)

Constructs a ReadRepresentation for a given type. This is the generic method for all types not specially handled below.

If hard_failure is true, then throw a TypeMappingException instead of attempting reconstruction. This helps in cases where we can't know if reconstructed parametric types will have a matching memory layout without first inspecting the memory layout.

create_dataset(parent, path, datatype, dataspace; kwargs...)

Arguments: - parent::Union{JLDfile, Group}: Containing group of new dataset - path: Path to new dataset relative to parent. If path is nothing, the dataset is unnamed. - datatype: Datatype of new dataset (element type in case of arrays) - dataspace: Dimensions or Dataspace of new dataset

Keyword arguments: - layout: DataLayout of new dataset - filters: FilterPipeline for describing the compression pipeline

default_typemap(f::JLDFile, typepath::String, params)

Default type mapping function used by JLD2 to resolve data types read from files.

Arguments:

• f::JLDFile: The JLD file being read.
• typepath::String: The path to the type as a string, e.g. "Main.MyModule.MyType".
• params: A list of type parameters for the type (may be empty).

fileoffset(f::JLDFile, x::RelOffset)

Converts an offset x relative to the superblock of file f to an absolute offset.

find_type(typepath::String)

Finds a type in the loaded modules by its path as a string. The type path should be a dot-separated string, e.g. "Main.MyModule.MyType". If the type is found, it returns the corresponding DataType or UnionAll. If the type is not found, it returns nothing.

flag2uint(flag::UInt8)

I Map the lowest to bits of flag to a UInt type, mapping 0 to UInt8, 1 to UInt16, 2 to UInt32, and 3 to UInt64.

get_dataset(parent::Union{JLDFile, Group}, name::String)

Get a stored dataset from a file by name or path as a Dataset object. This may be useful for inspecting the metadata incl. types of a dataset.

h5offset(f::JLDFile, x::Integer)

Converts an absolute file offset x to an offset relative to the superblock of file f.

ismmappable(dset::Dataset)

Check if a dataset can be memory-mapped. This can be useful for large arrays and for editing written arrays.

An Array dataset may be mmapped if: - JLD2.samelayout(T) == true: The element type is isbits and has a size that either 1, 2, 4, or a multiple of 8 bytes. - Uncompressed: Compressed arrays cannot be memory-mapped - Uses a contiguous layout: This is true for all array datasets written by JLD2 with version ≥ v0.4.52 - Windows: The file must be opened in read-only mode. This is a limitation of Mmap on Windows.

isset(flag, bit)

Return true if the bit-th bit of flag is set. (starting from 0)

jld_finalizer(f::JLDFile)

When a JLDFile is finalized, it is possible that the MmapIO has been munmapped, since Julia does not guarantee finalizer order. This means that the underlying file may be closed before we get a chance to write to it.

jldopen(file, mode::AbstractString; iotype=MmapIO, compress=false, typemap=JLD2.default_typemap)

Opens a JLD2 file at path file. Alternatively file may be a suitable IO object.

Options for mode:

• "r": Open for reading only, failing if no file exists
• "r+": Open for reading and writing, failing if no file exists
• "w"/"w+": Open for reading and writing, overwriting the file if it already exists
• "a"/"a+": Open for reading and writing, creating a new file if none exists, but preserving the existing file if one is present

jldsave(filename; kwargs...)
jldsave(filename, compress; kwargs...)
jldsave(filename, compress, iotype; kwargs...)

Creates a JLD2 file at filename and stores the variables given as keyword arguments.

Examples

jldsave("example.jld2"; a=1, b=2, c)

is equivalent to

jldopen("example.jld2", "w") do f
    f["a"] = 1
    f["b"] = 2
    f["c"] = c
end

To choose the io type IOStream instead of the default MmapIO use jldsave(fn, IOStream; kwargs...).

jlwrite(io::IO, x::Tuple)

Attempt to write a tuple to io by writing each element of the tuple in order.

links_size(pairs)

Returns the size of several link messages. pairs is an iterator of String => RelOffset pairs.

load_attributes(f::JLDFile, name::AbstractString)
load_attributes(g::Group, name::AbstractString)
load_attributes(g::Group)
load_attributes(f::JLDFile, offset::RelOffset)

Return a list of attributes attached to the dataset or group.

load_data_or_dict(g::Union{JLDFile,Group}, varname::AbstractString)

Return the value of key varname but if it represents a Group load the group as a nested dictionary.

load_datatypes(f::JLDFile)

Populate f.datatypes and f.jlh5types with all of the committed datatypes from a file. We need to do this before writing to make sure we reuse written datatypes.

load_object(filename)

Returns the only available object from the JLD2 file filename (The stored object name is inconsequential). If the file contains more than one or no objects, the function throws an ArgumentError.

For loading more than one object, use @load macro, jldopen or the FileIO API.

Example

To load the only object from the JLD2 file example.jld2:

hello = "world"
save_object("example.jld2", hello)
hello_loaded = load_object("example.jld2")

loadnesteddict(g::Union{JLDFile, Group})

Return a dictionary with all data contained in group or file. Nested groups are loaded as nested dictionaries.

lookup_offset(g::Group, name::AbstractString) -> RelOffset

Lookup the offset of a dataset in a group. Returns UNDEFINED_ADDRESS if the dataset is not present. Does not inspect unwritten_child_groups.

pathize(g::Group, name::AbstractString, create::Bool) -> Tuple{Group,String}

Converts a path to a group and name object. If create is true, any intermediate groups will be created, and the dataset name will be checked for uniqueness with existing names.

prewrite(f::JLDFile)

Check that a JLD file is actually writable, and throw an error if not. Sets the written flag on the file.

print_header_messages(f::JLDFile, name::AbstractString)
print_header_messages(g::Group, name::AbstractString)
print_header_messages(f::JLDFile, offset::RelOffset)

Prints the header messages of a group or dataset in a file.

printtoc([io::IO,] f::JLDFile [; numlines])

Prints an overview of the contents of f to the IO.

Use the optional numlines parameter to restrict the amount of items listed.

read_array!(v::Array, f::JLDFile, rr)

Fill the array v with the contents of JLDFile f at the current position, assuming a ReadRepresentation rr.

read_attr_data(f::JLDFile, attr::ReadAttribute, expected_datatype::H5Datatype,
               rr::ReadRepresentation)

jlread data from an attribute, assuming a specific HDF5 datatype and ReadRepresentation. If the HDF5 datatype does not match, throws an UnsupportedFeatureException. This allows better type stability while simultaneously validating the data.

read_attr_data(f::JLDFile, attr::ReadAttribute)

jlread data from an attribute.

read_compressed_array!(v::Array, f::JLDFile, rr, data_length::Int, Val(filter_id))

Fill the array v with the compressed contents of JLDFile f at the current position, assuming a ReadRepresentation rr and that the compressed data has length data_length.

read_data(f::JLDFile, dataspace::ReadDataspace, datatype_class::UInt8,
          datatype_offset::Int64, data_offset::Int64[, filters::FilterPipeline,
          header_offset::RelOffset, attributes::Vector{ReadAttribute}])

Read data from a file. If datatype_class is typemax(UInt8), the datatype is assumed to be committed, and datatype_offset points to the offset of the committed datatype's header. Otherwise, datatype_offset points to the offset of the datatype attribute.

read_dataset(dset::Dataset)

Read the data referenced by a dataset.

read_scalar(f::JLDFile, rr, header_offset::RelOffset)

Read raw data representing a scalar with read representation rr from the current position of JLDFile f. header_offset is the RelOffset of the object header, used to resolve cycles.

read_size(io::IO, flags::UInt8)

Loads a variable-length size according to flags

Expects that the first two bits of flags mean:

• 0: The size of the Length of Link Name field is 1 byte.
• 1: The size of the Length of Link Name field is 2 bytes.
• 2: The size of the Length of Link Name field is 4 bytes.
• 3: The size of the Length of Link Name field is 8 bytes.

Returns the size as an Int.

readas(::Type)::Type

Experimental feature: JLD2.readas can be overloaded to override which type a saved type is read as, and is used together with custom serialization using JLD2.writeas.

The typical case is custom serialization of parametric types, where not all type parameters are available during reading. Consider the following example for an anonymous function fun inside a Foo

struct Foo{F<:Function}
    fun::F
end
struct FooSerialization
    fun
end
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)

JLD2.readas(::Type{<:FooSerialization}) = Foo
struct UndefinedFunction <:Function
    fun
end
(f::UndefinedFunction)(args...; kwargs...) = error("The function $(f.fun) is not defined")
function Base.convert(::Type{<:Foo}, f::FooSerialization)
    isa(f.fun, Function) && return Foo(f.fun)
    return Foo(UndefinedFunction(f.fun))
end

If we include these definitions, call jldsave("foo.jld2"; foo=Foo(x->x^2)), restart julia, include the definitions again, and call foo = jldopen("foo.jld2") do io; io["foo"]; end, we get foo::Foo{UndefinedFunction} and foo::FooSerialization with and without defining the JLD2.readas above, respectively.

readmmap(dset::Dataset)

Memory-map a dataset. This can be useful for large arrays and for editing written arrays. See ismmappable for requirements.

save_group(g::Group) -> RelOffset

Stores a group to a file, updating it if it has already been saved. Returns UNDEFINED_ADDRESS if the group was already stored, or the offset of the new group otherwise.

save_object(filename, x)

Stores an object x in a new JLD2 file at filename. If a file exists at this path, it will be overwritten.

Since the JLD2 format requires that all objects have a name, the object will be stored as single_stored_object. If you want to store more than one object, use @save macro, jldopen or the FileIO API.

Example

To save the string hello to the JLD2 file example.jld2:

hello = "world"
save_object("example.jld2", hello)

shorttypestring(::Type{ <:UnknownType})

Convert an UnknownType to a corresponding string. This is only used to create names for reconstructed types.

See also typestring.

size_flag(sz::Integer)::UInt8

Return the flag that represents the smallest integer type that can represent sz. 0 -> UInt8, 1 -> UInt16, 2 -> UInt32, 3 -> UInt64

size_size(sz::Integer)

Return the number of bytes required to represent sz as an unsigned integer that actually exists. (e.g. UInt8, UInt16, UInt32, UInt64)

size_size2(sz::Integer)

Return the number of bytes required to represent sz as an unsigned integer. Note: this does not check if the integer is a valid julia integer.

skip_to_aligned!(io, rel=0)

Skip to nearest position aligned to a multiple of 8 bytes relative to rel.

symbol_length(x::Symbol)

Returns the length of the string represented by x.

typestring(::Type{ <:UnknownType})

Convert an UnknownType to a corresponding string. This is only used for warning during reconstruction errors.

See also shorttypestring.

uintofsize(sz::Integer)

Return the UInt type that has sz bytes.

write_dataset(dataset::Dataset, data)

Write data to file using metadata prepared in the dataset.

write_size(io::IO, sz::Integer)

Write the mininum number of bytes required to represent sz as (valid) unsigned integer.

write_zerobytes(io, n)

Write n zero bytes to io.

@load filename var1 [var2 ...]

Load one or more variables var1,... from JLD2 file filename into the current scope and return a vector of the loaded variable names.

For interactive use, the form @load "somefile.jld2" will load all variables from "somefile.jld2" into the current scope. This form only supports literal file names and should be avoided in more permanent code so that it's clear where the variables come from.

Example

To load the variables hello and foo from the file example.jld2, use

@load "example.jld2" hello foo

@pseudostruct name begin ... end

The @pseudostruct macro is used to define constructor, size computation, show, and and optimized getproperty function for Messages. The allowed syntax elements are:

• @skip(n): Mark n bytes as empty.

@save filename var1 [var2 ...]
@save filename {compress=true} var1 name2=var2

Write one or more variables var1,... from the current scope to a JLD2 file filename.

For interactive use you can save all variables in the current module's global scope using @save filename. More permanent code should prefer the explicit form to avoid saving unwanted variables.

Example

To save the string hello and array xs to the JLD2 file example.jld2:

hello = "world"
xs = [1,2,3]
@save "example.jld2" hello xs

For passing options to the saving command use {}

@save "example.jld2" {compress=true} hello xs

For saving variables under a different name use regular assignment syntax

@save "example.jld2" greeting=hello xarray = xs