Internals & Design

CommittedDatatype

Reference to a shared datatype message (stored elsewhere in a file). These are stored in the _types group and indexed.

CustomSerialization{T,S}

On-disk representation for data that is written as if it were of Julia type T, but is read as type S.

GlobalHeap

Represents an HDF5 global heap structure.

Group(file)

JLD2 group object.

Advanced Usage

Takes two optional keyword arguments: estnumentries::Int=4 estlinkname_len::Int=8 These determine how much (additional) empty space should be allocated for the group description. (list of entries) This can be useful for performance when one expects to append many additional datasets after first writing the file.

Group(f::JLDFile, name::AbstractString)

Construct an empty group named name at the top level of JLDFile f.

Group(g::Group, name::AbstractString)

Construct a group named name as a child of group g.

H5Datatype

Supertype of all HDF5 datatypes.

IndirectPointer

When writing data, we may need to enlarge the memory mapping, which would invalidate any memory addresses arising from the old mmap pointer. IndirectPointer holds a pointer to the startptr field of an MmapIO, and the offset relative to that pointer. It defers computing a memory address until converted to a Ptr{T}, so the memory mapping can be enlarged and addresses will remain valid.

InlineUnionEl{T1,T2}(mask::UInt8, t1::T1, t2::T2)

Custom serialization struct for two member isbits union fields e.g. in other structs or arrays. To indicate that t1 is relevant the mask takes the value UInt8(0) and for t2 UInt8(255)

JLDFile{T<:IO}

JLD file object.

JLDWriteSession{T}

A JLDWriteSession keeps track of references to serialized objects. If T is a Dict, h5offset maps an object ID (returned by calling objectid) to th RelOffset of the written dataset. If it is Union{}, then references are not tracked, and objects referenced multiple times are written multiple times.

ReadRepresentation{T,ODR}

A type encoding both the Julia type T and the on-disk (HDF5) representation ODR.

RelOffset

Represents an HDF5 relative offset. This differs from a file offset (used elsewhere) in that it is relative to the superblock base address. fileoffset and h5offset convert between RelOffsets and file offsets.

SharedDatatype

Reference to a shared datatype message (stored elsewhere in a file).

constructrr(f::JLDFile, T::DataType, dt::CompoundType, attrs::Vector{ReadAttribute}, hard_failure::Bool=false)

Constructs a ReadRepresentation for a given type. This is the generic method for all types not specially handled below.

If hard_failure is true, then throw a TypeMappingException instead of attempting reconstruction. This helps in cases where we can't know if reconstructed parametric types will have a matching memory layout without first inspecting the memory layout.

Upgrade(T)

Specify an upgrade path for serialized structs using the typemapkeyword argument andrconvert`.

behead(T)

Given a UnionAll type, recursively eliminates the where clauses

construct_array{T}(io::IO, ::Type{T}, ::Val{ndims})

Construct array by reading ndims dimensions from io. Assumes io has already been seeked to the correct position.

fileoffset(f::JLDFile, x::RelOffset)

Converts an offset x relative to the superblock of file f to an absolute offset.

group_payload_size(g)

Returns the size of a group payload, including link info, group info, and link messages, but not the object header. Provides space after the last object message for a continuation message.

h5offset(f::JLDFile, x::Integer)

Converts an absolute file offset x to an offset relative to the superblock of file f.

jld_finalizer(f::JLDFile)

When a JLDFile is finalized, it is possible that the MmapIO has been munmapped, since Julia does not guarantee finalizer order. This means that the underlying file may be closed before we get a chance to write to it.

jldopen(fname::AbstractString, mode::AbstractString; iotype=MmapIO, compress=false, typemap=Dict())

Opens a JLD2 file at path fname.

"r": Open for reading only, failing if no file exists "r+": Open for reading and writing, failing if no file exists "w"/"w+": Open for reading and writing, overwriting the file if it already exists "a"/"a+": Open for reading and writing, creating a new file if none exists, but preserving the existing file if one is present

jldsave(filename; kwargs...)
jldsave(filename, compress; kwargs...)
jldsave(filename, compress, iotype; kwargs...)

Creates a JLD2 file at filename and stores the variables given as keyword arguments.

Examples

jldsave("example.jld2"; a=1, b=2, c)

is equivalent to

jldopen("example.jld2, "w") do f
    f["a"] = 1
    f["b"] = 2
    f["c"] = c
end

To choose the io type IOStream instead of the default MmapIO use jldsave(fn, IOStream; kwargs...).

link_size(name::String)

Returns the size of a link message, including message header.

links_size(pairs)

Returns the size of several link messages. pairs is an iterator of String => RelOffset pairs.

load_attributes(f::JLDFile, name::AbstractString)
load_attributes(g::Group, name::AbstractString)
load_attributes(f::JLDFile, offset::RelOffset)

Return a list of attributes attached to the dataset or group.

load_data_or_dict(g::Union{JLDFile,Group}, varname::AbstractString)

Return the value of key varname but if it represents a Group load the group as a nested dictionary.

load_datatypes(f::JLDFile)

Populate f.datatypes and f.jlh5types with all of the committed datatypes from a file. We need to do this before writing to make sure we reuse written datatypes.

load_object(filename)

Returns the only available object from the JLD2 file filename (The stored object name is inconsequential). If the file contains more than one or no objects, the function throws an ArgumentError.

For loading more than one object, use @load macro, jldopen or the FileIO API.

Example

To load the only object from the JLD2 file example.jld2:

hello = "world"
save_object("example.jld2", hello)
hello_loaded = load_object("example.jld2")

loadnesteddict(g::Union{JLDFile, Group})

Return a dictionary with all data contained in group or file. Nested groups are loaded as nested dictionaries.

lookup_offset(g::Group, name::AbstractString) -> RelOffset

Lookup the offset of a dataset in a group. Returns UNDEFINED_ADDRESS if the dataset is not present. Does not inspect unwritten_child_groups.

pathize(g::Group, name::AbstractString, create::Bool) -> Tuple{Group,String}

Converts a path to a group and name object. If create is true, any intermediate groups will be created, and the dataset name will be checked for uniqueness with existing names.

prewrite(f::JLDFile)

Check that a JLD file is actually writable, and throw an error if not. Sets the written flag on the file.

printtoc([io::IO,] f::JLDFile [; numlines])

Prints an overview of the contents of f to the IO.

Use the optional numlines parameter to restrict the amount of items listed.

read_array!(v::Array, f::JLDFile, rr)

Fill the array v with the contents of JLDFile f at the current position, assuming a ReadRepresentation rr.

read_attr_data(f::JLDFile, attr::ReadAttribute, expected_datatype::H5Datatype,
               rr::ReadRepresentation)

jlread data from an attribute, assuming a specific HDF5 datatype and ReadRepresentation. If the HDF5 datatype does not match, throws an UnsupportedFeatureException. This allows better type stability while simultaneously validating the data.

read_attr_data(f::JLDFile, attr::ReadAttribute)

jlread data from an attribute.

read_attribute(io::IO, f::JLDFile)

Read an attribute message at the current position of the io object. Supports attribute message version 1 and 2.

read_compressed_array!(v::Array, f::JLDFile, rr, data_length::Int, ::Val{filter_id})

Fill the array v with the compressed contents of JLDFile f at the current position, assuming a ReadRepresentation rr and that the compressed data has length data_length.

read_data(f::JLDFile, dataspace::ReadDataspace, datatype_class::UInt8,
          datatype_offset::Int64, data_offset::Int64[, filters::FilterPipeline,
          header_offset::RelOffset, attributes::Vector{ReadAttribute}])

Read data from a file. If datatype_class is typemax(UInt8), the datatype is assumed to be committed, and datatype_offset points to the offset of the committed datatype's header. Otherwise, datatype_offset points to the offset of the datatype attribute.

read_scalar(f::JLDFile, rr, header_offset::RelOffset)

Read raw data representing a scalar with read representation rr from the current position of JLDFile f. header_offset is the RelOffset of the object header, used to resolve cycles.

read_size(io::IO, flags::UInt8)

Loads a variable-length size according to flags Expects that the first two bits of flags mean: 0 The size of the Length of Link Name field is 1 byte. 1 The size of the Length of Link Name field is 2 bytes. 2 The size of the Length of Link Name field is 4 bytes. 3 The size of the Length of Link Name field is 8 bytes. Returns the size as an Int

readas(::Type)::Type

Experimental feature: JLD2.readas can be overloaded to override which type a saved type is read as, and is used together with custom serialization using JLD2.writeas.

The typical case is custom serialization of parametric types, where not all type parameters are available during reading. Consider the following example for an anonymous function fun inside a Foo

struct Foo{F<:Function}
    fun::F
end
struct FooSerialization
    fun
end
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)

JLD2.readas(::Type{<:FooSerialization}) = Foo
struct UndefinedFunction <:Function
    fun
end
(f::UndefinedFunction)(args...; kwargs...) = error("The function $(f.fun) is not defined")
function Base.convert(::Type{<:Foo}, f::FooSerialization)
    isa(f.fun, Function) && return Foo(f.fun)
    return Foo(UndefinedFunction(f.fun))
end

If we include these definitions, call jldsave("foo.jld2"; foo=Foo(x->x^2)), restart julia, include the definitions again, and call foo = jldopen("foo.jld2") do io; io["foo"]; end, we get foo::Foo{UndefinedFunction} and foo::FooSerialization with and without defining the JLD2.readas above, respectively.

save_group(g::Group) -> RelOffset

Stores a group to a file, updating it if it has already been saved. Returns UNDEFINED_ADDRESS if the group was already stored, or the offset of the new group otherwise.

save_object(filename, x)

Stores an object x in a new JLD2 file at filename. If a file exists at this path, it will be overwritten.

Since the JLD2 format requires that all objects have a name, the object will be stored as single_stored_object. If you want to store more than one object, use @save macro, jldopen or the FileIO API.

Example

To save the string hello to the JLD2 file example.jld2:

hello = "world"
save_object("example.jld2", hello)

shorttypestring(::Type{ <:UnknownType})

Convert an UnknownType to a corresponding string. This is only used to create names for reconstructed types. See also typestring.

skip_to_aligned!(io, rel=0)

Skip to nearest position aligned to a multiple of 8 bytes relative to rel.

symbol_length(x::Symbol)

Returns the length of the string represented by x.

typestring(::Type{ <:UnknownType})

Convert an UnknownType to a corresponding string. This is only used for warning during reconstruction errors. See also shorttypestring.

write_link(cio, name, offset)

Write a link message at current position in cio.

@load filename var1 [var2 ...]

Load one or more variables var1,... from JLD2 file filename into the current scope and return a vector of the loaded variable names.

For interactive use, the form @load "somefile.jld2" will load all variables from "somefile.jld2" into the current scope. This form only supports literal file names and should be avoided in more permanent code so that it's clear where the variables come from.

Example

To load the variables hello and foo from the file example.jld2, use

@load "example.jld2" hello foo

@save filename var1 [var2 ...]
@save filename {compress=true} var1 name2=var2

Write one or more variables var1,... from the current scope to a JLD2 file filename.

For interactive use you can save all variables in the current module's global scope using @save filename. More permanent code should prefer the explicit form to avoid saving unwanted variables.

Example

To save the string hello and array xs to the JLD2 file example.jld2:

hello = "world"
xs = [1,2,3]
@save "example.jld2" hello xs

For passing options to the saving command use {}

@save "example.jld2" {compress=true} hello xs

For saving variables under a different name use regular assignment syntax

@save "example.jld2" greeting=hello xarray = xs