Internals & Design

File Interface

The JLDFile object mimics the API of Base.Dict as much as it can. In particular, keys, length, haskey, isempty, get, get! should work as expected.

JLD2.CommittedDatatypeType
CommittedDatatype <: H5Datatype

Reference to a shared datatype message (stored elsewhere in a file). These are stored in the _types group and indexed.

source
JLD2.CustomSerializationType
CustomSerialization{T,S}

On-disk representation for data that is written as if it were of Julia type T, but is read as type S.

source
JLD2.GroupType
Group{T}
Group(file::T)

JLD2 group object.

Advanced Usage

Takes two optional keyword arguments:

  • est_num_entries::Int = 4
  • est_link_name_len::Int = 8

These determine how much (additional) empty space should be allocated for the group description. (list of entries) This can be useful for performance when one expects to append many additional datasets after first writing the file.

source
JLD2.GroupMethod
Group(f::JLDFile, name::AbstractString)

Construct an empty group named name at the top level of JLDFile f.

source
JLD2.GroupMethod
Group(g::Group, name::AbstractString)

Construct a group named name as a child of group g.

source
JLD2.HeaderMessageIteratorType
mutable struct HeaderMessageIterator{IO}
    HeaderMessageIterator(f::JLDFile, offset::RelOffset)

Implements an iterator over header messages.

source
JLD2.HmessageType
Hmessage{IO}

Representation of a Header Message in memory. Provides getproperty access to the fields of the message. Can also be used to construct and write custom messages.

source
JLD2.IndirectPointerType
IndirectPointer

When writing data, we may need to enlarge the memory mapping, which would invalidate any memory addresses arising from the old mmap pointer. IndirectPointer holds an offset relative to the MemoryBackedIO. It defers computing a memory address until converted to a Ptr{T}, so the memory mapping can be enlarged and addresses will remain valid.

source
JLD2.InlineUnionElType
InlineUnionEl{T1,T2}(mask::UInt8, t1::T1, t2::T2)

Custom serialization struct for two member isbits union fields e.g. in other structs or arrays. To indicate that t1 is relevant the mask takes the value UInt8(0) and for t2 the mask takes the value UInt8(255).

source
JLD2.JLDWriteSessionType
JLDWriteSession{T}

A JLDWriteSession keeps track of references to serialized objects. If T is a Dict, h5offset maps an object ID (returned by calling objectid) to th RelOffset of the written dataset. If it is Union{}, then references are not tracked, and objects referenced multiple times are written multiple times.

source
JLD2.MemoryBackedIOType
MemoryBackedIO <: IO

Abstract type for IO objects that are backed by memory in such a way that one can use pointer based unsafe_load and unsafe_store! operations after ensuring that there is enough memory allocated.

It needs to provide:

  • getproperty(io, :curptr) to get the current pointer
  • ensureroom(io, nb) to ensure that there are at least nb bytes available
  • position(io) to get the current (zero-based) position
  • seek(io, pos) to set the current position (zero-based)
source
JLD2.MessageType
Message{IO}

Representation of a Message in memory. Provides getproperty access

source
JLD2.RelOffsetType
RelOffset

Represents an HDF5 relative offset. This differs from a file offset (used elsewhere) in that it is relative to the superblock base address. fileoffset and h5offset convert between RelOffsets and file offsets.

source
JLD2.SharedDatatypeType
SharedDatatype <: H5Datatype

Reference to a shared datatype message (stored elsewhere in a file).

source
JLD2.UpgradeType
Upgrade(T)

Specify an upgrade path for serialized structs using the typemap keyword argument and rconvert.

source
JLD2.attributesMethod
attributes(dset::Dataset; plain::Bool=false)

Return the attributes of a dataset as an OrderedDict. If plain is set to true then the values are returned as stored in the dataset object.

source
JLD2.beheadMethod
behead(T)

Given a UnionAll type, recursively eliminates the where clauses

source
JLD2.bufferposMethod
bufferpos(io::Union{BufferedReader, BufferedWriter})

Get the current position in the buffer.

source
JLD2.construct_arrayMethod
construct_array(io::IO, eltype, ndims::Int)

Construct array by reading ndims dimensions from io. Assumes io has already been seeked to the correct position.

source
JLD2.constructrrFunction
constructrr(f::JLDFile, T::DataType, dt::CompoundType, attrs::Vector{ReadAttribute},
            hard_failure::Bool=false)

Constructs a ReadRepresentation for a given type. This is the generic method for all types not specially handled below.

If hard_failure is true, then throw a TypeMappingException instead of attempting reconstruction. This helps in cases where we can't know if reconstructed parametric types will have a matching memory layout without first inspecting the memory layout.

source
JLD2.create_datasetMethod
create_dataset(parent, path, datatype, dataspace; kwargs...)

Arguments: - parent::Union{JLDfile, Group}: Containing group of new dataset - path: Path to new dataset relative to parent. If path is nothing, the dataset is unnamed. - datatype: Datatype of new dataset (element type in case of arrays) - dataspace: Dimensions or Dataspace of new dataset

Keyword arguments: - layout: DataLayout of new dataset - filters: FilterPipeline for describing the compression pipeline

source
JLD2.fileoffsetMethod
fileoffset(f::JLDFile, x::RelOffset)

Converts an offset x relative to the superblock of file f to an absolute offset.

source
JLD2.flag2uintMethod
flag2uint(flag::UInt8)

I Map the lowest to bits of flag to a UInt type, mapping 0 to UInt8, 1 to UInt16, 2 to UInt32, and 3 to UInt64.

source
JLD2.get_datasetMethod
get_dataset(parent::Union{JLDFile, Group}, name::String)

Get a stored dataset from a file by name or path as a Dataset object. This may be useful for inspecting the metadata incl. types of a dataset.

source
JLD2.h5offsetMethod
h5offset(f::JLDFile, x::Integer)

Converts an absolute file offset x to an offset relative to the superblock of file f.

source
JLD2.ismmappableMethod
ismmappable(dset::Dataset)

Check if a dataset can be memory-mapped. This can be useful for large arrays and for editing written arrays.

An Array dataset may be mmapped if: - JLD2.samelayout(T) == true: The element type is isbits and has a size that is a multiple of 8 bytes. - Uncompressed: Compressed arrays cannot be memory-mapped - Uses a contiguous layout: This is true for all array datasets written by JLD2 with version ≥ v0.4.52 - Offset in file is a multiple of 8 bytes: This is a requirement for Mmap. - Windows: The file must be opened in read-only mode. This is a limitation of Mmap on Windows.

source
JLD2.issetMethod
isset(flag, bit)

Return true if the bit-th bit of flag is set. (starting from 0)

source
JLD2.jld_finalizerMethod
jld_finalizer(f::JLDFile)

When a JLDFile is finalized, it is possible that the MmapIO has been munmapped, since Julia does not guarantee finalizer order. This means that the underlying file may be closed before we get a chance to write to it.

source
JLD2.jldopenFunction
jldopen(file, mode::AbstractString; iotype=MmapIO, compress=false, typemap=Dict())

Opens a JLD2 file at path file. Alternatively file may be a suitable IO object.

Options for mode:

  • "r": Open for reading only, failing if no file exists
  • "r+": Open for reading and writing, failing if no file exists
  • "w"/"w+": Open for reading and writing, overwriting the file if it already exists
  • "a"/"a+": Open for reading and writing, creating a new file if none exists, but preserving the existing file if one is present
source
JLD2.jldsaveFunction
jldsave(filename; kwargs...)
jldsave(filename, compress; kwargs...)
jldsave(filename, compress, iotype; kwargs...)

Creates a JLD2 file at filename and stores the variables given as keyword arguments.

Examples

jldsave("example.jld2"; a=1, b=2, c)

is equivalent to

jldopen("example.jld2", "w") do f
    f["a"] = 1
    f["b"] = 2
    f["c"] = c
end

To choose the io type IOStream instead of the default MmapIO use jldsave(fn, IOStream; kwargs...).

source
JLD2.jlwriteMethod
jlwrite(io::IO, x::Tuple)

Attempt to write a tuple to io by writing each element of the tuple in order.

source
JLD2.links_sizeMethod
links_size(pairs)

Returns the size of several link messages. pairs is an iterator of String => RelOffset pairs.

source
JLD2.load_attributesFunction
load_attributes(f::JLDFile, name::AbstractString)
load_attributes(g::Group, name::AbstractString)
load_attributes(g::Group)
load_attributes(f::JLDFile, offset::RelOffset)

Return a list of attributes attached to the dataset or group.

source
JLD2.load_data_or_dictMethod
load_data_or_dict(g::Union{JLDFile,Group}, varname::AbstractString)

Return the value of key varname but if it represents a Group load the group as a nested dictionary.

source
JLD2.load_datatypesMethod
load_datatypes(f::JLDFile)

Populate f.datatypes and f.jlh5types with all of the committed datatypes from a file. We need to do this before writing to make sure we reuse written datatypes.

source
JLD2.load_objectMethod
load_object(filename)

Returns the only available object from the JLD2 file filename (The stored object name is inconsequential). If the file contains more than one or no objects, the function throws an ArgumentError.

For loading more than one object, use @load macro, jldopen or the FileIO API.

Example

To load the only object from the JLD2 file example.jld2:

hello = "world"
save_object("example.jld2", hello)
hello_loaded = load_object("example.jld2")
source
JLD2.loadnesteddictMethod
loadnesteddict(g::Union{JLDFile, Group})

Return a dictionary with all data contained in group or file. Nested groups are loaded as nested dictionaries.

source
JLD2.lookup_offsetMethod
lookup_offset(g::Group, name::AbstractString) -> RelOffset

Lookup the offset of a dataset in a group. Returns UNDEFINED_ADDRESS if the dataset is not present. Does not inspect unwritten_child_groups.

source
JLD2.pathizeMethod
pathize(g::Group, name::AbstractString, create::Bool) -> Tuple{Group,String}

Converts a path to a group and name object. If create is true, any intermediate groups will be created, and the dataset name will be checked for uniqueness with existing names.

source
JLD2.prewriteMethod
prewrite(f::JLDFile)

Check that a JLD file is actually writable, and throw an error if not. Sets the written flag on the file.

source
JLD2.print_header_messagesMethod
print_header_messages(f::JLDFile, name::AbstractString)
print_header_messages(g::Group, name::AbstractString)
print_header_messages(f::JLDFile, offset::RelOffset)

Prints the header messages of a group or dataset in a file.

source
JLD2.printtocMethod
printtoc([io::IO,] f::JLDFile [; numlines])

Prints an overview of the contents of f to the IO.

Use the optional numlines parameter to restrict the amount of items listed.

source
JLD2.read_attr_dataMethod
read_attr_data(f::JLDFile, attr::ReadAttribute, expected_datatype::H5Datatype,
               rr::ReadRepresentation)

jlread data from an attribute, assuming a specific HDF5 datatype and ReadRepresentation. If the HDF5 datatype does not match, throws an UnsupportedFeatureException. This allows better type stability while simultaneously validating the data.

source
JLD2.read_compressed_array!Function
read_compressed_array!(v::Array, f::JLDFile, rr, data_length::Int, Val(filter_id))

Fill the array v with the compressed contents of JLDFile f at the current position, assuming a ReadRepresentation rr and that the compressed data has length data_length.

source
JLD2.read_dataFunction
read_data(f::JLDFile, dataspace::ReadDataspace, datatype_class::UInt8,
          datatype_offset::Int64, data_offset::Int64[, filters::FilterPipeline,
          header_offset::RelOffset, attributes::Vector{ReadAttribute}])

Read data from a file. If datatype_class is typemax(UInt8), the datatype is assumed to be committed, and datatype_offset points to the offset of the committed datatype's header. Otherwise, datatype_offset points to the offset of the datatype attribute.

source
JLD2.read_scalarFunction
read_scalar(f::JLDFile, rr, header_offset::RelOffset)

Read raw data representing a scalar with read representation rr from the current position of JLDFile f. header_offset is the RelOffset of the object header, used to resolve cycles.

source
JLD2.read_sizeMethod
read_size(io::IO, flags::UInt8)

Loads a variable-length size according to flags

Expects that the first two bits of flags mean:

  • 0: The size of the Length of Link Name field is 1 byte.
  • 1: The size of the Length of Link Name field is 2 bytes.
  • 2: The size of the Length of Link Name field is 4 bytes.
  • 3: The size of the Length of Link Name field is 8 bytes.

Returns the size as an Int.

source
JLD2.readasMethod
readas(::Type)::Type

Experimental feature: JLD2.readas can be overloaded to override which type a saved type is read as, and is used together with custom serialization using JLD2.writeas.

The typical case is custom serialization of parametric types, where not all type parameters are available during reading. Consider the following example for an anonymous function fun inside a Foo

struct Foo{F<:Function}
    fun::F
end
struct FooSerialization
    fun
end
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)

JLD2.readas(::Type{<:FooSerialization}) = Foo
struct UndefinedFunction <:Function
    fun
end
(f::UndefinedFunction)(args...; kwargs...) = error("The function $(f.fun) is not defined")
function Base.convert(::Type{<:Foo}, f::FooSerialization)
    isa(f.fun, Function) && return Foo(f.fun)
    return Foo(UndefinedFunction(f.fun))
end

If we include these definitions, call jldsave("foo.jld2"; foo=Foo(x->x^2)), restart julia, include the definitions again, and call foo = jldopen("foo.jld2") do io; io["foo"]; end, we get foo::Foo{UndefinedFunction} and foo::FooSerialization with and without defining the JLD2.readas above, respectively.

source
JLD2.readmmapMethod
readmmap(dset::Dataset)

Memory-map a dataset. This can be useful for large arrays and for editing written arrays. See ismmappable for requirements.

source
JLD2.save_groupMethod
save_group(g::Group) -> RelOffset

Stores a group to a file, updating it if it has already been saved. Returns UNDEFINED_ADDRESS if the group was already stored, or the offset of the new group otherwise.

source
JLD2.save_objectMethod
save_object(filename, x)

Stores an object x in a new JLD2 file at filename. If a file exists at this path, it will be overwritten.

Since the JLD2 format requires that all objects have a name, the object will be stored as single_stored_object. If you want to store more than one object, use @save macro, jldopen or the FileIO API.

Example

To save the string hello to the JLD2 file example.jld2:

hello = "world"
save_object("example.jld2", hello)
source
JLD2.shorttypestringMethod
shorttypestring(::Type{ <:UnknownType})

Convert an UnknownType to a corresponding string. This is only used to create names for reconstructed types.

See also typestring.

source
JLD2.size_flagMethod
size_flag(sz::Integer)::UInt8

Return the flag that represents the smallest integer type that can represent sz. 0 -> UInt8, 1 -> UInt16, 2 -> UInt32, 3 -> UInt64

source
JLD2.size_sizeMethod
size_size(sz::Integer)

Return the number of bytes required to represent sz as an unsigned integer that actually exists. (e.g. UInt8, UInt16, UInt32, UInt64)

source
JLD2.size_size2Method
size_size2(sz::Integer)

Return the number of bytes required to represent sz as an unsigned integer. Note: this does not check if the integer is a valid julia integer.

source
JLD2.skip_to_aligned!Function
skip_to_aligned!(io, rel=0)

Skip to nearest position aligned to a multiple of 8 bytes relative to rel.

source
JLD2.typestringMethod
typestring(::Type{ <:UnknownType})

Convert an UnknownType to a corresponding string. This is only used for warning during reconstruction errors.

See also shorttypestring.

source
JLD2.write_datasetFunction
write_dataset(dataset::Dataset, data)

Write data to file using metadata prepared in the dataset.

source
JLD2.write_sizeMethod
write_size(io::IO, sz::Integer)

Write the mininum number of bytes required to represent sz as (valid) unsigned integer.

source
JLD2.@loadMacro
@load filename var1 [var2 ...]

Load one or more variables var1,... from JLD2 file filename into the current scope and return a vector of the loaded variable names.

For interactive use, the form @load "somefile.jld2" will load all variables from "somefile.jld2" into the current scope. This form only supports literal file names and should be avoided in more permanent code so that it's clear where the variables come from.

Example

To load the variables hello and foo from the file example.jld2, use

@load "example.jld2" hello foo
source
JLD2.@pseudostructMacro
@pseudostruct name begin ... end

The @pseudostruct macro is used to define constructor, size computation, show, and and optimized getproperty function for Messages. The allowed syntax elements are:

  • @skip(n): Mark n bytes as empty.
source
JLD2.@saveMacro
@save filename var1 [var2 ...]
@save filename {compress=true} var1 name2=var2

Write one or more variables var1,... from the current scope to a JLD2 file filename.

For interactive use you can save all variables in the current module's global scope using @save filename. More permanent code should prefer the explicit form to avoid saving unwanted variables.

Example

To save the string hello and array xs to the JLD2 file example.jld2:

hello = "world"
xs = [1,2,3]
@save "example.jld2" hello xs

For passing options to the saving command use {}

@save "example.jld2" {compress=true} hello xs

For saving variables under a different name use regular assignment syntax

@save "example.jld2" greeting=hello xarray = xs
source