Internals & Design
File Interface
The JLDFile object mimics the API of Base.Dict
as much as it can. In particular, keys
, length
, haskey
, isempty
, get
, get!
should work as expected.
JLD2.CommittedDatatype
— TypeCommittedDatatype <: H5Datatype
Reference to a shared datatype message (stored elsewhere in a file). These are stored in the _types
group and indexed.
JLD2.CustomSerialization
— TypeCustomSerialization{T,S}
On-disk representation for data that is written as if it were of Julia type T
, but is read as type S
.
JLD2.GlobalHeap
— TypeGlobalHeap
Represents an HDF5 global heap structure.
JLD2.Group
— TypeGroup{T}
Group(file::T)
JLD2 group object.
Advanced Usage
Takes two optional keyword arguments:
est_num_entries::Int
= 4est_link_name_len::Int
= 8
These determine how much (additional) empty space should be allocated for the group description. (list of entries) This can be useful for performance when one expects to append many additional datasets after first writing the file.
JLD2.Group
— MethodGroup(f::JLDFile, name::AbstractString)
Construct an empty group named name
at the top level of JLDFile
f
.
JLD2.Group
— MethodGroup(g::Group, name::AbstractString)
Construct a group named name
as a child of group g
.
JLD2.H5Datatype
— Typeabstract type H5Datatype
Supertype of all HDF5 datatypes.
JLD2.HeaderMessage
— TypeHeaderMessage
Helper struct to read and write the first part of a header message.
JLD2.HeaderMessageIterator
— Typemutable struct HeaderMessageIterator{IO}
HeaderMessageIterator(f::JLDFile, offset::RelOffset)
Implements an iterator over header messages.
JLD2.Hmessage
— TypeHmessage{IO}
Representation of a Header Message in memory. Provides getproperty
access to the fields of the message. Can also be used to construct and write custom messages.
JLD2.IndirectPointer
— TypeIndirectPointer
When writing data, we may need to enlarge the memory mapping, which would invalidate any memory addresses arising from the old mmap
pointer. IndirectPointer
holds an offset relative to the MemoryBackedIO. It defers computing a memory address until converted to a Ptr{T}
, so the memory mapping can be enlarged and addresses will remain valid.
JLD2.InlineUnionEl
— TypeInlineUnionEl{T1,T2}(mask::UInt8, t1::T1, t2::T2)
Custom serialization struct for two member isbits union fields e.g. in other structs or arrays. To indicate that t1
is relevant the mask
takes the value UInt8(0)
and for t2
the mask
takes the value UInt8(255)
.
JLD2.JLDFile
— TypeJLDFile{T<:IO}
JLD file object.
JLD2.JLDWriteSession
— TypeJLDWriteSession{T}
A JLDWriteSession
keeps track of references to serialized objects. If T
is a Dict, h5offset
maps an object ID (returned by calling objectid
) to th RelOffset
of the written dataset. If it is Union{}
, then references are not tracked, and objects referenced multiple times are written multiple times.
JLD2.MemoryBackedIO
— TypeMemoryBackedIO <: IO
Abstract type for IO objects that are backed by memory in such a way that one can use pointer based unsafe_load
and unsafe_store!
operations after ensuring that there is enough memory allocated.
It needs to provide:
getproperty(io, :curptr)
to get the current pointerensureroom(io, nb)
to ensure that there are at least nb bytes availableposition(io)
to get the current (zero-based) positionseek(io, pos)
to set the current position (zero-based)
JLD2.Message
— TypeMessage{IO}
Representation of a Message in memory. Provides getproperty
access
JLD2.ReadRepresentation
— TypeReadRepresentation{T,ODR}
A type encoding both the Julia type T
and the on-disk (HDF5) representation ODR
.
JLD2.RelOffset
— TypeRelOffset
Represents an HDF5 relative offset. This differs from a file offset (used elsewhere) in that it is relative to the superblock base address. fileoffset
and h5offset
convert between RelOffset
s and file offsets.
JLD2.SharedDatatype
— TypeSharedDatatype <: H5Datatype
Reference to a shared datatype message (stored elsewhere in a file).
JLD2.Upgrade
— TypeUpgrade(T)
Specify an upgrade path for serialized structs using the typemap
keyword argument and rconvert
.
JLD2.attributes
— Methodattributes(dset::Dataset; plain::Bool=false)
Return the attributes of a dataset as an OrderedDict
. If plain
is set to true
then the values are returned as stored in the dataset object.
JLD2.behead
— Methodbehead(T)
Given a UnionAll
type, recursively eliminates the where
clauses
JLD2.bufferpos
— Methodbufferpos(io::Union{BufferedReader, BufferedWriter})
Get the current position in the buffer.
JLD2.construct_array
— Methodconstruct_array(io::IO, eltype, ndims::Int)
Construct array by reading ndims
dimensions from io
. Assumes io
has already been seeked to the correct position.
JLD2.constructrr
— Functionconstructrr(f::JLDFile, T::DataType, dt::CompoundType, attrs::Vector{ReadAttribute},
hard_failure::Bool=false)
Constructs a ReadRepresentation
for a given type. This is the generic method for all types not specially handled below.
If hard_failure
is true, then throw a TypeMappingException
instead of attempting reconstruction. This helps in cases where we can't know if reconstructed parametric types will have a matching memory layout without first inspecting the memory layout.
JLD2.create_dataset
— Methodcreate_dataset(parent, path, datatype, dataspace; kwargs...)
Arguments: - parent::Union{JLDfile, Group}
: Containing group of new dataset - path
: Path to new dataset relative to parent
. If path
is nothing
, the dataset is unnamed. - datatype
: Datatype of new dataset (element type in case of arrays) - dataspace
: Dimensions or Dataspace
of new dataset
Keyword arguments: - layout
: DataLayout
of new dataset - filters
: FilterPipeline
for describing the compression pipeline
JLD2.fileoffset
— Methodfileoffset(f::JLDFile, x::RelOffset)
Converts an offset x
relative to the superblock of file f
to an absolute offset.
JLD2.flag2uint
— Methodflag2uint(flag::UInt8)
I Map the lowest to bits of flag
to a UInt
type, mapping 0 to UInt8
, 1 to UInt16
, 2 to UInt32
, and 3 to UInt64
.
JLD2.get_dataset
— Methodget_dataset(parent::Union{JLDFile, Group}, name::String)
Get a stored dataset from a file by name or path as a Dataset
object. This may be useful for inspecting the metadata incl. types of a dataset.
JLD2.h5offset
— Methodh5offset(f::JLDFile, x::Integer)
Converts an absolute file offset x
to an offset relative to the superblock of file f
.
JLD2.ismmappable
— Methodismmappable(dset::Dataset)
Check if a dataset can be memory-mapped. This can be useful for large arrays and for editing written arrays.
An Array dataset may be mmapped if: - JLD2.samelayout(T) == true
: The element type is isbits
and has a size that is a multiple of 8 bytes. - Uncompressed: Compressed arrays cannot be memory-mapped - Uses a contiguous layout: This is true for all array datasets written by JLD2 with version ≥ v0.4.52 - Offset in file is a multiple of 8 bytes: This is a requirement for Mmap. - Windows: The file must be opened in read-only mode. This is a limitation of Mmap on Windows.
JLD2.isset
— Methodisset(flag, bit)
Return true if the bit-th bit of flag
is set. (starting from 0)
JLD2.jld_finalizer
— Methodjld_finalizer(f::JLDFile)
When a JLDFile is finalized, it is possible that the MmapIO
has been munmapped, since Julia does not guarantee finalizer order. This means that the underlying file may be closed before we get a chance to write to it.
JLD2.jldopen
— Functionjldopen(file, mode::AbstractString; iotype=MmapIO, compress=false, typemap=Dict())
Opens a JLD2 file at path file
. Alternatively file
may be a suitable IO object.
Options for mode
:
"r"
: Open for reading only, failing if no file exists"r+"
: Open for reading and writing, failing if no file exists"w"
/"w+"
: Open for reading and writing, overwriting the file if it already exists"a"
/"a+"
: Open for reading and writing, creating a new file if none exists, but preserving the existing file if one is present
JLD2.jldsave
— Functionjldsave(filename; kwargs...)
jldsave(filename, compress; kwargs...)
jldsave(filename, compress, iotype; kwargs...)
Creates a JLD2 file at filename
and stores the variables given as keyword arguments.
Examples
jldsave("example.jld2"; a=1, b=2, c)
is equivalent to
jldopen("example.jld2", "w") do f
f["a"] = 1
f["b"] = 2
f["c"] = c
end
To choose the io type IOStream
instead of the default MmapIO
use jldsave(fn, IOStream; kwargs...)
.
JLD2.jlwrite
— Methodjlwrite(io::IO, x::Tuple)
Attempt to write a tuple to io
by writing each element of the tuple in order.
JLD2.links_size
— Methodlinks_size(pairs)
Returns the size of several link messages. pairs
is an iterator of String => RelOffset
pairs.
JLD2.load_attributes
— Functionload_attributes(f::JLDFile, name::AbstractString)
load_attributes(g::Group, name::AbstractString)
load_attributes(g::Group)
load_attributes(f::JLDFile, offset::RelOffset)
Return a list of attributes attached to the dataset or group.
JLD2.load_data_or_dict
— Methodload_data_or_dict(g::Union{JLDFile,Group}, varname::AbstractString)
Return the value of key varname
but if it represents a Group
load the group as a nested dictionary.
JLD2.load_datatypes
— Methodload_datatypes(f::JLDFile)
Populate f.datatypes
and f.jlh5types
with all of the committed datatypes from a file. We need to do this before writing to make sure we reuse written datatypes.
JLD2.load_object
— Methodload_object(filename)
Returns the only available object from the JLD2 file filename
(The stored object name is inconsequential). If the file contains more than one or no objects, the function throws an ArgumentError
.
For loading more than one object, use @load
macro, jldopen
or the FileIO API.
Example
To load the only object from the JLD2 file example.jld2:
hello = "world"
save_object("example.jld2", hello)
hello_loaded = load_object("example.jld2")
JLD2.loadnesteddict
— Methodloadnesteddict(g::Union{JLDFile, Group})
Return a dictionary with all data contained in group or file. Nested groups are loaded as nested dictionaries.
JLD2.lookup_offset
— Methodlookup_offset(g::Group, name::AbstractString) -> RelOffset
Lookup the offset of a dataset in a group. Returns UNDEFINED_ADDRESS
if the dataset is not present. Does not inspect unwritten_child_groups
.
JLD2.pathize
— Methodpathize(g::Group, name::AbstractString, create::Bool) -> Tuple{Group,String}
Converts a path to a group and name object. If create
is true, any intermediate groups will be created, and the dataset name will be checked for uniqueness with existing names.
JLD2.prewrite
— Methodprewrite(f::JLDFile)
Check that a JLD file is actually writable, and throw an error if not. Sets the written
flag on the file.
JLD2.print_header_messages
— Methodprint_header_messages(f::JLDFile, name::AbstractString)
print_header_messages(g::Group, name::AbstractString)
print_header_messages(f::JLDFile, offset::RelOffset)
Prints the header messages of a group or dataset in a file.
JLD2.printtoc
— Methodprinttoc([io::IO,] f::JLDFile [; numlines])
Prints an overview of the contents of f
to the IO
.
Use the optional numlines
parameter to restrict the amount of items listed.
JLD2.read_array!
— Functionread_array!(v::Array, f::JLDFile, rr)
Fill the array v
with the contents of JLDFile f
at the current position, assuming a ReadRepresentation
rr
.
JLD2.read_attr_data
— Methodread_attr_data(f::JLDFile, attr::ReadAttribute, expected_datatype::H5Datatype,
rr::ReadRepresentation)
jlread
data from an attribute, assuming a specific HDF5 datatype and ReadRepresentation
. If the HDF5 datatype does not match, throws an UnsupportedFeatureException
. This allows better type stability while simultaneously validating the data.
JLD2.read_attr_data
— Methodread_attr_data(f::JLDFile, attr::ReadAttribute)
jlread
data from an attribute.
JLD2.read_compressed_array!
— Functionread_compressed_array!(v::Array, f::JLDFile, rr, data_length::Int, Val(filter_id))
Fill the array v
with the compressed contents of JLDFile f
at the current position, assuming a ReadRepresentation
rr
and that the compressed data has length data_length
.
JLD2.read_data
— Functionread_data(f::JLDFile, dataspace::ReadDataspace, datatype_class::UInt8,
datatype_offset::Int64, data_offset::Int64[, filters::FilterPipeline,
header_offset::RelOffset, attributes::Vector{ReadAttribute}])
Read data from a file. If datatype_class
is typemax(UInt8)
, the datatype is assumed to be committed, and datatype_offset
points to the offset of the committed datatype's header. Otherwise, datatype_offset
points to the offset of the datatype attribute.
JLD2.read_dataset
— Methodread_dataset(dset::Dataset)
Read the data referenced by a dataset.
JLD2.read_scalar
— Functionread_scalar(f::JLDFile, rr, header_offset::RelOffset)
Read raw data representing a scalar with read representation rr
from the current position of JLDFile f
. header_offset
is the RelOffset
of the object header, used to resolve cycles.
JLD2.read_size
— Methodread_size(io::IO, flags::UInt8)
Loads a variable-length size according to flags
Expects that the first two bits of flags mean:
- 0: The size of the Length of Link Name field is 1 byte.
- 1: The size of the Length of Link Name field is 2 bytes.
- 2: The size of the Length of Link Name field is 4 bytes.
- 3: The size of the Length of Link Name field is 8 bytes.
Returns the size as an Int
.
JLD2.readas
— Methodreadas(::Type)::Type
Experimental feature: JLD2.readas
can be overloaded to override which type a saved type is read as, and is used together with custom serialization using JLD2.writeas
.
The typical case is custom serialization of parametric types, where not all type parameters are available during reading. Consider the following example for an anonymous function fun
inside a Foo
struct Foo{F<:Function}
fun::F
end
struct FooSerialization
fun
end
JLD2.writeas(::Type{<:Foo}) = FooSerialization
Base.convert(::Type{<:FooSerialization}, f::Foo) = FooSerialization(f.fun)
JLD2.readas(::Type{<:FooSerialization}) = Foo
struct UndefinedFunction <:Function
fun
end
(f::UndefinedFunction)(args...; kwargs...) = error("The function $(f.fun) is not defined")
function Base.convert(::Type{<:Foo}, f::FooSerialization)
isa(f.fun, Function) && return Foo(f.fun)
return Foo(UndefinedFunction(f.fun))
end
If we include these definitions, call jldsave("foo.jld2"; foo=Foo(x->x^2))
, restart julia, include the definitions again, and call foo = jldopen("foo.jld2") do io; io["foo"]; end
, we get foo::Foo{UndefinedFunction}
and foo::FooSerialization
with and without defining the JLD2.readas
above, respectively.
JLD2.readmmap
— Methodreadmmap(dset::Dataset)
Memory-map a dataset. This can be useful for large arrays and for editing written arrays. See ismmappable
for requirements.
JLD2.save_group
— Methodsave_group(g::Group) -> RelOffset
Stores a group to a file, updating it if it has already been saved. Returns UNDEFINED_ADDRESS
if the group was already stored, or the offset of the new group otherwise.
JLD2.save_object
— Methodsave_object(filename, x)
Stores an object x
in a new JLD2 file at filename
. If a file exists at this path, it will be overwritten.
Since the JLD2 format requires that all objects have a name, the object will be stored as single_stored_object
. If you want to store more than one object, use @save
macro, jldopen
or the FileIO API.
Example
To save the string hello
to the JLD2 file example.jld2:
hello = "world"
save_object("example.jld2", hello)
JLD2.shorttypestring
— Methodshorttypestring(::Type{ <:UnknownType})
Convert an UnknownType
to a corresponding string. This is only used to create names for reconstructed types.
See also typestring
.
JLD2.size_flag
— Methodsize_flag(sz::Integer)::UInt8
Return the flag that represents the smallest integer type that can represent sz
. 0 -> UInt8, 1 -> UInt16, 2 -> UInt32, 3 -> UInt64
JLD2.size_size
— Methodsize_size(sz::Integer)
Return the number of bytes required to represent sz
as an unsigned integer that actually exists. (e.g. UInt8, UInt16, UInt32, UInt64)
JLD2.size_size2
— Methodsize_size2(sz::Integer)
Return the number of bytes required to represent sz
as an unsigned integer. Note: this does not check if the integer is a valid julia integer.
JLD2.skip_to_aligned!
— Functionskip_to_aligned!(io, rel=0)
Skip to nearest position aligned to a multiple of 8 bytes relative to rel
.
JLD2.symbol_length
— Methodsymbol_length(x::Symbol)
Returns the length of the string represented by x
.
JLD2.typestring
— Methodtypestring(::Type{ <:UnknownType})
Convert an UnknownType
to a corresponding string. This is only used for warning during reconstruction errors.
See also shorttypestring
.
JLD2.uintofsize
— Methoduintofsize(sz::Integer)
Return the UInt
type that has sz
bytes.
JLD2.write_dataset
— Functionwrite_dataset(dataset::Dataset, data)
Write data to file using metadata prepared in the dataset
.
JLD2.write_size
— Methodwrite_size(io::IO, sz::Integer)
Write the mininum number of bytes required to represent sz
as (valid) unsigned integer.
JLD2.write_zerobytes
— Methodwrite_zerobytes(io, n)
Write n
zero bytes to io
.
JLD2.@load
— Macro@load filename var1 [var2 ...]
Load one or more variables var1,...
from JLD2 file filename
into the current scope and return a vector of the loaded variable names.
For interactive use, the form @load "somefile.jld2"
will load all variables from "somefile.jld2"
into the current scope. This form only supports literal file names and should be avoided in more permanent code so that it's clear where the variables come from.
Example
To load the variables hello
and foo
from the file example.jld2, use
@load "example.jld2" hello foo
JLD2.@pseudostruct
— Macro@pseudostruct name begin ... end
The @pseudostruct
macro is used to define constructor, size computation, show, and and optimized getproperty function for Message
s. The allowed syntax elements are:
@skip(n)
: Markn
bytes as empty.
JLD2.@save
— Macro@save filename var1 [var2 ...]
@save filename {compress=true} var1 name2=var2
Write one or more variables var1,...
from the current scope to a JLD2 file filename
.
For interactive use you can save all variables in the current module's global scope using @save filename
. More permanent code should prefer the explicit form to avoid saving unwanted variables.
Example
To save the string hello
and array xs
to the JLD2 file example.jld2:
hello = "world"
xs = [1,2,3]
@save "example.jld2" hello xs
For passing options to the saving command use {}
@save "example.jld2" {compress=true} hello xs
For saving variables under a different name use regular assignment syntax
@save "example.jld2" greeting=hello xarray = xs