CodecLz4

LZ4FastCompressor(; kwargs...)

Creates an LZ4 compression codec.

Keywords

• acceleration::Integer=0: acceleration factor
• block_size::Integer=1024: The size in bytes to encrypt into each block.

LZ4FastCompressor(stream::IO; kwargs...)

Creates an LZ4 compression stream. See LZ4FastCompressor() and TranscodingStream() for arguments.

LZ4FrameCompressor(; kwargs...)

Creates an LZ4 compression codec.

Keywords

• blocksizeid::BlockSizeID=default_size: max64KB, max256KB, max1MB, or max4MB or default_size
• blockmode::BlockMode=block_linked: block_linked or block_independent
• contentchecksum::Bool=false: if true, frame is terminated with a 32-bits checksum of decompressed data
• frametype::FrameType=normal_frame): normal_frame or skippable_frame
• contentsize::Integer=0: Size of uncompressed content (0 for unknown)
• blockchecksum::Bool=false: if true, each block is followed by a checksum of block's compressed data
• compressionlevel::Integer=0: compression level (-1..12)
• autoflush::Bool=false: always flush if true

LZ4FrameCompressorStream(stream::IO; kwargs...)

Creates an LZ4 compression stream. See LZ4FrameCompressor() and TranscodingStream() for arguments.

LZ4FrameCompressor()

Creates an LZ4 decompression codec.

LZ4FrameCompressorStream(stream::IO; kwargs...)

Creates an LZ4 decompression stream. See TranscodingStream() for arguments.

LZ4HCCompressor(; kwargs...)

Creates an LZ4 HC compression codec.

Keywords

• compressionlevel::Integer=9: compression level
• block_size::Integer=1024: The size in bytes to encrypt into each block. (Max 4MB)

LZ4HCCompressorStream(stream::IO; kwargs...)

Creates an LZ4 HC compression stream. See LZ4HCCompressorStream() and TranscodingStream() for arguments.

LZ4SafeDecompressor(; kwargs...)

Creates an LZ4 compression codec.

Keywords

• block_size::Integer=1024: The size in bytes of unecrypted data contained in each block. Must match or exceed the compression block_size or decompression will fail.

LZ4SafeDecompressorStream(stream::IO; kwargs...)

Creates an LZ4 compression stream. See LZ4SafeDecompressor() and TranscodingStream() for arguments.

Will write the frame header into dstBuffer.

dstCapacity must be >= LZ4F_HEADER_SIZE_MAX bytes. prefsPtr is optional : you can provide C_NULL as argument, all preferences will then be set to default. Returns the number of bytes written into dstBuffer for the header or throws an error.

Provides minimum dstCapacity for a given srcSize to guarantee operation success in worst case situations.

prefsPtr is optional : when C_NULL is provided, preferences will be set to cover worst case scenario. Result is always the same for a srcSize and prefsPtr, so it can be trusted to size reusable buffers. When srcSize==0, LZ4F_compressBound() provides an upper bound for LZ4F_flush() and LZ4F_compressEnd() operations.

Invoke to properly finish an LZ4 frame.

It will flush whatever data remained within cctx (like LZ4_flush()) and properly finalize the frame, with an endMark and a checksum. cOptPtr is optional : C_NULL can be provided, in which case all options will be set to default. Returns the number of bytes written into dstBuffer (necessarily >= 4 (endMark), or 8 if optional frame checksum is enabled) or throws an error if it fails (which can be tested using LZ4F_isError()) A successful call to LZ4F_compressEnd() makes cctx available again for another compression task.

Can be called repetitively to compress as much data as necessary.

An important rule is that dstCapacity MUST be large enough to ensure operation success even in worst case situations. This value is provided by LZ4F_compressBound(). If this condition is not respected, LZ4F_compress() will fail. LZ4FcompressUpdate() doesn't guarantee error recovery. When an error occurs, compression context must be freed or resized. cOptPtr is optional : `CNULLcan be provided, in which case all options are set to default. Returns the number of bytes written intodstBuffer` (it can be zero, meaning input data was just buffered). or throws an error if it fails.

The first thing to do is to create a compressionContext object, which will be used in all compression operations. This is achieved using LZ4F_createCompressionContext(), which takes as argument a version. The version provided MUST be the current version. It is intended to track potential version mismatch, notably when using DLL. The function will provide a pointer to a fully allocated LZ4F_cctx object. Will throw an error if there was an error during context creation.

Create an LZ4F_dctx object, to track all decompression operations.

The version provided MUST be the current LZ4F version. The function provides a pointer to an allocated and initialized LZ4F_dctx object. The the function throws an error if the LZ4F_dctx object cannot be initialized. The dctx memory can be released using LZ4F_freeDecompressionContext().

Call this function repetitively to regenerate compressed data from srcBuffer.

The function will read up to srcSizePtr bytes from srcBuffer, and decompress data into dstBuffer, of capacity dstSizePtr. The number of bytes consumed from srcBuffer will be written into srcSizePtr (necessarily <= original value). The number of bytes decompressed into dstBuffer will be written into dstSizePtr (necessarily <= original value). The function does not necessarily read all input bytes, so always check value in srcSizePtr. Unconsumed source data must be presented again in subsequent invocations.

dstBuffer can freely change between each consecutive function invocation. dstBuffer content will be overwritten.

Returns an hint of how many srcSize bytes LZ4F_decompress() expects for next call. Schematically, it's the size of the current (or remaining) compressed block + header of next block. Respecting the hint provides some small speed benefit, because it skips intermediate buffers. This is just a hint though, it's always possible to provide any srcSize.

When a frame is fully decoded, returns 0 (no more data expected). When provided with more bytes than necessary to decode a frame, LZ4F_decompress() will stop reading exactly at end of current frame, and return 0.

If decompression failed, an error is thrown. After a decompression error, the dctx context is not resumable. Use LZ4F_resetDecompressionContext() to return to clean state. After a frame is fully decoded, dctx can be used again to decompress another frame.

When data must be generated and sent immediately, without waiting for a block to be completely filled, it's possible to call LZ4_flush(). It will immediately compress any data buffered within cctx. dstCapacity must be large enough to ensure the operation will be successful. cOptPtr is optional : it's possible to provide C_NULL, all options will be set to default. Returns the number of bytes written into dstBuffer (it can be zero, which means there was no data stored within cctx) or throws an error if it fails.

Releases the memory of a LZ4F_cctx.

Frees the decompressionContext.

The result of LZ4F_freeDecompressionContext() is indicative of the current state of decompressionContext when being released. That is, it should be == 0 if decompression has been completed fully and correctly.

Extracts frame parameters (max blockSize, dictID, etc.).

Its usage is optional. Extracted information is typically useful for allocation and dictionary. This function works in 2 situations :

• At the beginning of a new frame, in which case it will decode information from srcBuffer, starting the decoding process. Input size must be large enough to successfully decode the entire frame header. Frame header size is variable, but is guaranteed to be <= LZ4F_HEADER_SIZE_MAX bytes. It's allowed to provide more input data than this minimum.
• After decoding has been started. In which case, no input is read, frame parameters are extracted from dctx.
• If decoding has barely started, but not yet extracted information from header, LZ4F_getFrameInfo() will fail.

The number of bytes consumed from srcBuffer will be updated within srcSizePtr (necessarily <= original value). Decompression must resume from (srcBuffer + srcSizePtr). Returns an hint about how many srcSize bytes LZ4F_decompress() expects for next call or throws an error.

Note 1 : In case of error, dctx is not modified. Decoding operation can resume from beginning safely. Note 2 : Frame parameters are copied into an already allocated LZ4F_frameInfo_t structure.

Gets the current LZ4F version.

Re-initializes decompression context

In case of an error, the context is left in "undefined" state. In which case, it's necessary to reset it, before re-using it. This method can also be used to abruptly stop any unfinished decompression, and start a new one using same context resources.

LZ4_compressBound(inputsize)

Provides the maximum size that LZ4 compression may output in a "worst case" scenario (input data not compressible) This function is primarily useful for memory allocation purposes (destination buffer size). Macro LZ4COMPRESSBOUND() is also provided for compilation-time evaluation (stack memory allocation for example). Note that LZ4compressdefault() compress faster when dest buffer size is >= LZ4compressBound(srcsize) inputsize : max supported value is LZ4MAXINPUT_SIZE

Returns maximum output size in a "worst case" scenario or 0, if input size is too large ( > LZ4MAXINPUT_SIZE)

LZ4_compress_HC(src, dst, srcsize, dstcapacity, compressionlevel)

Compress data from src into dst, using the more powerful but slower "HC" algorithm. dst must be already allocated. Compression is guaranteed to succeed if dstcapacity >= LZ4_compressBound(srcsize) Max supported srcsize value is LZ4MAXINPUTSIZE compressionlevel: any value between 1 and LZ4HCCLEVELMAX will work. Values > LZ4HCCLEVELMAX behave the same as LZ4HCCLEVEL_MAX. Returns the number of bytes written into dst

LZ4_compress_HC_continue(streamptr::Ptr{LZ4_streamHC_t}, src, dst, srcsize, maxdstsize)

Compress data in successive blocks of any size, using previous blocks as dictionary. One key assumption is that previous blocks (up to 64 KB) remain read-accessible while compressing next blocks. There is an exception for ring buffers, which can be smaller than 64 KB. Ring buffers scenario is automatically detected and handled by LZ4compressHC_continue().

Before starting compression, state must be properly initialized, using LZ4_resetStreamHC().

Then, use LZ4compressHCcontinue() to compress each successive block. Previous memory blocks (including initial dictionary when present) must remain accessible and unmodified during compression. dst buffer should be sized to handle worst case scenarios (see LZ4compressBound()), to ensure operation success. Because in case of failure, the API does not guarantee context recovery, and context will have to be reset.

LZ4_compress_destSize(src, dst, srcsize, dstcapacity)

Reverse the logic : compresses as much data as possible from src buffer into already allocated buffer dst of size dstcapacity. This function either compresses the entire src content into dst if it's large enough, or fill dst buffer completely with as much data as possible from src. *srcsize : will be modified to indicate how many bytes where read from src to fill dst. New value is necessarily <= old value.

Returns number of bytes written into dst (necessarily <= dstcapacity)

LZ4_compress_fast(src, dst, srcsize, dstcapacity=1)

Compresses srcsize bytes from buffer src into already allocated dst buffer of size dstcapacity. Compression is guaranteed to succeed if dstcapacity >= LZ4_compressBound(srcsize). It also runs faster, so it's a recommended setting. If the function cannot compress src into a limited dst budget, compression stops immediately, and the function result is zero. As a consequence, dst content is not valid.

Also allows to select an acceleration factor. The larger the acceleration value, the faster the algorithm, but also the lesser the compression. It's a trade-off. It can be fine tuned, with each successive value providing roughly +~3% to speed. Values <= 0 will be replaced by the default which is 1.

This function never writes outside dst buffer, nor read outside source buffer. srcsize : supported max value is LZ4MAXINPUT_VALUE dstcapacity : full or partial size of buffer dst (which must be already allocated)

Returns the number of bytes written into buffer dst (necessarily <= dstcapacity)

LZ4_compress_fast_continue(streamptr, src, dst, srcsize, dstcapacity, acceleration)

Compress content into src using data from previously compressed blocks, improving compression ratio. dst buffer must be already allocated. If dstcapacity >= LZ4_compressBound(srcsize), compression is guaranteed to succeed, and runs faster.

Important : Up to 64KB of previously compressed data is assumed to remain present and unmodified in memory ! Special 1 : If input buffer is a double-buffer, it can have any size, including < 64 KB. Special 2 : If input buffer is a ring-buffer, it can have any size, including < 64 KB.

Returns size of compressed block. After an error, the stream status is invalid, it can only be reset or freed.

LZ4_createStream()

Will allocate and initialize an LZ4_stream_t structure.

LZ4_createStreamDecode()

These decoding functions work the same as creation / destruction of streaming decompression tracking structure. A tracking structure can be re-used multiple times sequentially.

LZ4_createStreamHC()

Create memory for LZ4 HC streaming state. Newly created states are automatically initialized. Existing states can be re-used several times, using LZ4_resetStreamHC().

LZ4_decompress_safe(src, dst, cmpsize, dstcapacity)

Decompresses cmpsize bytes from buffer src into dst. If destination buffer is not large enough, decoding will stop and output an error. If the source stream is detected malformed, the function will stop decoding and error. This function is protected against buffer overflow exploits, including malicious data packets. It never writes outside output buffer, nor reads outside input buffer.

cmpsize : is the exact complete size of the compressed block. dstcapacity : is the size of destination buffer, which must be already allocated.

Returns the number of bytes decompressed into destination buffer (necessarily <= dstcapacity)

LZ4_decompress_safe_continue(streamptr, src, dst, srcsize, dstcapacity)

These decoding functions allow decompression of consecutive blocks in "streaming" mode. A block is an unsplittable entity, it must be presented entirely to a decompression function. Decompression functions only accept one block at a time. Previously decoded blocks must remain available at the memory position where they were decoded (up to 64 KB).

Special : if application sets a ring buffer for decompression, it must respect one of the following conditions :

• Exactly same size as encoding buffer, with same update rule (block boundaries at same positions) In which case, the decoding & encoding ring buffer can have any size, including very small ones ( < 64 KB).
• Larger than encoding buffer, by a minimum of maxBlockSize more bytes. maxBlockSize is implementation dependent. It's the maximum size of any single block. In which case, encoding and decoding buffers do not need to be synchronized, and encoding ring buffer can have any size, including small ones ( < 64 KB).
• At least 64 KB + 8 bytes + maxBlockSize. In which case, encoding and decoding buffers do not need to be synchronized, and encoding ring buffer can have any size, including larger than decoding buffer.

Whenever these conditions are not possible, save the last 64KB of decoded data into a safe buffer, and indicate where it is saved using LZ4_setStreamDecode() before decompressing next block.

LZ4_freeStream(streamptr::Ptr{LZ4_stream_t})

Releases memory allocated by LZ4_createStream.

LZ4_freeStreamDecode(streamptr)

These decoding functions work the same as creation / destruction of streaming decompression tracking structure. A tracking structure can be re-used multiple times sequentially.

LZ4_freeStreamHC(streamptr)

Release memory for LZ4 HC streaming state. Existing states can be re-used several times, using LZ4_resetStreamHC().

LZ4_resetStream(streamptr)

An LZ4streamt structure can be allocated once and re-used multiple times. Use this function to start compressing a new stream.

LZ4_setStreamDecode(streamptr, dictionary, dictsize)

An LZ4streamDecodet structure can be allocated once and re-used multiple times. Use this function to start decompression of a new stream of blocks. A dictionary can optionally be set. Use NULL or size 0 for a simple reset order. Returns 1 if OK

lz4_compress(input::Union{Vector{UInt8},Base.CodeUnits{UInt8}}, acceleration::Integer=0)

Compresses input using LZ4compressfast. Returns a Vector{UInt8} of the compressed data.

lz4_decompress(input::Union{Vector{UInt8},Base.CodeUnits{UInt8}}, expected_size::Integer=input.size * 2)

Decompresses input using LZ4decompresssafe. expected_size must be equal to or larger than the expected decompressed size of the input or decompression will fail. Returns a Vector{UInt8} of the decompressed data.

lz4_hc_compress(input::Union{Vector{UInt8},Base.CodeUnits{UInt8}}, acceleration::Integer=9)

Compresses input using LZ4compressHC. Returns a Vector{UInt8} of the compressed data.

TranscodingStreams.expectedsize(codec::Union{LZ4FastCompressor, LZ4HCCompressor}, input::Memory)

Returns the expected size of the transcoded data.

TranscodingStreams.expectedsize(codec::LZ4FrameCompressor, input::Memory)

Returns the expected size of the transcoded data.

TranscodingStreams.expectedsize(codec::Union{LZ4FastCompressor, LZ4HCCompressor}, input::Memory)

Returns the expected size of the transcoded data.

TranscodingStreams.expectedsize(codec::LZ4SafeDecompressor, input::Memory)

Returns the expected size of the transcoded data.

TranscodingStreams.finalize(codec::LZ4FastCompressor)

Finalizes the LZ4 Compression Codec.

TranscodingStreams.finalize(codec::LZ4FrameCompressor)

Finalizes the LZ4F Compression Codec.

TranscodingStreams.finalize(codec::LZ4FrameDecompressor)

Finalizes the LZ4F Decompression Codec.

TranscodingStreams.finalize(codec::LZ4HCCompressor)

Finalizes the LZHC Compression Codec.

TranscodingStreams.finalize(codec::LZ4SafeDecompressor)

Finalizes the LZ4SafeDecompression Codec.

TranscodingStreams.initialize(codec::LZ4FastCompressor)

Initializes the LZ4 Compression Codec.

TranscodingStreams.initialize(codec::LZ4FrameCompressor)

Initializes the LZ4F Compression Codec.

TranscodingStreams.initialize(codec::LZ4FrameDecompressor)

Initializes the LZ4F Decompression Codec.

TranscodingStreams.initialize(codec::LZ4HCCompressor)

Initializes the LZ4 HC compression Codec.

TranscodingStreams.initialize(codec::LZ4SafeDecompressor)

Initializes the LZ4 Compression Codec.

TranscodingStreams.minoutsize(codec::Union{LZ4FastCompressor, LZ4HCCompressor}, input::Memory)

Returns the minimum output size of process.

TranscodingStreams.minoutsize(codec::LZ4FrameCompressor, input::Memory)

Returns the minimum output size of process.

TranscodingStreams.minoutsize(codec::Union{LZ4FastCompressor, LZ4HCCompressor}, input::Memory)

Returns the minimum output size of process.

TranscodingStreams.minoutsize(codec::LZ4SafeDecompressor, input::Memory)

Returns the minimum output size of process.

TranscodingStreams.process(codec::LZ4FastCompressor, input::Memory, output::Memory, error::Error)

Compresses the data from input and writes to output. The process follows the compression example from https://github.com/lz4/lz4/blob/dev/examples/blockStreaming_doubleBuffer.c wherein each block of encoded data is prefixed by the number of bytes contained in that block. Decoding can be done through the LZ4SafeDecompressor.

TranscodingStreams.process(codec::LZ4FrameCompressor, input::Memory, output::Memory, error::Error)

Compresses the data from input and writes to output. The LZ4 compression algorithm may simply buffer the input data a full frame can be produced, so data_written may be 0. flush() may be used to force output to be written.

TranscodingStreams.process(codec::LZ4FrameDecompressor, input::Memory, output::Memory, error::Error)

Decompresses the data from input and writes to output. If the input data is not properly formatted this function will throw an error.

TranscodingStreams.process(codec::LZ4HCCompressor, input::Memory, output::Memory, error::Error)

Compresses the data from input and writes to output. The process follows the compression example from https://github.com/lz4/lz4/blob/dev/examples/HCStreaming_ringBuffer.c wherein each block of encoded data is prefixed by the number of bytes contained in that block. Decoding can be done through the LZ4SafeDecompressor.

TranscodingStreams.process(codec::LZ4SafeDecompressor, input::Memory, output::Memory, error::Error)

Compresses the data from input and writes to output. The process follows the decompression example from https://github.com/lz4/lz4/blob/dev/examples/blockStreaming_doubleBuffer.c and requires each block of encoded data to be prefixed by the number of bytes contained in that block.

TranscodingStreams.startproc(codec::LZ4FastCompressor, mode::Symbol, error::Error)

Starts processing with the codec

TranscodingStreams.startproc(codec::LZ4FrameCompressor, mode::Symbol, error::Error)

Starts processing with the codec Creates the LZ4F header to be written to the output.

TranscodingStreams.startproc(codec::LZ4HCCompressor, mode::Symbol, error::Error)

Starts processing with the codec

TranscodingStreams.startproc(codec::LZ4SafeDecompressor, mode::Symbol, error::Error)

Starts processing with the codec