TranscodingStreams.jl

Overview

TranscodingStreams.jl is a package for transcoding (e.g. compression) data streams. It exports a type TranscodingStream, which is a subtype of IO and supports various I/O operations like other usual I/O streams in the standard library. Operations are quick, simple, and consistent.

In this page, we intorduce the basic concepts of TranscodingStreams.jl and available packages. The Examples page demonstrates common usage. The References page offers a comprehensive API document.

Introduction

TranscodingStream has two type parameters, C<:Codec and S<:IO, and hence the actual type should be written as TranscodingStream{C<:Codec,S<:IO}. This type wraps an underlying I/O stream S by a codec C. The codec defines transformation (or transcoding) of the stream. For example, when C is a lossless decompressor type and S is a file, TranscodingStream{C,S} behaves like a data stream that incrementally decompresses data from the file.

Codecs are defined in other packages listed below:

Package	Library	Format	Codec	Stream	Description
CodecZlib.jl	zlib	RFC1952	`GzipCompressor`	`GzipCompressorStream`	Compress data in gzip (.gz) format.
		RFC1952	`GzipDecompressor`	`GzipDecompressorStream`	Decompress data in gzip (.gz) format.
		RFC1950	`ZlibCompressor`	`ZlibCompressorStream`	Compress data in zlib format.
		RFC1950	`ZlibDecompressor`	`ZlibDecompressorStream`	Decompress data in zlib format.
		RFC1951	`DeflateCompressor`	`DeflateCompressorStream`	Compress data in deflate format.
		RFC1951	`DeflateDecompressor`	`DeflateDecompressorStream`	Decompress data in deflate format.
CodecBzip2.jl	bzip2		`Bzip2Compressor`	`Bzip2CompressorStream`	Compress data in bzip2 (.bz2) format.
CodecBzip2.jl	bzip2		`Bzip2Decompressor`	`Bzip2DecompressorStream`	Decompress data in bzip2 (.bz2) format.
CodecXz.jl	xz	The .xz File Format	`XzCompressor`	`XzCompressorStream`	Compress data in xz (.xz) format.
CodecXz.jl	xz	The .xz File Format	`XzDecompressor`	`XzDecompressorStream`	Decompress data in xz (.xz) format.
CodecZstd.jl	zstd	Zstandard Compressor Format	`ZstdCompressor`	`ZstdCompressorStream`	Compress data in zstd (.zst) format.
CodecZstd.jl	zstd	Zstandard Compressor Format	`ZstdDecompressor`	`ZstdDecompressorStream`	Decompress data in zstd (.zst) format.
CodecBase.jl	native	RFC4648	`Base16Encoder`	`Base16EncoderStream`	Encode binary in base16 format.
			`Base16Decoder`	`Base16DecoderStream`	Decode binary in base16 format.
			`Base32Encoder`	`Base32EncoderStream`	Encode binary in base32 format.
			`Base32Decoder`	`Base32DecoderStream`	Decode binary in base32 format.
			`Base64Encoder`	`Base64EncoderStream`	Encode binary in base64 format.
			`Base64Decoder`	`Base64DecoderStream`	Decode binary in base64 format.

Install packages you need by calling Pkg.add(<package name>) in a Julia session. For example, if you want to read gzip-compressed files, call Pkg.add("CodecZlib") to use GzipDecompressor or GzipDecompressorStream. By convention, codec types have a name that matches .*(Co|Deco)mpression and I/O types have a codec name with Stream suffix. All codecs are a subtype TranscodingStreams.Codec and streams are a subtype of Base.IO. An important thing is these packages depend on TranscodingStreams.jl and not vice versa. This means you can install any codec package you need without installing all codec packages. Also, if you want to define your own codec, it is totally feasible like these packages. TranscodingStreams.jl requests a codec to implement some interface functions which will be described later.

Error handling

You may encounter an error while processing data with this package. For example, your compressed data may be corrupted or truncated and the decompressor codec cannot handle it properly. In this case, the codec informs the stream of the error and the stream goes to an unrecoverable mode. In this mode, the only possible operations are isopen and close. Other operations, such as read or write, will result in an argument error exception. Resources allocated in the codec will be released by the stream and hence you must not call the finalizer of a codec that is once passed to a transcoding stream object.