Video Reading

Note: Reading of audio streams is not yet implemented

Reading Video Files

VideoIO contains a simple high-level interface which allows reading of video frames from a supported video file (or from a camera device, shown later).

The simplest form will load the entire video into memory as a vector of image arrays.

using VideoIO
VideoIO.load("video.mp4")

VideoIO.load — Function

load(filename::String, args...; kwargs...)

Load video file filename into memory as vector of image arrays, setting args and kwargs on the openvideo process.

source

Frames can be read sequentially until the end of the file:

using VideoIO

# Construct a AVInput object to access the video and audio streams in a video container
# io = VideoIO.open(video_file)
io = VideoIO.testvideo("annie_oakley") # for testing purposes

# Access the video stream in an AVInput, and return a VideoReader object:
f = VideoIO.openvideo(io) # you can also use a file name, instead of a AVInput

img = read(f)

while !eof(f)
    read!(f, img)
    # Do something with frames
end
close(f)

VideoIO.openvideo — Function

openvideo(file[, video_stream = 1]; <keyword arguments>) -> reader
openvideo(f, ...)

Open file and create an object to read and decode video stream number video_stream. file can either be a AVInput created by VideoIO.open, the name of a file as an AbstractString, or instead an IO object. However, support for IO objects is incomplete, and does not currently work with common video containers such as *.mp4 files.

Frames can be read from the reader with read or read!, or alternatively by using the iterator interface provided for reader. To close the reader, simply use close. Seeking within the reader can be accomplished using seek, seekstart. Frames can be skipped with skipframe, or skipframes. The current time in the video stream can be accessed with gettime. Details about the frame dimension can be found with out_frame_size. The total number of frames can be found with counttotalframes.

If called with a single argument function as the first argument, the reader will be passed to the function, and will be closed once the call returns whether or not an error occurred.

The decoder options and conversion to Julia arrays is controlled by the keyword arguments listed below.

Keyword arguments

transcode::Bool = true: Determines whether decoded frames are transferred into a Julia matrix with easily interpretable element type, or instead returned as raw byte buffers.
target_format::Union{Nothing, Cint} = nothing: Determines the target pixel format that decoded frames will be transformed into before being transferred to an output array. This can either by a VideoIO.AV_PIX_FMT_* value corresponding to a FFmpeg AVPixelFormat, and must then also be a format supported by the VideoIO, or instead nothing, in which case the format will be automatically chosen by FFmpeg. This list of currently supported pixel formats, and the matrix element type that each pixel format corresponds with, are elements of VideoIO.VIO_PIX_FMT_DEF_ELTYPE_LU.
pix_fmt_loss_flags = 0: Loss flags to control how transfer pixel format is chosen. Only valid if target_format = nothing. Flags must correspond to FFmpeg loss flags.
target_colorspace_details = nothing: Information about the color space of output Julia arrays. If nothing, then this will correspond to a best-effort interpretation of Colors.jl for the corresponding element type. To override these defaults, create a VideoIO.VioColorspaceDetails object using the appropriate AVCOL_ definitions from FFmpeg, or use VideoIO.VioColorspaceDetails() to use the FFmpeg defaults. To avoid rescaling limited color range data (mpeg) to full color range output (jpeg), then set this to VideoIO.VioColorspaceDetails() to avoid additional scaling by sws_scale.
allow_vio_gray_transform = true: Instead of using sws_scale for gray data, use a more accurate color space transformation implemented in VideoIO if allow_vio_gray_gransform = true. Otherwise, use sws_scale.
swscale_options::OptionsT = (;): A Namedtuple, or Dict{Symbol, Any} of options for the swscale object used to perform color space scaling. Options must correspond with options for FFmpeg's scaler filter.
sws_color_options::OptionsT = (;): Additional keyword arguments passed to sws_setColorspaceDetails.
thread_count::Union{Nothing, Int} = Sys.CPU_THREADS: The number of threads the codec is allowed to use or nothing for default codec behavior. Defaults to Sys.CPU_THREADS.

source

Alternatively, you can open the video stream in a file directly with VideoIO.openvideo(filename), without making an intermediate AVInput object, if you only need the video.

VideoIO also provides an iterator interface for VideoReader, which behaves like other mutable iterators in Julia (e.g. Channels). If iteration is stopped early, for example with a break statement, then it can be resumed in the same spot by iterating on the same VideoReader object. Consequently, if you have already iterated over all the frames of a VideoReader object, then it will be empty for further iteration unless its position in the video is changed with seek.

using VideoIO

f = VideoIO.openvideo("video.mp4")
for img in f
    # Do something with img
end
# Alternatively use collect(f) to get all of the frames

# Further iteration will show that f is now empty
@assert isempty(f)

close(f)

Seeking through the video can be achieved via seek(f, seconds::Float64) and seekstart(f) to return to the start.

Base.seek — Function

seek(reader::VideoReader, seconds)

Seeks into the parent AVInput using this video stream's index. See [seek] for AVInput.

source

seek(avin::AVInput, seconds::AbstractFloat, video_stream::Integer=1)

Seek through the container format avin so that the next frame returned by the stream indicated by video_stream will have a timestamp greater than or equal to seconds.

source

Base.seekstart — Function

seekstart(reader::VideoReader)

Seek to time zero of the parent AVInput using reader's stream index. See seekstart for AVInput objects.

source

seekstart(avin::AVInput{T}, video_stream_index=1) where T <: AbstractString

Seek to time zero of AVInput object.

source

Frames can be skipped without reading frame content via skipframe(f) and skipframes(f, n)

VideoIO.skipframe — Function

skipframe(s::VideoReader; throwEOF=true)

Skip the next frame. If End of File is reached, EOFError thrown if throwEOF=true. Otherwise returns true if EOF reached, false otherwise.

source

VideoIO.skipframes — Function

skipframes(s::VideoReader, n::Int; throwEOF=true) -> n

Skip the next n frames. If End of File is reached and throwEOF=true, a EOFError will be thrown. Returns the number of frames that were skipped.

source

Total available frame count is available via counttotalframes(f)

VideoIO.counttotalframes — Function

counttotalframes(reader) -> n::Int

Count the total number of frames in the video by seeking to start, skipping through each frame, and seeking back to the start.

For a faster alternative that relies on video container metadata, try get_number_frames.

source

!!! note H264 videos encoded with crf>0 have been observed to have 4-fewer frames available for reading.

Changing the target pixel format for reading

It can be helpful to be explicit in which pixel format you wish to read frames as. Here a grayscale video is read and parsed into a Vector(Array{UInt8}}

f = VideoIO.openvideo(filename, target_format=VideoIO.AV_PIX_FMT_GRAY8)

while !eof(f)
    img = reinterpret(UInt8, read(f))
end
close(f)

Reading Camera Output

Frames can be read iteratively

using VideoIO
cam = VideoIO.opencamera()
fps = VideoIO.framerate(cam)
for i in 1:100
    img = read(cam)
    sleep(1/fps)
end

To change settings such as the frame rate or resolution of the captured frames, set the appropriate value in the options positional argument.

julia> opts = VideoIO.DEFAULT_CAMERA_OPTIONS
VideoIO.AVDict with 2 entries:
  "framerate"    => "30"
  "pixel_format" => "uyvy422"

julia> opts["framerate"] = "24"
"24"

julia> opts["video_size"] = "640x480"
"640x480"

julia> opencamera(VideoIO.DEFAULT_CAMERA_DEVICE[], VideoIO.DEFAULT_CAMERA_FORMAT[], opts)
VideoReader(...)

Or more simply, change the default. For example:

julia> VideoIO.DEFAULT_CAMERA_OPTIONS["video_size"] = "640x480"

julia> VideoIO.DEFAULT_CAMERA_OPTIONS["framerate"] = 30

julia> julia> opencamera()
VideoReader(...)

Video Properties & Metadata

VideoIO.get_start_time — Function

get_start_time(file::String) -> DateTime

Return the starting date & time of the video file. Note that if the starting date & time are missing, this function will return the Unix epoch (00:00 1st January 1970).

source

VideoIO.get_time_duration — Function

get_time_duration(file::String) -> (DateTime, Microsecond)

Return the starting date & time as well as the duration of the video file. Note that if the starting date & time are missing, this function will return the Unix epoch (00:00 1st January 1970).

source

VideoIO.get_duration — Function

get_duration(file::String) -> Float64

Return the duration of the video file in seconds (float).

source

VideoIO.get_number_frames — Function

get_number_frames(file [, streamno])

Query the the container file for the number of frames in video stream streamno if applicable, instead returning nothing if the container does not report the number of frames. Will not decode the video to count the number of frames in a video.

source