ProtoBuf.jl Documentation

Overview

ProtoBuf.jl provides a compiler and codec for Google's Protocol Buffers serialization format.

Use the protojl function to translate your .proto files to Julia, then you can encode and decode your messages with

encode(e::ProtoEncoder, x::YourMessage)
decode(d::ProtoDecoder, ::Type{YourMessage})

Where the ProtoEncoder and ProtoDecoder are simple types wrapping your IO.

Installation

The package is not currently registered, to install it, use:

import Pkg; Pkg.add("ProtoBuf")

Quickstart

Given a test.proto file in your current working directory:

syntax = "proto3";

enum MyEnum {
    DEFAULT = 0;
    OTHER = 1;
}

message MyMessage {
    sint32 a = 1;
    repeated string b = 2;
}

You can generate Julia bindings with the protojl function:

julia> using ProtoBuf

julia> protojl("test.proto", ".", "output_dir")

This will create a Julia file at output_dir/test_pb.jl which you can simply include and start using it to encode and decode messages:

julia> include("output_dir/test_pb.jl")
Main.test_pb

julia> io = IOBuffer();

julia> e = ProtoEncoder(io);

julia> encode(e, test_pb.MyMessage(-1, ["a", "b"]))
8

julia> seekstart(io);

julia> d = ProtoDecoder(io);

julia> decode(d, test_pb.MyMessage)
Main.test_pb.MyMessage(-1, ["a", "b"])

If you are curious, this is what the generated file looks like:

# Autogenerated using ProtoBuf.jl v0.1.0 on 2022-07-25T11:32:05.368
# original file: /Users/tdrvostep/_proj/ProtoBuf.jl/test.proto (proto3 syntax)

module test_pb

import ProtoBuf as PB
using ProtoBuf: OneOf
using EnumX: @enumx

export MyEnum, MyMessage

@enumx MyEnum DEFAULT=0 OTHER=1

struct MyMessage
    a::Int32
    b::Vector{String}
end
PB.default_values(::Type{MyMessage}) = (;a = zero(Int32), b = Vector{String}())
PB.field_numbers(::Type{MyMessage}) = (;a = 1, b = 2)

function PB.decode(d::PB.AbstractProtoDecoder, ::Type{<:MyMessage})
    a = zero(Int32)
    b = PB.BufferedVector{String}()
    while !PB.message_done(d)
        field_number, wire_type = PB.decode_tag(d)
        if field_number == 1
            a = PB.decode(d, Int32, Val{:zigzag})
        elseif field_number == 2
            PB.decode!(d, b)
        else
            PB.skip(d, wire_type)
        end
    end
    return MyMessage(a, b[])
end

function PB.encode(e::PB.AbstractProtoEncoder, x::MyMessage)
    initpos = position(e.io)
    x.a != zero(Int32) && PB.encode(e, 1, x.a, Val{:zigzag})
    !isempty(x.b) && PB.encode(e, 2, x.b)
    return position(e.io) - initpos
end
function PB._encoded_size(x::MyMessage)
    encoded_size = 0
    x.a != zero(Int32) && (encoded_size += PB._encoded_size(x.a, 1, Val{:zigzag}))
    !isempty(x.b) && (encoded_size += PB._encoded_size(x.b, 2))
    return encoded_size
end
end # module

Migrating from earlier versions of ProtoBuf.jl

Below is a list of notable differences that were introduced in 1.0.

Method Names

For translating proto files, use protojl function (previously protoc). To decode proto messages, use the decode method (previously readproto) and to encode Julia structs, use encode (previously writeproto). See Quickstart for an example.

Mutability

Messages are now translated to immutable structs. This means that code that used the mutable structs to accumulate data will now have to prepare each field and construct the struct after all of them are ready.

Common abstract type

By default, the generated structs don't share any common abstract type (well, except Any), when you set the common_abstract_type option to true, every struct definition will be a subtype of ProtoBuf.AbstractProtoBufMessage.

Naming Conventions

The naming of nested messages and enums now generates names, that cannot collide with other definitions. For example:

message A {
    message B {}
}

now generates structs named A and var"A.B". In protobuf, it is legal to a define message called A_B but not A.B, which is a syntax to refer to these nested definitions.

Similarly, field names that coincide with Julia reserved keywords were previously prefixed with an underscore (e.g. _function), now we prefix them with a # (e.g. var"#function").

Enumerations

EnumX.jl is used to define enums instead of NamedTuples. This means that to get the type of the enum, one must use MyEnum.T as MyEnum is a Module. You can now dispatch on Base.Enum when working with @enumx-based enums.

oneof Fields

oneof fields are now explicitly represented in the generated struct. Specifically, for a message

message MyMessage {
    oneof oneof_field {
        int32 option1 = 1;
        string option2 = 2;
    }
}

a struct with a single field will be generated:

struct MyMessage
   oneof_field::Union{Nothing,OneOf{<:Union{Int32,String}}}
end

Once instantiated, it will contain a value like OneOf{:option1, 42} or OneOf{:option2, "The answer to life, the universe, and everything"}. One can access the name and the value of the field via the :name and :value properties, respectively. Dereferencing the field will return the value of the field (e.g. my_message.oneof_field[] == 42)

Packages and Code Structure

For .proto files that are packages, a nested directory structure will be generated. For example, {file_name}.proto containing package foo_bar.baz_grok, the following directory structure is created:

root  # `output_directory` arg from from `protojl`
└── foo_bar
    ├── foo_bar.jl  # defines module `foo_bar`, imports `baz_grok`
    └── baz_grok
        ├── {file_name}_pb.jl
        └── baz_grok.jl  # defines module `baz_grok`, includes `{file_name}_pb.jl`

You should include the top-level module of a generated package, i.e. foo_bar.jl in this example. When reading .proto files that use import statements, the imported files have to be located at the respective import paths relative to search_directories.

.proto files that don't have a package specifier will generate a single file containing a module. For example, {file_name}.proto will translate to a {file_name}_pb.jl file defining a {file_name}_pb module. You can generate a file without a module by providing always_use_modules=false options to protojl.

Constructors

By default, there are no additional constructors generated for the Julia structs. You can use the add_kwarg_contructors=true option to protojl to create outer constructors that accept keyword arguments and provide default values where available.