Skip to content

mattnite/protobuf

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

109 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

protobuf

Pure-Zig Protocol Buffers. Compile .proto files to Zig structs at build time with no dependency on protoc or any C library. Zero external dependencies.

This library was almost entirely generated by AI (Claude). Correctness is validated by 640+ unit tests, 24 AFL++ fuzz targets, Google's official protobuf conformance test suite, and Go interoperability tests for RPC.

Features

  • Proto2 and proto3 wire formats
  • Build-time code generation via build.zig step
  • JSON and text format serialization
  • Dynamic (descriptor-driven) messages without codegen
  • Transport-agnostic RPC stub generation (all four streaming modes)
  • Works as a protoc-gen-zig plugin
  • Google well-known types (Timestamp, Duration, Any, wrappers, etc.)
  • Unknown field preservation for forward/backward compatibility

Quick start

Add the dependency:

zig fetch --save git+https://codeberg.org/mattnite/protobuf

In your build.zig:

const protobuf = @import("protobuf");

pub fn build(b: *std.Build) void {
    const proto_dep = b.dependency("protobuf", .{});
    const proto_mod = protobuf.generate(b, proto_dep, .{
        .proto_sources = b.path("proto/"),
    });

    const exe = b.addExecutable(.{
        .name = "my-app",
        .root_module = b.createModule(.{
            .root_source_file = b.path("src/main.zig"),
            .imports = &.{
                .{ .name = "proto", .module = proto_mod },
            },
        }),
    });
    b.installArtifact(exe);
}

If your protos import from other directories:

const proto_mod = protobuf.generate(b, proto_dep, .{
    .proto_sources = b.path("proto/"),
    .import_paths = &.{
        b.path("third_party/"),
    },
});

Usage examples

Encode and decode a message

Given proto/example.proto:

syntax = "proto3";
package example;

message SearchRequest {
    string query = 1;
    int32 page_number = 2;
    int32 results_per_page = 3;
}
const std = @import("std");
const proto = @import("proto");
const SearchRequest = proto.example.SearchRequest;

pub fn main() !void {
    var gpa: std.heap.DebugAllocator(.{}) = .init;
    const allocator = gpa.allocator();

    const request = SearchRequest{
        .query = "protobuf zig",
        .page_number = 1,
        .results_per_page = 10,
    };

    // Encode
    var buf: [256]u8 = undefined;
    var writer: std.Io.Writer = .fixed(&buf);
    try request.encode(&writer);
    const bytes = writer.buffered();

    // Decode
    const decoded = try SearchRequest.decode(allocator, bytes);
    defer decoded.deinit(allocator);

    std.debug.assert(std.mem.eql(u8, decoded.query, "protobuf zig"));
    std.debug.assert(decoded.page_number == 1);
}

Nested messages and enums

syntax = "proto3";
package example;

enum Color { RED = 0; GREEN = 1; BLUE = 2; }

message Outer {
    message Inner { int32 value = 1; }
    Inner inner = 1;
    Color color = 2;
    repeated string tags = 3;
}
const Outer = proto.example.Outer;
const Color = proto.example.Color;

const msg = Outer{
    .inner = .{ .value = 42 },
    .color = .GREEN,
    .tags = &.{ "fast", "safe" },
};

Oneofs

message Event {
    oneof payload {
        string text = 1;
        int32 code = 2;
        bytes data = 3;
    }
}
const Event = proto.example.Event;

const text_event = Event{ .payload = .{ .text = "hello" } };
const code_event = Event{ .payload = .{ .code = 404 } };

switch (text_event.payload.?) {
    .text => |t| std.debug.print("text: {s}\n", .{t}),
    .code => |c| std.debug.print("code: {}\n", .{c}),
    .data => |d| std.debug.print("data: {} bytes\n", .{d.len}),
}

Maps

message Config {
    map<string, string> settings = 1;
    map<int32, string> labels = 2;
}
const Config = proto.example.Config;

// String-keyed maps use StringArrayHashMapUnmanaged
// Integer-keyed maps use AutoArrayHashMapUnmanaged
const decoded = try Config.decode(allocator, bytes);
defer decoded.deinit(allocator);

if (decoded.settings.get("key")) |value| {
    std.debug.print("setting: {s}\n", .{value});
}

RPC services

service RouteGuide {
    rpc GetFeature(Point) returns (Feature);
    rpc ListFeatures(Rectangle) returns (stream Feature);
    rpc RecordRoute(stream Point) returns (RouteSummary);
    rpc RouteChat(stream RouteNote) returns (stream RouteNote);
}

Generates a RouteGuide namespace containing:

  • service_descriptor — method metadata (names, paths, streaming flags)
  • Server — vtable interface to implement for handling RPCs
  • Client — stub that wraps a Channel transport with typed methods

All four streaming modes are supported: unary, server-streaming, client-streaming, and bidirectional. The transport (gRPC, Connect, in-process, etc.) is a separate concern that implements the Channel interface.

JSON and text format

Generated messages include JSON and text format support:

const msg = SearchRequest{ .query = "hello", .page_number = 1 };

// JSON
var buf: [4096]u8 = undefined;
var w: std.Io.Writer = .fixed(&buf);
try msg.to_json(&w);
// {"query":"hello","pageNumber":1}

// Parse from JSON
const from_json = try SearchRequest.from_json(allocator, "{\"query\":\"hello\"}");
defer from_json.deinit(allocator);

// Text format
w = .fixed(&buf);
try msg.to_text(&w);
// query: "hello"
// page_number: 1

Low-level wire format

The encoding module is available for direct wire format operations:

const encoding = @import("protobuf").encoding;
const message = @import("protobuf").message;

// Iterate fields in raw protobuf bytes without generated code
var iter = message.iterate_fields(bytes);
while (try iter.next()) |field| {
    switch (field.value) {
        .varint => |v| std.debug.print("field {}: varint {}\n", .{ field.number, v }),
        .len => |data| std.debug.print("field {}: {} bytes\n", .{ field.number, data.len }),
        .i32 => |v| std.debug.print("field {}: fixed32 {}\n", .{ field.number, v }),
        .i64 => |v| std.debug.print("field {}: fixed64 {}\n", .{ field.number, v }),
        .sgroup, .egroup => {},
    }
}

Dynamic messages (no codegen)

Decode protobuf bytes using runtime descriptors, without generated code:

const protobuf = @import("protobuf");

// Build a descriptor at runtime or from reflection data
const desc = protobuf.descriptor.MessageDescriptor{ ... };

var msg = try protobuf.dynamic.DynamicMessage.decode(allocator, &desc, bytes);
defer msg.deinit();

What gets generated

Each .proto message produces a Zig struct with:

pub const SearchRequest = struct {
    query: []const u8 = "",
    page_number: i32 = 0,
    results_per_page: i32 = 0,
    _unknown_fields: []const u8 = "",

    pub fn encode(self: @This(), writer: *std.Io.Writer) !void { ... }
    pub fn decode(allocator: std.mem.Allocator, bytes: []const u8) !SearchRequest { ... }
    pub fn calc_size(self: @This()) usize { ... }
    pub fn deinit(self: @This(), allocator: std.mem.Allocator) void { ... }
    pub fn to_json(self: @This(), writer: *std.Io.Writer) !void { ... }
    pub fn from_json(allocator: std.mem.Allocator, text: []const u8) !SearchRequest { ... }
    pub fn to_text(self: @This(), writer: *std.Io.Writer) !void { ... }
};

Field type mapping

Proto type Zig type
double / float f64 / f32
int32 / sint32 / sfixed32 i32
int64 / sint64 / sfixed64 i64
uint32 / fixed32 u32
uint64 / fixed64 u64
bool bool
string / bytes []const u8
repeated T []const T
map<string, V> std.StringArrayHashMapUnmanaged(V)
map<K, V> std.AutoArrayHashMapUnmanaged(K, V)
oneof ?union(enum) { ... }
message field ?MessageType

Proto2 optional fields are ?T (nullable). Proto3 implicit-presence scalars use zero defaults and are omitted from the wire when equal to the default.

Testing

The library is tested at multiple levels:

  • 640+ unit tests covering encoding, parsing, linking, codegen, JSON, text format, and dynamic messages (zig build test)
  • 24 AFL++ fuzz targets stress-testing the lexer, parser, varint codec, field iterator, JSON/text scanners, and message roundtrips with random inputs
  • Conformance tests against Google's official protobuf conformance suite validating wire format, JSON, and text format compliance
  • Go interoperability tests for RPC client/server communication
# Run unit tests
zig build test

# Run benchmarks
zig build bench

# Build a fuzz target (requires AFL++)
zig build -Dfuzz-target=parser fuzz

# Replay a fuzz crash
zig build -Dfuzz-target=parser fuzz-replay

Project structure

src/
  protobuf.zig         Root module
  encoding.zig         Wire format primitives (varint, zigzag, tags)
  message.zig          Schema-agnostic message reader/writer
  json.zig             Proto-JSON encoding/decoding
  text_format.zig      Proto text format serialization
  descriptor.zig       Runtime descriptor types
  dynamic.zig          Schema-driven dynamic messages
  rpc.zig              RPC types (StatusCode, streams, Channel)
  plugin.zig           Protoc plugin support
  well_known_types.zig Google well-known type handling
  GenerateStep.zig     build.zig integration step
  proto/
    lexer.zig          .proto tokenizer
    parser.zig         Recursive descent parser
    ast.zig            AST node definitions
    linker.zig         Type resolution and validation
  codegen/
    emitter.zig        Zig source text emitter
    messages.zig       Message struct generation
    enums.zig          Enum generation
    services.zig       RPC stub generation

Requirements

Zig 0.15.2

License

BSD-3-Clause

About

A pure-Zig Protocol Buffers library with a standalone .proto parser, build-time code generator, and transport-agnostic RPC stub generation. Proto2 + proto3. Zero external dependencies.

Topics

Resources

License

Stars

Watchers

Forks

Contributors