protobuf

  1. protobuf
    1. Description
    2. Programming Interface
      1. protobuf
      2. protobuf-generic
      3. protobuf-reflection
      4. protobuf-syntax
      5. protobuf-encoding
    3. Example
    4. Author
    5. Requirements
    6. License

Description

The egg provides facilities to read and write Protocol Buffers in binary format and comes with a plug-in for the protoc compiler that generates CHICKEN code from a .proto definition file.

Programming Interface

Most of the time you will run protoc --chicken_out=... on your definition files and import only the generated modules plus the protobuf module from the protobuf library into your code. The generated modules contain SRFI-99 record types and a convenient constructur for every protocol buffer message type, as well as reflection information used by the (de-)serialization process for every protocol buffer message or enumeration type. The generated modules are written in terms of protobuf-syntax.

If you don't want to bother with application specific protocol buffer definitions, you can also use the protobuf-generic module to serialize almost any value you might encounter in a CHICKEN program.

Additionally the protobuf library contains modules that allow you to hook into the protocol buffer (de-)serialization process or perform low-level manipulations of the wire format manually.

protobuf

This module contains the high-level entry points for protocol buffer (de-)serialization and re-exports message?, message-extensions and message-unknown from the protobuf-reflection module.

[procedure] (deserialize TYPE [PORT])

Deserializes a binary protocol buffer stream from PORT (or (current-input-port)) into a record of the given TYPE, which must be the SRFI-99 record type descriptor of a protocol buffer message type.

[procedure] (serialize MSG [PORT])

Serializes the protocol buffer message MSG to the PORT (or (current-output-port)) as a binary protocol buffer stream. MSG must be a SRFI-99 record instance of a protocol buffer message type.

protobuf-generic

This module contains the high-level entry points for protocol buffer (de-)serialization of CHICKEN values without any application specific schema definition. The module code is optimized for this task and uses only the protobuf-encoding facilities. It emits protocol buffers conforming to the schema defined in extend/protobuf/chicken.proto.

The following values can be serialized:

[parameter] current-serialization-context

Holds the current default serialization context, an object that tracks the identity of values that have already been serialized or deserialized.

The parameter value may also be #f, which is its default value.

[procedure] (make-serialization-context [V ...])

Creates a serialization context and initializes it with a set of well known values. This can be used to replace certain non-serializable values with "external" references upon serialization and to inject those external dependencies upon deserialization. The context always sees itself as an external dependency.

[procedure] (serialization-context? V)

Checks whether something is a serialization context.

[type] serialization-info

A SRFI-99 record type holding a custom reader and writer procedures in its two immutable fields.

[constant] prop:serialization-info

A record property that associates a serialization-info structure with SRFI-99 record types and instances.

[procedure] (serialize V [PORT] [CONTEXT])

Writes the value V in generic protocol buffer format to the PORT (defaulting to (current-output-port)). The given CONTEXT (defaulting to (current-serialization-context)) is used to track serialized values, allowing the correct external representation of shared structure.

If the context is #f, a new context is allocated and initialized with the current input, output and error ports as external dependencies.

[procedure] (deserialize [PORT] [CONTEXT])

Reads a value in generic protocol buffer format from the PORT (defaulting to (current-input-port)). The given CONTEXT (defaulting to (current-serialization-context)) is used to track deserialized values, allowing the correct reconstruction of shared structure.

If the context is #f, a new context is allocated and initialized with the current input, output and error ports as external dependencies.

protobuf-reflection

This module contains the data structures used by the (de-)serialization code to guide its operation. It can also be used for purposes of introspection.

[type] type-info

A SRFI-99 record type that serves as the basis for primitive, enumeration or message type descriptors. It contains just an immutable name field.

This abstract type has no constructor.

[type] primitive-info

The record type of descriptors for primitive protocol buffer wire types. It contains

[type] enum-info

The record type of descriptors for protocol buffer enumeration types. It contains

[type] message-info

The record type of descriptors for protocol buffer message types. It contains

[type] field-info

The record type of message field descriptors. It contains

The type field refers to a SRFI-99 record type descriptor of another protocol buffer message or to an enumeration or primitive type descriptor record. The field-info-type accessor automatically forces the promise contained in this field.

[type] message

A SRFI-99 record type that serves as the basis for any protocol buffer message type. It contains

This abstract type has no constructor.

protobuf-syntax

This module contains the syntactic forms used by modules created through protoc-gen-chicken.

[constant] int32
[constant] int64
[constant] uint32
[constant] uint64
[constant] sint32
[constant] sint64
[constant] fixed32
[constant] fixed64
[constant] sfixed32
[constant] sfixed64
[constant] bool
[constant] float
[constant] double
[constant] bytes
[constant] string

Descriptors for the standard primitive protocol buffer types.

[procedure] (uint* MAX-SIZE)
[procedure] (sint* MAX-SIZE)

Constructors for primitive type descriptors compatible with standard unsigned or signed integers, but with arbitrary maximum encoding size. MAX-SIZE can also be #f to allow arbitrary precision integers.

The protocol compiler plugin allows access to these types through a max_size extension to the google.protobuf.FieldOptions message defined in extend/protobuf/bigint.proto.

[syntax] (define-enum-type NAME (ITEM TAG) ...)

Defines a new enumeration type.

[syntax] (define-message-type NAME ({required|optional|repeated|packed} TYPE FIELD TAG [DEFAULT]) ...)

Defines a new message type, which is a SRFI-99 record type with a specialized constructor.

The constructors for protocol buffer messages take a keyword argument for every field. Missing fields are initialized to their defaults or a void value.

[syntax] (define-message-extension NAME ({required|optional|repeated|packed} TYPE FIELD TAG [DEFAULT]) ...)

Extends an existing message type with additional synthetic fields.

protobuf-encoding

This module forms the low-level layer for protocol buffer wire format encoding and decoding.

[procedure] (make-limited-input-port PORT LIMIT CLOSE-ORIG?)

Creates an input port reading up to LIMIT bytes from PORT before entering an end-of-file state.

[procedure] (read-uint* [PORT] [MAX-SIZE])

Reads a variable length base 128 encoded unsigned integer from PORT (defaulting to (current-input-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

[procedure] (write-uint* N [PORT] [MAX-SIZE])

Writes a variable length base 128 encoded unsigned integer to PORT (defaulting to (current-output-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

[procedure] (read-sint* [PORT] [MAX-SIZE])

Reads a variable length base 128 zig-zag-encoded signed integer from PORT (defaulting to (current-input-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

[procedure] (write-sint* I [PORT] [MAX-SIZE])

Writes a variable length base 128 zig-zag-encoded signed integer to PORT (defaulting to (current-output-port)). Stops with a syntax error if the encoding is longer than MAX-SIZE bytes (defaulting to 10); pass #f to disable the length check.

[procedure] (read-int* [PORT])

Reads a variable length base 128 two's complement encoded signed integer from PORT (defaulting to (current-input-port)). Stops with a syntax error if the encoding is longer than 10 bytes.

[procedure] (write-int* I [PORT])

Writes a variable length base 128 two's complement encoded signed integer to PORT (defaulting to (current-output-port)). Stops with a syntax error if the encoding is longer than 10 bytes.

[procedure] (read-bool [PORT])

Reads a boolean from PORT (defaulting to (current-input-port)).

[procedure] (write-bool V [PORT])

Writes a boolean to PORT (defaulting to (current-output-port)).

[procedure] ((read-fixed* SIZE SIGNED?) [PORT])

Reads a fixed length, possibly SIGNED?, integer from PORT (defaulting to (current-input-port)). SIZE must be 4 or 8.

[procedure] ((write-fixed* SIZE SIGNED?) N [PORT])

Writes a fixed length, possibly SIGNED?, integer to PORT (defaulting to (current-output-port)). SIZE must be 4 or 8.

[constant] read-fixed32

An alias for (read-fixed* 4 #f).

[constant] write-fixed32

An alias for (write-fixed* 4 #f).

[constant] read-fixed64

An alias for (read-fixed* 8 #f).

[constant] write-fixed64

An alias for (write-fixed* 8 #f).

[constant] read-sfixed32

An alias for (read-fixed* 4 #t).

[constant] write-sfixed32

An alias for (write-fixed* 4 #t).

[constant] read-sfixed64

An alias for (read-fixed* 8 #t).

[constant] write-sfixed64

An alias for (write-fixed* 8 #t).

[procedure] ((read-float* SIZE) [PORT])

Reads a fixed length floating point number from PORT (defaulting to (current-input-port)). SIZE must be 4 or 8.

[procedure] ((write-float* SIZE) X [PORT])

Writes a fixed length floating point number to PORT (defaulting to (current-output-port)). SIZE must be 4 or 8.

[constant] read-float

An alias for (read-float* 4).

[constant] write-float

An alias for (write-float* 4).

[constant] read-double

An alias for (read-float* 8).

[constant] write-double

An alias for (write-float* 8).

[procedure] (read-sized-bytes [PORT])

Reads a SRFI-4 u8vector with size information from PORT (defaulting to (current-input-port)).

[procedure] (write-sized-bytes BSTR [PORT])

Writes a SRFI-4 u8vector with size information to PORT (defaulting to (current-output-port)).

[procedure] (read-sized-string [PORT])

Reads a string with size information from PORT (defaulting to (current-input-port)).

[procedure] (write-sized-string STR [PORT])

Writes a string with size information to PORT (defaulting to (current-output-port)).

[procedure] (read-sized READ [PORT])

Reads a block with size information from PORT (defaulting to (current-input-port)). Uses READ to decode the contents of the block.

[procedure] (write-sized WRITE V [PORT])

Uses WRITE to encode a block to PORT (defaulting to (current-output-port)). Automatically adds size information to the output.

[procedure] (read-tag/type [PORT])

Reads the tag and wire type of a protocol buffer message field from PORT (defaulting to (current-input-port)). The type is one of the symbols int*, 32bit, 64bit or sized.

[procedure] (write-tag/type TAG TYPE [PORT])

Writes the tag and wire type of a protocol buffer message field to PORT (defaulting to (current-output-port)). The TYPE must be one of the symbols int*, 32bit, 64bit or sized.

Example

$ cat msg.proto
message Msg {
  optional uint64 ts = 2;
  optional string data = 3; 
}


$ protoc --chicken_out=. msg.proto
$ cat msg.scm
;; Generated by protoc-gen-chicken v1.0.0
(module
  main
  *
  (import (except scheme string) chicken protobuf-syntax)
  (define-message-type msg (optional uint64 ts 2) (optional string data 3)))


$ cat serializer.scm
(use protobuf) (include "msg.scm") (import main)
(serialize (make-msg data: "hello world"))
;; obs: don't concat multiple messages,
;; you need a message boundary.


$ cat deserialize.scm
(use protobuf) (include "msg.scm") (import main)
(pp (msg-data (deserialize msg)))


$ csi -s serializer.scm | csi -s deserialize.scm
"hello world"

Author

Thomas Chust

Requirements

License

 Copyright (C) 2013 Thomas Chust <chust@web.de>.  All rights reserved.
 
 Permission is hereby granted, free of charge, to any person obtaining a
 copy of this software and associated documentation files (the Software),
 to deal in the Software without restriction, including without limitation
 the rights to use, copy, modify, merge, publish, distribute, sublicense,
 and/or sell copies of the Software, and to permit persons to whom the
 Software is furnished to do so, subject to the following conditions:
 
 The above copyright notice and this permission notice shall be included
 in all copies or substantial portions of the Software.
 
 THE SOFTWARE IS PROVIDED ASIS, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
 IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
 FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL
 THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR
 OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE,
 ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
 OTHER DEALINGS IN THE SOFTWARE.