2. Introduction

2.1. Google Protocol Buffers for DDS

Vortex OpenSplice is capable of using the Google Protocol Buffer (GPB) system for publishing and subscribing GPB messages in a DDS system. This makes it possible to use GPB as an alternative to OMG-IDL for those who prefer to use GPB rather than IDL. With the seamless integration of GPB and DDS technologies there is no need for OMG-IDL knowledge or visibility when working with GPB data models, and no OMG-DDS data-types are needed in the application (no explicit type-mapping between GPB and DDS types is required).

This results in an easy migration of GPB users to DDS(-based data-sharing) with data-centric GPB with support for keys, filters and (future) QoS-annotations (ony a few DDS calls are needed). Also easy migration of DDS applications to GPB(-based data-modeling), only the field accessors change.

This Tutorial will describe how this is done for the language bindings Java5 and ISO-C++ by defining a GPB message layout which is compiled into proper interfaces for the Vortex DDS system.

2.1.1. GPB Installation and usage with DDS

Google Protocol Buffers (GPB) can be downloaded from the following locations:

Linux: https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.tar.gz

Windows: https://github.com/google/protobuf/releases/download/v2.6.1/protobuf-2.6.1.zip

After unpacking follow the install instructions located in install.txt in the unpacked directory. For windows there is a visual studio solution file that will build everything that is needed.

In order for GPB to work with DDS an environment variable PROTOBUF_HOME needs to be

set that points to the unpacked directory.

For windows also another environment variable PROTOBUF_LIB_HOME needs to be set that

points to the directory that contains the generated libprotobuf.lib.

2.1.2. IDL usage in a DDS system

In a Data Distributed System (DDS) as a Global DataSpace (GDS) for ubiquitous information-sharing in distributed systems as specified by the Object Management Group (OMG), the data is traditionally captured in the platform- and language-independent OMG-IDL language. The relational model of DDS is supported by the notion of identifying key fields in these data structures where structure/content-awareness by the middleware allows for dynamic querying and filtering of data.

2.1.3. Google Protocol Buffers

Google Protocol Buffers (GPB) are a flexible, efficient, automated mechanism for serializing structured data; think XML, but smaller, faster, and simpler. One can define how data needs to be structured once, after which language-specific source code can be generated to easily write and read this structured data to and from a variety of data streams using a variety of languages. The information structure is defined in so-called protocol buffer message types in .proto files. Each protocol buffer message is a small logical record of information, containing a series of name-value pairs. This approach is quite similar to using IDL for data modeling in combination with an IDL compiler (as available in OpenSplice and DDS implementations in general).

Additionally, the GPB data structure can be updated without breaking deployed programs that are compiled against the ‘old’ format, similar to the xTypes concept as defined for DDS.

2.1.3.1. Using a GPB data-model instead of an IDL data-model

For an IDL-OMG based application, the IDL file is compiled with the IDL-PP compiler to generate the needed classes.

For Java as an example, Address.idl will (among others) be compiled into:

• Address.java

• AddressTypesSupport.java

• AddressDataWriter.java

• AddressDataReader.java

• …

Using a GPB data-model, it is not necessary to create IDL files. The protoc_gen_ddsJava plug-in in OpenSplice will create them from the given .proto data-model.

For the GPB .proto based application, the .proto file is first compiled by the Google protoc compiler. This compiler will call the protoc_gen_ddsJava plug-in in OpenSplice with the .proto data parsed into a CodeGeneratorRequest protocol buffer.

The OpenSplice plug-in will generate an IDL file from this data. Any field member that is marked as key or filterable is explicitly mapped to a member in the IDL type.

The complete serialized .proto message is stored in the generic ospl_protobuf_data attribute as a sequence of bytes (making it opaque data for DDS). The mapping between data types is given in the table Mapping of GPB types to DDS types.

As the next step the IDL-PP compiler will generate the previously-named files from the idl file needed for the DDS domain. The Google protoc compiler will generate the classes needed for the GPB domain.

The dds options for the proto file are given in the omg/dds/descriptor.proto file listed below. This proto file shows how the different dds options on the proto file are interpreted, and gives the unique id 1016 to the dds types.

Note that the id 1016 has officially been granted to the Vortex product by Google.

This ensures these options are always unique and won’t clash with any options used by users.

How mapping is done between the different languages is shown below in the table Mapping of GPB types to DDS types.

2.1.3.2. omg/dds/descriptor.proto

syntax = "proto2";

import "google/protobuf/descriptor.proto";

package omg.dds;

option java_package = "org.omg.dds.protobuf";
option java_outer_classname = "DescriptorProtos";

/* These options are required for any .proto message that needs to be available
 * in DDS.
 *
 * - name: An optional scoped name to allow overriding the name of the type in
 *   DDS. The dot('.') can be used as a scoping separator. In case the name
 *   starts with a dot, the name will be interpreted as an absolute scope name.
 *   If not, the name will be considered relative to the scope of the message
 *   including its 'package'.
 */
message MessageOptions {
    optional string name = 1 [default = ""];
}
 
extend google.protobuf.MessageOptions {
  optional omg.dds.MessageOptions type  = 1016;
}

/* These options are provided to assign specific behaviour to a member of a
 * DDS-enabled .proto message in DDS. These options will only be applied in case
 * the omg.dds.MessageOptions.type has been applied to the message in which the
 * member is modeled.
 *
 * - key: Make the member part of the key of the type in DDS. Each unique
 *   key-value will become a separate instance with its own history in DDS. Only
 *   'required' members can be made part of the key and key-definitions cannot
 *   be modified in future versions of the message. Members that are part of the
 *   key are automatically filterable as well.
 *
 * - filterable: Ensure the member is filterable in DDS using a so-called
 *   ContentFilteredTopic or QueryCondition. Only 'required' members can be made
 *   filterable and filterable definitions cannot be modified in future versions
 *   of the message.
 *
 * - name: Override the name of the member in DDS. This only applies to members
 *   that are marked as key and/or filterable.
 */
message FieldOptions {
    optional bool key = 1 [default = false];
    optional bool filterable = 2 [default = false];
    optional string name = 3 [default = ""];
}

extend google.protobuf.FieldOptions {
  optional omg.dds.FieldOptions member = 1016;
}

2.1.3.3. Mapping of GPB types to DDS types

.proto Type	Notes	C++ Type	Java Type	DDS IDL Type
double		double	double	double
float		float	float	float
int32	Uses variable-length encoding. Inefficient for encoding negative numbers; if your field is likely to have negative values, use sint32 instead	int32	int	long
int64	Uses variable-length encoding. Inefficient for encoding negative numbers; if your field is likely to have negative values, use sint64 instead	int64	long	long long
uint32	Uses variable-length encoding	uint32	int	unsigned long
uint64	Uses variable-length encoding	uint64	long	unsigned long long
sint32	Uses variable-length encoding. Signed int value. These encode negative numbers more efficiently than regular int32s.	int32	int	long
sint64	Uses variable-length encoding. Signed int value. These encode negative numbers more efficiently than regular int64s.	int64	long	long long
fixed32	Always four bytes. More efficient than uint32 if values are often greater than 2^28.	uint32	int	unsigned long
fixed64	Always eight bytes. More efficient than uint64 if values are often greater than 2^56.	uint64	long	unsigned long long
sfixed32	Always four bytes.	int32	int	long
sfixed64	Always eight bytes.	int64	long	long long
bool		bool	boolean	bool
string	A string must always contain UTF-8 encoded or 7-bit ASCII text	string	String	string