Skip to main content
Elias Groot

Elias Groot

Software Lead, Project Administrator

Stream

A stream is a method of communication between services in a pipeline. A service writes output data on a write stream, which can be read by any other service using an read stream.

Streams are meant to give a higher-level overview of communication patterns to understand how data flows between services. However, all streams are implemented using ZeroMQ PUB/SUB sockets.

Declarations​

All streams are named by the service that outputs them in its service.yaml definition. Per service, each stream name must be unique. Among services, stream names can be reused. Any stream is uniquely identified in a pipeline by the tuple of service name and stream name.

Stream declarations in service.yaml files are processed by roverd and specific ZMQ properties (including protocol and ports for the socket to set up) are injected into a service's runtime using the bootspec.

Compatibility​

Streams are first-class objects in the ASE framework. They are identified, validated and visualized using roverd and roverctl. You can use any mechanism to communicate between services, but if you wish to use streams, you must set up these streams using ZMQ PUB/SUB sockets.

Stream Set-up​

After parsing stream information through a bootspec, it is the service's duty to set up the correct ZMQ sockets for reading and writing data. When using a roverlib, APIs in the programming language of choice are provided to create and use read/write streams without setting up low-level ZMQ sockets.

View the streams implementation here.

When not using a roverlib, sockets should be set up manually using the PUB/SUB messaging pattern as follows:

Write Stream​

  • Must create a ZMQ_PUB socket
  • This socket must bind to the address specified in the bootspec
  • This socket must not specify a topic (no topics are used)

Read Stream​

  • Must create a ZMQ_SUB socket
  • This socket must connect to the address specified in the bootspec
  • This socket must subscribe to all topics using the subscribe("") call in the chosen language

Documentation on setting up PUB/SUB sockets in python can be found here. General documentation about PUB/SUB messaging using ZeroMQ can be found here. The code used by a roverlib to set up ZMQ sockets can be found here.