The design of all pipeline processing components is based on the GStreamer framework. GStreamer is an open source multimedia application framework, which “allows the construction of graphs of media-handling components.” A high-level overview of GStreamer features may be found on the GStreamer website at http://gstreamer.freedesktop.org/features/.[3] The following sections provide simple descriptions of the GStreamer framework in the context of the backend software.
GStreamer supports the construction of pipelines as arbitrary, directed graphs of processing elements. Such pipeline elements are the basic building blocks of the pipelines in the backend. Elements are connected via the “pads” that may exist on an element. Terminal elements in a pipeline are classified as source or sink elements, depending on whether pipeline data is put into or taken out of the pipeline by that element. Non-terminal pipeline elements may be classified in a variety of ways, such as filter, demuxer, etc. Elements are available to application programs by dynamically loaded plugins, providing the backend software with a clear mechanism for extensibility.
Pads are the component structures of elements over which data flows in a pipeline. Every pad may be classified as either a sink or source pad (as seen from a point of view outside of the element; in this way, a source element has only source pads, and a sink element, only sink pads). Elements are connected in a pipeline through their pads. Since any given element may be able to process data in a limited number of formats, each pad has associated with it a set of allowed data types (media types, in the GStreamer terminology). Whether two pipeline elements can be connected depends on the type of the data that would pass between the elements, and the data types allowed by elements' respective sets of source and sink pads.
Data (or media) types in the GStreamer sense are based on MIME types with added properties. Pad capabilities (or caps in GStreamer terminology) are effectively sets of MIME types together with arbitrary properties associated with each type. Whereas a pad's set of allowed caps is typically specified at compile time, caps negotiation occurs at run time before data can pass between connected pads. Caps negotiation ensures that connected elements pass data in a format acceptable to both sender and receiver.
The set of data types used by the backend software is intentionally somewhat limited, but the set may be extended using the plugin mechanism.
GStreamer pipeline elements are based on GObject objects, and, consequently, have object properties available to them. GObject properties function as simple attributes in an object-oriented model. Various pipeline element properties are available to provide monitor and control functionality for the backend pipelines. For example, an element may provide a boolean-valued property to be used to switch on or off the data flow through a pipeline.
Individual elements may be linked together into higher order elements termed bins. In the context of the backend software, it is envisioned that bins may be a useful mechanism by which to extend the available pipeline elements. Because certain elements of backend pipelines are likely to be used universally, and those elements will have fixed pad capabilities, new elements will be required to use a common backend data type as input, and produce a common backend data type as output. Thus, it may be useful to implement a new element as a bin whenever intermediate data types are desired.
Another potential application of bins in the backend software would be to package the most commonly used pipelines. Bundling elements in this way would allow a simplified configuration specification whenever the backend were to be configured in some widely applicable, common mode. Whether this approach might be useful or not is uncertain.
Signals are a notification mechanism used in the GStreamer framework[4] to connect arbitrary application-specific events with any number of listeners. Signals are user-definable, and are used to trigger callbacks conforming to an interface specified by the type of the signal. In a multi-threaded framework, such as GStreamer, it is important to understand that signals are synchronous, and the callbacks triggered by a signal run in the same thread in which the signal was emitted. In the backend software, signals may be used as a low-level notification mechanism between components of a pipeline (such as between an element and the application).
A message bus is a system to forward messages from the pipeline threads to the application thread. This allows an application to remain thread-unaware, although GStreamer pipelines are heavily threaded. In the backend software, a commonly used pattern is for callbacks connected to particular signals to simply create application messages and post them to the message bus, which has the effect of transferring signals from the pipeline threads to the application thread.
One use of the message bus in the backend software is to read from and write to pipeline element properties, which provides the pipeline application with important monitor and control capabilities. Another use of the message bus is described in the section called “Lag processing stage management”.
Events in GStreamer are “control particles that are sent both up- and downstream in a pipeline”. Downstream events are commonly used in GStreamer to notify elements of stream states,[5] and these types of events are the only ones used by the backend software. Events are also classified according to whether they are serialized in the data stream or not, allowing both data-synchronous and asynchronous event types.
One use of application-specific events in the backend is to signal changes in the backend configuration, as described in the section called “Configuration”. Because events may carry data, they provide a means to bring auxiliary data (for example, the zero lags, or state counts) to the pipeline elements. While it is planned to use events as carriers of auxiliary information, this feature is not yet implemented.
[3] All GStreamer documentation may be found on the website at http://gstreamer.freedesktop.org.
[4] These signals have nothing to do with standard Unix system signals.
[5] Upstream events are commonly used in GStreamer to allow an application to control data streaming, for example, seeking the position of an audio stream by a playback application.