Pattern Oriented Software Architecture V1
From Mud to Structure
****************************************
The Pipes and Filters pattern
****************************************
provides a structure for systems
that process a stream of data. Each processing step is
encapsulated in a filter component. Data is passed through pipes
between adjacent filters. Recombining filters allows you to build
families of related systems.
The Pipes and Filters pattern, in contrast, is less often used, but is
attractive in areas where data streams can be processed
incrementally.
_______________________________________
Context
Processing data streams.
_______________________________________
Problem
Imagine you are building a system that must process or transform a
stream of input data. Implementing such a system as a single
component may not be feasible for several reasons: the system has to
be built by several developers, the global system task decomposes
naturally into several processing stages, and the requirements are
likely to change.
You therefore plan for future flexibility by exchanging or reordering
the processing steps. By incorporating such flexibility, it is possible
to build a family of systems using existing processing components.
The design of the system-especially the interconnection of processing
________________________________________
Solution
The Pipes and Filters architectural pattern divides the task of a
system into several sequential processing steps. These steps are
connected by the data flow through the system-the output data of a
step is the input to the subsequent step. Each processing step is
implemented by a filter component. A filter consumes and delivers
data incrementally-in contrast to consuming all its input before
producing any output-to achieve low latency and enable real parallel
processing. The input to the system is provided by a data source such
as a text file. The output flows into a data sink such as a file, terminal,
animation program and so on. The data source, the filters and the
data sink are connected sequentially by pipes. Each pipe implements
the data flow between adjacent processing steps. The sequence of
filters combined by pipes is called a processing pipeline.
_______________________________________
Structure
//1
Filter components are the processing units of the pipeline. A filter
enriches, refines or transforms its input data. It enriches data by
computing and adding information, refines data by concentrating or
extracting information, and transforms data by delivering the data in
some other representation. A concrete filter implementation may
combine any of these three basic principles.
The activity of a filter can be triggered by several events:
1 The subsequent pipeline element pulls output data from the filter.
2 The previous pipeline element pushes new input data to the filter.
3 Most commonly, the filter is active in a loop, pulling its input from
and pushing its output down the pipeline.
The first two cases denote so-called passive filters, whereas the last
case is an active filter5. An active filter starts processing on its own as
a separate program or thread. A passive filter component is activated
by being called either as a function (pull) or as a procedure (push).
//2
Pipes denote the connections between filters, between the data source
and the first filter, and between the last filter and the data sink. If two
active components are joined, the pipe synchronizes them. This
synchronization is done with a first-in- first-out buffer. If activity is
controlled by one of the adjacent filters, the pipe can be implemented
by a direct call from the active to the passive component. Direct calls
make filter recombination harder, however.
____________________________________________
Dynamics
The following scenarios show different options for control flow
between adjacent filters. Assume that F i l t e r 1 computes function f 1
on its input data and ~ i l t e r 2 function f 2. The first three scenarios
show passive filters that use direct calls to the adjacent pipeline
components, with different components controlling the activity-no
explicit pipe components therefore exist. The last scenario shows the
commonest case, in which all filters are active, with a synchronizing
pipe between them.
Scenario I shows a push pipeline in which activity starts with the
data source. Filter activity is triggered by writing data to the passive
filters.
Scenario II shows a pull pipeline. Here control flow is started by the
data sink calling for data.
Scenario III shows a mixed push-pull pipeline with passive data
source and sink. Here the second filter plays the active role and starts
the processing.
Scenario lV shows a more complex but typical behavior of a Pipes
and Filters system. All filters actively pull, compute. and push data in
a loop. Each filter therefore runs in its own thread of control, for
example as a separate process. The filters are synchronized by a
buffering pipe between them. For simplicity we assume that the pipe
buffers only a single value. This scenario also shows how you can
achieve parallel execution using filters.
________________________________________________
Implementation
Implementing a Pipes and Filters architecture is straightforward. You
can use a system service such as message queues or UNIX pipes for
pipe connections, or other options like the direct call implementation,
as described in steps 3 through 6 below. The design decisions in these
steps are closely interrelated, so you may make them in an order
other than that given here. The implementation of the data source
and data sink is not addressed explicitly, because it follows the
guidelines for pipes or filters closely.
1 Divide the system's task into a sequence of processing stages.
2 Define the data format to be passed along each pipe.
3 Decide how to implement each pipe connection
4 Design and implement theflters.
5 Design the emr handling.
6 Set up the processing pipeline.
_________________________________________________
Varlants
Tee and join pipeline systems.The single-input slngle-output filter
specification of the Pipes and Filters pattern can be varied to allow
filters wlth more than one input and/or more than one output.Processing
can then be set up as a directed graph that can even contaln
feedback loops. The deslgn of such a system, especially one wlth
feedback loops, requires a solid foundation to explain and
understand the complete calculation-a rigorous theoretical analysls
and specification using formal methods are appropriate, to prove that
the system terminates and produces the desired result. If we restrict
ourselves to simple directed acyclic graphs, however, it is still posslble
to build useful systems.
_____________________________________________________
Known Uses
UNIX
CMS Pipelines
LASSPTools
____________________________________________________
Consequences
The Pipes and Filters architectural pattern has the following benefits:
No intermediate files necessary, but possible.
Flexibility by filter exchange.
Flexibility by recombination.
Reuse ofJlter components.
Rapid prototyping ofpiplines.
Egiciemy by parallel processing.
___________________________________________________
See Also
The Layers pattern