XProc: An XML Pipeline Language

2.1 Steps

[Definition: A step is the basic computational unit of a pipeline.] A typical step has zero or more inputs, from which it receives XML documents to process, zero or more outputs, to which it sends XML document results, and can have options and/or parameters.

There are three kinds of steps: atomic, compound, and multi-container.

[Definition: An atomic step is a step that performs a unit of XML processing, such as XInclude or transformation, and has no internal subpipeline. ] Atomic steps carry out fundamental XML operations and can perform arbitrary amounts of computation, but they are indivisible. An XSLT step, for example, performs XSLT processing; a Validate with XML Schema step validates one input with respect to some set of XML Schemas, etc.

There are many types of atomic steps. The standard library of atomic steps is described in Section 7, “Standard Step Library”, but implementations may provide others as well. What additional step types, if any, are provided is implementation-defined. Each use, or instance, of an atomic step invokes the processing defined by that type of step. A pipeline may contain instances of many types of steps and many instances of the same type of step.

Compound steps, on the other hand, control and organize the flow of documents through a pipeline, reconstructing familiar programming language functionality such as conditionals, iterators and exception handling. They contain other steps, whose evaluation they control.

[Definition: A compound step is a step that contains a subpipeline.] That is, a compound step differs from an atomic step in that its semantics are at least partially determined by the steps that it contains.

Finally, there are two “multi-container steps”: p:choose and p:try. [Definition: A multi-container step is a step that contains several alternate subpipelines. ] Each subpipeline is identified by a non-step wrapper element: p:when and p:otherwise in the case of p:choose, p:group and p:catch in the case of p:try.

The runtime semantics of a multi-container step are that it behaves as if it evaluated exactly one of its subpiplines. In this sense, they function like compound steps.

[Definition: A compound step or multi-container step is a container for the steps directly within it or within non-step wrappers directly within it.] [Definition: The steps that occur directly within, or within non-step wrappers directly within, a step are called that step's contained steps. In other words, “container” and “contained steps” are inverse relationships.] [Definition: The ancestors of a step are its container, the container of its container, and all other containers above it.]

[Definition: Sibling steps (and the connections between them) form a subpipeline.] [Definition: The last step in a subpipeline is its last step in document order.]

subpipeline = p:variable*, (p:for-each|p:viewport|p:choose|p:group|p:try|p:standard-step|pfx:user-pipeline)+

Note that user-defined pipelines, pfx:user-pipeline, are atomic; although a pipeline declaration, contains a subpipeline, a step which invokes a user-defined pipeline does not.

Steps have “ports” into which inputs and outputs are connected or “bound”. Each step has a number of input ports and a number of output ports; a step can have zero input ports and/or zero output ports. (All steps have an implicit output port for reporting errors that must not be declared.) The names of all ports on each step must be unique on that step (you can't have two input ports named “source”, nor can you have an input port named “schema” and an output port named “schema”).

Steps may have any number of options, all with unique names. A step can have zero options.

Steps may have parameter input ports, on which parameters can be passed. The parameters passed on a particular parameter input port must be uniquely named. If multiple parameters with the same name are used, only one of the values will actually be available to the step. A step can have zero, one, or many parameter input ports, and each parameter port can have zero or more parameters passed on it.

All of the different instances of steps (atomic or compound) in a pipeline can be distinguished from one another by name. If the pipeline author does not provide a name for a step, a default name is manufactured automatically.

2.2 Inputs and Outputs

Although some steps can read and write non-XML resources, what flows between steps through input ports and output ports are exclusively XML documents or sequences of XML documents.

For the purposes of this specification, an XML document is an [Infoset]. Implementations are free to transmit infosets as sequences of characters, sequences of events, object models, or any other representation that preserves the necessary infoset properties (see Section A.3, “Infoset Conformance”).

Most steps in this specification manipulate XML documents, or portions of XML documents. In these cases, we speak of changing elements, attributes, or nodes without prejudice to the actual representation used by an implementation.

An implementation may make it possible for a step to produce non-XML output (through channels other than a named output port)—for example, writing a PDF document to a URI—but that output cannot flow through the pipeline. Similarly, one can imagine a step that takes no pipeline inputs, reads a non-XML file from a URI, and produces an XML output. But the non-XML data cannot arrive on an input port to a step.

It is a dynamic error (err:XD0001) if a non-XML resource is produced on a step output or arrives on a step input.

The common case is that each step has one or more inputs and one or more outputs. Figure 3, “An atomic step” illustrates symbolically an atomic step with two inputs and one output.

Figure 3. An atomic step

All atomic steps are defined by a p:declare-step. The declaration of an atomic step type defines the input ports, output ports, and options of all steps of that type. For example, every p:validate-with-xml-schema step has two inputs, named “source” and “schema”, one output named “result”, and the same set of options.

Like atomic steps, top level, user-defined pipelines also have declarations. The situation is slightly more complicated for the other compound steps because they don't have separate declarations; each instance of the compound step serves as its own declaration. On these compound steps, the number and names of the outputs can be different on each instance of the step.

Figure 4, “A compound step” illustrates symbolically a compound step with one subpipeline and one output. As you can see from the diagram, the output from the compound step comes from one of the outputs of the subpipeline within the step.

Figure 4. A compound step

[Definition: The input ports declared on a step are its declared inputs.] [Definition: The output ports declared on a step are its declared outputs.] When a step is used in a pipeline, it is connected to other steps through its inputs and outputs.

When a step is used, all of the declared inputs of the step must be connected. Each input can be connected to:

When an input accepts a sequence of documents, the documents can come from any combination of these locations.

The declared outputs of a step may be connected to:

The primary output port of a step must be connected, but other outputs can remain unconnected. Any documents produced on an unconnected output port are discarded.

Output ports on compound steps have a dual nature: from the perspective of the compound step's siblings, its outputs are just ordinary outputs and must be connected as described above. From the perspective of the subpipeline inside the compound step, they are inputs into which something must be connected.

Within a compound step, the declared outputs of the step can be connected to:

Each input and output is declared to accept or produce either a single document or a sequence of documents. It is not an error to connect a port that is declared to produce a sequence of documents to a port that is declared to accept only a single document. It is, however, an error if the former step actually produces more than one document at run time.

It is also not an error to connect a port that is declared to produce a single document to a port that is declared to accept a sequence. A single document is the same as a sequence of one document.

An output port may be connected to more than one input port. At runtime this will result in distinct copies of the output.

[Definition: The signature of a step is the set of inputs, outputs, and options that it is declared to accept.] The declaration for a step provides a fixed signature which all its instances share.

[Definition: A step matches its signature if and only if it specifies an input for each declared input, it specifies no inputs that are not declared, it specifies an option for each option that is declared to be required, and it specifies no options that are not declared.] In other words, every input and required option must be specified and only inputs and options that are declared may be specified. Options that aren't required do not have to be specified.

Steps may also produce error, warning, and informative messages. These messages are captured and provided on the error port inside of a p:catch. Outside of a try/catch, the disposition of error messages is implementation-dependent.

2.3 Primary Inputs and Outputs

As a convenience for pipeline authors, each step may have one input port designated as the primary input port and one output port designated as the primary output port.

[Definition: If a step has a document input port which is explicitly marked “primary='true'”, or if it has exactly one document input port and that port is not explicitly marked “primary='false'”, then that input port is the primary input port of the step.] If a step has a single input port and that port is explicitly marked “primary='false'”, or if a step has more than one input port and none is explicitly marked as the primary, then the primary input port of that step is undefined. A step can have at most one primary input port.

[Definition: If a step has a document output port which is explicitly marked “primary='true'”, or if it has exactly one document output port and that port is not explicitly marked “primary='false'”, then that output port is the primary output port of the step.] If a step has a single output port and that port is explicitly marked “primary='false'”, or if a step has more than one output port and none is explicitly marked as the primary, then the primary output port of that step is undefined. A step can have at most one primary output port.

The special significance of primary input and output ports is that they are connected automatically by the processor if no explicit binding is given. Generally speaking, if two steps appear sequentially in a subpipeline, then the primary output of the first step will automatically be connected to the primary input of the second.

Additionally, if a compound step has no declared outputs and the last step in its subpipeline has an unbound primary output, then an implicit primary output port will be added to the compound step (and consequently the last step's primary output will be bound to it). This implicit output port has no name. It inherits the sequence property of the port bound to it.

2.4 Connections

Steps are connected together by their input ports and output ports. It is a static error (err:XS0001) if there are any loops in the connections between steps: no step can be connected to itself nor can there be any sequence of connections through other steps that leads back to itself.

2.5 Environment

[Definition: The environment is a context-dependent collection of information available withing sub-pipelines.] Most of the information in the environment is static and can be computed for each subpipeline before evaluation of the pipeline as a whole begins. The in-scope bindings have to be calculated as the pipeline is being evaluated.

The environment consists of:

[Definition: The empty environment contains no readable ports, an undefined default readable port and no in-scope bindings.]

Unless otherwise specified, the environment of a contained step is its inherited environment. [Definition: The inherited environment of a contained step is an environment that is the same as the environment of its container with the standard modifications. ]

The standard modifications made to an inherited environment are:

A step with no parent inherits the empty environment.

2.6 XPaths in XProc

XProc uses XPath as an expression language. XPath expressions are evaluated by the XProc processor in several places: on compound steps, to compute the default values of options and the values of variables; on atomic steps, to compute the actual values of options and the values of parameters.

XPath expressions are also passed to some steps. These expressions are evaluated by the implementations of the individual steps.

This distinction can be seen in the following example:

<p:variable name="home" select="'http://example.com/docs'"/>

<p:load name="read-from-home">
  <p:with-option name="href" select="concat($home,'/document.xml')"/>
</p:load>

<p:split-sequence name="select-chapters" test="@role='chapter'">
  <p:input port="source" select="//section"/>
</p:split-sequence>

The href option of the p:load step is evaluated by the XProc processor. The actual href option received by the step is simply the string literal “http://example.com/docs/document.xml”. (The selection on the source input of the select-chapters step is also evaluated by the XProc processor.)

The XPath expression “@role='chapter'” is passed literally to the test option on the p:split-sequence step. That's because the nature of the p:split-sequence is that it evaluates the expression. Only some options on some steps expect XPath expressions.

The XProc processor evaluates all of the XPath expressions in select attributes on variables, options, parameters, and inputs, in match attributes on p:viewport, and in test attributes on p:when steps.

An XProc implementation can use either [XPath 1.0] or [XPath 2.0] to evaluate these expressions. This is a compromise driven entirely by the timing of XProc development. During the development of this specification, the community indicated that it was too early to mandate that all implementations use XPath 2.0 and too late to mandate that all implementations use XPath 1.0.

Many, many expressions that are likely to be used in XProc pipelines are the same in both versions (simple element tests, ancestor and descendant tests, string-based attribute tests, etc.).

As an aid to interoperability, pipeline authors may indicate the version of XPath that they are using. The attribute xpath-version may be used on p:pipeline, p:declare-step, (or p:library) to identify the XPath version that should be used to evaluate XPath expressions on the pipeline(s). The attribute is lexically scoped, but see below.

If an xpath-version is specified on a p:pipeline or p:declare-step, then that is the version of XPath that the step uses. If it does not specify a version, but a version is specified on one of its ancestors, the nearest ancestor version specified is the version that it uses. If no version is specified on the step or among its ancestors, then its XPath version is implementation-defined.

The following rules determine how the indicated version and the implementation's actual version interact:

2.7 Variables

Variables are name/value pairs. Pipeline authors can create variables to hold computed values.

[Definition: A variable is a name/value pair where the name is an expanded name and the value must be a string.] If a document, node, or other value is given, its XPath string value is computed and that string is used.

Variables and options share the same scope and may shadow each other.

2.8 Options

Some steps accept options. Options are name/value pairs, like variables. Unlike variables, the value of an option can be changed by the caller.

[Definition: An option is a name/value pair where the name is an expanded name and the value must be a string.] If a document, node, or other value is given, its XPath string value is computed and that string is used.

[Definition: The options declared on a step are its declared options.] Option names are always expressed as literal values, pipelines cannot construct option names dynamically.

[Definition: The options on a step which have specified values, either because a p:with-option element specifies a value or because the declaration included a default value, are its specified options.]

2.9 Parameters

Some steps accept parameters. Parameters are name/value pairs, like variables and options. Unlike variables and options, which have names known in advance to the pipeline, parameters are not declared and their names may be unknown to the pipeline author. Pipelines can dynamically construct sets of parameters. Steps can read dynamically constructed sets on parameter input ports.

[Definition: A parameter is a name/value pair where the name is an expanded name and the value must be a string.] If a document, node, or other value is given, its XPath string value is computed and that string is used.

[Definition: A parameter input port is a distinguished kind of input port which accepts (only) dynamically constructed parameter name/value pairs.] See Section 5.1.2, “Parameter Inputs”.

Analogous to primary input ports, steps that have parameter inputs may designate at most one parameter input port as a primary parameter input port.

[Definition: If a step has a parameter input port which is explicitly marked “primary='true'”, or if it has exactly one parameter input port and that port is not explicitly marked “primary='false'”, then that parameter input port is the primary parameter input port of the step.] If a step has a single parameter input port and that port is explicitly marked “primary='false'”, or if a step has more than one parameter input port and none is explicitly marked as the primary, then the primary parameter input port of that step is undefined.

How an implementation maps parameters specified to the application, or through some API, to parameters accepted by the top level pipeline is implementation-defined.

2.10 Security Considerations

An XProc pipeline may attempt to access arbitrary network resources: steps such as p:load and p:http-request can attempt to read from an arbitrary URI; steps such as p:store can attempt to write to an arbitrary location; p:exec can attempt to execute an arbitrary program. Note, also, that some steps, such as p:xslt and p:xquery, include extension mechanisms which may attempt to execute arbitrary code.

In some environments, it may be inappropriate to provide the XProc pipeline with access to these resources. In a server environment, for example, it may be impractical to allow pipelines to store data. In environments where the pipeline cannot be trusted, allowing the pipeline to access arbitrary resources or execute arbitrary code may be a security risk.

It is a dynamic error (err:XD0021) for a pipeline to attempt to access a resource for which it has insufficient privileges or perform a step which is forbidden.

A conformant XProc processor may limit the resources available to any or all steps in a pipeline. A conformant implementation may raise dynamic errors, or take any other corrective action, for any security problems that it detects.

2.11 Versioning Considerations

A pipeline author may identify the version of XProc against which a particular pipeline was authored by explicitly importing the library that identifies the steps defined by that version of XProc. For the version defined by this specification, the library is “http://www.w3.org/2008/xproc-1.0.xpl”.

If the version is not explicitly identified, the implicit version should be the most recent version known to the processor.

When a processor encounters a version it does not recognize, it proceeds in forwards-compatible mode. In forwards-compatible mode:

As a consequence, future specifications must not change the semantics of existing step types without changing their names.

3.1 XProc Namespaces

There are four namespaces associated with XProc:

3.2 Scoping of Names

Names are used to identify step types, steps, ports, options and variables, and parameters. Step types, options, variables, and parameters are named with QNames. Steps and ports are named with NCNames. The scope of a name is a measure of where it is available in a pipeline. [Definition: If two names are in the same scope, we say that they are visible to each other. ]

The scope of the names of the step types is the union of all the pipelines and pipeline libraries available directly or via p:import. In other words, the step types visible in a pipeline or library are:

All the step types in a pipeline must have unique names: it is a static error (err:XS0036) if any step type name is built-in and/or declared or defined more than once in the same scope.

The scope of the names of the steps themselves is determined by the environment of each step. In general, the name of a step, the names of its sibling steps, the names of any steps that it contains directly, the names of its ancestors, and the names of the siblings of its ancestors are all in a common scope. All steps in the same scope must have unique names: it is a static error (err:XS0002) if two steps with the same name appear in the same scope.

The scope of an input or output port name is the step on which it is defined. The names of all the ports on any step must be unique.

Taken together, these uniqueness constraints guarantee that the combination of a step name and a port name uniquely identifies exactly one port on exactly one in-scope step.

The scope of option and variable names is determined by where they are declared. When an option is declared with p:option (or a variable with p:variable), unless otherwise specified, its scope consists of the sibling elements that follow its declaration and the descendants of those siblings.

Parameter names are not scoped; they are distinct on each step.

3.3 Base URIs and xml:base

When a relative URI appears in an option value, the base URI against which it must be made absolute is the base URI of the p:option element. If an option value is specified using a syntactic shortcut, the base URI of the step on which the shortcut attribute appears must be used. In general, whenever a relative URI appears, its base URI is the base URI of the nearest ancestor element.

The pipeline author can control the base URIs of elements within the pipeline document with the xml:base attribute. The xml:base attribute may appear on any element in a pipeline and has the semantics outlined in [XML Base].

3.4 Unique identifiers

A pipeline author can provide a globally unique identifier for any element in a pipeline with the xml:id attribute.

The xml:id attribute may appear on any element in a pipeline and has the semantics outlined in [xml:id].

3.5 Associating Documents with Ports

[Definition: A binding associates an input or output port with some data source.] A document or a sequence of documents can be bound to a port in four ways: by source, by URI, by providing an inline document, or by making it explicitly empty. Each of these mechanisms is allowed on the p:input, p:output, p:xpath-context, p:iteration-source, and p:viewport-source elements.