xproc 2006-09-12 XML Norman Walsh Sun Microsystems, Inc. Norman.Walsh@Sun.COM This document is a shell where ideas, points of consensus, and early draft text is being collected. It does not necessarily represent the consensus of the Working Group. This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/. More boilerplate goes here…

An XML Pipeline describes a sequence of operations to be performed on a collection of input documents. Pipelines take zero or more XML documents as their input and produce zero or more XML documents as their output. Components in the pipeline may read or write non-XML resources as well. Each operation in a pipeline is performed by a component. Like pipelines, components take zero or more XML documents as their input and produce zero or more XML documents as their output. The inputs to a component come from the web, from the pipeline document, from the inputs to the pipeline itself, or from the outputs of other components in the pipeline. The outputs from a component are either consumed by other components or are outputs of the pipeline as a whole. Outputs may also be ignored. This specification defines a standard component library, . Pipeline implementations may support additional components as well. is a graphical representation of a simple pipeline that performs XInclude processing and validation on a document.

A simple, linear XInclude/Validate pipeline

This is a pipeline that consists of two components, XInclude and Validate. The pipeline itself has two inputs, “document” and “schema”. How these inputs are connected to XML documents is implementation-defined. The XInclude component reads the pipeline input “Document” and produces a result document. The Validate component reads the pipeline input “Schema” and the output from the XInclude component and produces a result document. The result of the validation is the result of the pipeline, “Result Document”. How pipeline outputs are connected to XML documents is implementation-defined. is a more complex example: it performs schema validation with an appropriate schema and then styles the validated document.

A validate and transform pipeline

The heart of this example is the conditional. The standard “choose” component evaluates an XPath expression over a test document. Based on the result of that expression, one or another branch is evaulated.

A pipeline is an acyclic, directed graph of components connected together by inputs and outputs. A pipeline is itself a component and must satisfy the constraints on components. The result of evaluating a pipeline is the result of evaluating the components that it contains. A pipeline must behave as if it evaluated each component each time it occurs. Unless otherwise indicated, implementations must not assume that components are functional (that is, that their output depends only on their explicit inputs and parameters) or side-effect free.

In a pipeline document, the components of a pipeline are represented with steps. Each step represents one or more components and describes how the component(s) it represents fit into the pipeline. Many steps are simple, atomic representations of a single component. An XSLT step, for example, represents an XSLT component; a validation step rerpesents a validation component, etc. However, some steps include a description of the component-level behavior that they instantiate in terms of additional steps. A step which contains other steps is called a step containers. The steps that occur inside another step are called contained steps. Some additional semantic rules apply to contained steps: Is this a good place/the right place for this kind of summary? The contained steps can read input from the declared inputs of their step container (or ancestor step containers). The output of one component can only be connected to the input of another component that is in the same step container. Any set of components, instantiated from one or more steps or a set of contained steps forms a subpipeline. Subpipelines are most often invoked by components that instantiate step containers, where the collection of components instantiated by the contained steps forms a subpipeline.

Components are the basic computational units of a pipeline. A component is a unit of XML processing, such as XInclude or transformation. Components may perform arbitrary amounts of computation but they are indivisible from the point of view of the pipeline that instantiates them. Components have “ports” into which inputs and outputs are connected. Each component has a number of input ports and a number of output ports, all with unique names. A component may have zero input ports and/or zero output ports. (All components have a standard output port for reporting errors that does not have to be, and cannot be, declared.) Components have any number of parameters, all with unique names. A component may have zero parameters. All of the input ports of a component must be connected to inputs. It is a static error if a component has an input port which is not connected. Unconnected output ports are allowed; any documents produced on those ports are simply discarded. The signature of a component is the set of inputs, outputs, and parameters that it is declared to accept. The instantiation of a component matches its signature if and only if it specifies an input for each declared input and it specifies no inputs that are not declared, it specifies no outputs that are not declared, it specifies a parameter for each parameter that is declared to be required, and it specifies no parameters that are not declared. In other words, every input and required parameter must be specified and only inputs, outputs, and parameters that are declared may be specified. Outputs and optional parameters do not have to be specified.

Although components are free to read and write non-XML resources, what flows between components as inputs and outputs are exclusively XML documents or sequences of XML documents. Each XML document (or document in a sequence) must be a well formed or document. The inputs and outputs may be implemented as sequences of characters, events, or object models, or any other representation the implementation chooses. Is support for XML 1.1 optional? It is a dynamic error if a non-XML resource is produced on a component output or arrives on a component input. What about the cases where it's impractical to test for this error? An implementation may make it possible for a component to produce non-XML output, for example, writing a PDF document, but those results cannot flow through the pipeline. Similarly, one can imagine a component that takes no inputs, reads a non-XML file from a URI, and produces an XML output. But the non-XML file cannot be an input to the component or pipeline. Each input and output is declared to accept or produce either a single document or a sequence of documents. It is not a static error to connect a port which produces a sequence of documents to a port that accepts only a single document. It is, however, a dynamic error if the former component actually produces more than a single document at run time. Components may also produce error, warning, and informative messages. These messages appear on a special “error output” that is available in the catch clause of a try/catch.

A parameter is a QName/value pair. The value of a parameter must be a string. If a document, node, or other value is given, its (XPath 1.0) string value is computed and that string is used.

The components of a pipeline are the nodes of a flow graph. The inputs and outputs of the components are connected by arcs in that graph. Consider two components in such a graph, “component A” and “component B”. Components A and B are connected if they are either directly or indirectly connected. Component A is directly connected to B if an output of A is associated with an input port of B. Component A is indirectly connected to B if there is a chain of directly connected components that allows traversal from A to B. With respect to connected components, we can speak of one component being either before or after another. Component A is before component B if component B is one of the contained steps of component A, either directly or indirectly, or if any output from component A is connected to any input of component B, either directly or indirectly. after is the converse of before. It is static error if a component is either before or after itself. In other words, the flow graph must be acyclic.

This section describes the core language constructs of XProc.

A [[pipeline]] component encapsulates the behavior of a set of components. It has a number of declared input and output ports and parameters. Viewed from the outside, it is a black box which performs some calculation on the inputs and produces the outputs. From the pipeline author's perspective, the computation performed by the pipeline is described in terms of a set of contained steps which reads the pipeline's inputs and produce the pipeline's outputs. For example, a pipeline might accept a document and a stylesheet as input; perform XInclude, validation, and transformation over its inputs; and produce a sequence of formatted documents as its output.

A [[for-each]] component processes a sequence of documents, applying a subpipeline to each document in turn. In cases where a subpipeline requires a single document input, but a pipeline needs to process a sequence of documents with that subpipeline, the [[for-each]] construct can be used. The result of the [[for-each]] is a sequence of the documents produced by processing each individual document in the input sequence. If the subpipeline is connected to one or more output ports on the [[for-each]], what appears on each of those ports is the sequence of documents produced by each iteration of the loop. For example, a [[for-each]] might accept a sequence of DocBook chapters as its input, process each chapter in turn with XSLT, and produce a sequence of formatted chapters as its output.

A [[viewport]] component processes a single document, applying a subpipeline to one or more subsections of the document. The result of the [[viewport]] is a copy of the original document with the selected subsections replaced by the results of applying the subpipeline to them. For example, a [[viewport]] might accept an XHTML document as its input, apply encryption to selected div elements within that document, and return an XHTML document that is the same as the original except that each selected div has been replaced by its encrypted result.

A [[choose]] component selects exactly one of a list of possible subpipelines based on the evaluation of XPath expressions. Each subpipeline is effectively guarded by an XPath expression. The [[choose]] considers each subpipeline in turn and selects the first (and only the first) subpipeline for which the guard expression evaluates to true. The last subpipeline may be designated the default subpipeline; it will be selected if no preceding subpipeline was selected. After a subpipeline is selected, it is evaluated as if only it had been present. The result of the [[choose]] is the result of the selected subpipeline. For example, a [[choose]] might test a schema and apply XML Schema validation to an input document if the schema is an XML Schema document, apply RELAX NG validation if the schema is a RELAX NG grammar, or perform no validation otherwise. In order to ensure that the result of the [[choose]] is consistent irrespective of the subpipeline chosen, each subpipeline must declare the same number of outputs with the same names. It is a static error if two subpipelines in a [[choose]] declare different outputs. It is a dynamic error if no subpipeline is selected by the [[choose]] and no default is provided.

A [[group]] component encapsulates the behavior of a set of components. It is a convenience wrapper for a collection of components and may be used to perform parameter renaming to aid in reuse of sets of components.

A [[try]] component isolates a subpipeline, preventing any errors that arise within it from being exposed to the rest of the pipeline. A [[try]] component begins with two subpipelines: an initial [[group]] and a [[catch]]. It evaluates the initial subpipeline and, if no errors occur, the results of that pipeline are the results of the component. However, if any errors occur, it abandons the first subpipeline, discarding any output that it may have generated, and evaluates the [[catch]] subpipeline. In this case, the results of the [[catch]] subpipeline are the results of the [[try]] component. For example, a pipeline might attempt to process a document by dispatching it to some web service. If the web service succeeds, then those results are passed to the rest of the pipeline. However, if the web service cannot be contacted or reports an error, the [[catch]] component can provide some sort of default for the rest of the pipeline. In order to ensure that the result of the [[try]] is consistent irrespective of whether the [[try]] subpipeline provides its output or the [[catch]] subpipeline does, both the [[try]] and [[catch]] subpipeline must declare the same number of outputs with the same names. It is a static error if the two subpipelines declare different outputs. In order to support corrective action in the [[catch]] subpipeline, components inside the [[catch]] have access to all the error output of the components that were in the [[try]] subpipeline on a special port named “#error”. In evaluating the [[try]] subpipeline, failure of one component may cause other components to fail. In addition, some components that fail may not produce output on their error ports and some components that succeeded may produce such output. This pipeline language places no constraints on the order of error messages provided to the [[catch]] subpipeline, nor does it attempt to guarantee that such output will be available in all cases. The error documents that appear should conform to .

A pipeline document may declare additional components. These may be implementation-defined components or may be defined through some implementation-dependent extension mechanism. Each declared component must have a name and a signature. It is a static error if a pipeline refers to a component that is not recognized by the processor.

This section describes the syntactic elements necessary to instantiate a pipeline.

At the highest level, a pipeline is a collection of steps. Each step has a unique name and instantiates a particular component. The component has a signature and that signature has to be satisified by the inputs and parameters specified on the step. Recall that a pipeline is not required to consume all the outputs of a component, but it must identify all the inputs and required parameters. Components that contain subpipelines are represented naturally using XML hierarchy by placing steps inside other steps.

A step can bind a document or a sequence of documents to a port in three ways: by source, by URI, or by providing it “here”, as the content of the element establishing the binding. A document must be specified in exactly one of these ways. Specified by URI A document is specified by URI if it refers to it with a URI. The href attribute is used for this purpose. In this example, the input to the Identity step named “otherstep” comes from “http://example.com/input.xml”. ]]> It is a dynamic error if the processor attempts to retrieve the specified URI and fails. (For example, if the resource does not exist or is not accessible with the users authentication credentials.) Specified by source A document is specified by source if it refers to a specific port on another step. The step and source attributes are used for this purpose. The specified port must either be a declared input on some ancestor (e.g., an enclosing p:choose or p:for-each) or it must be an output port of some other step. In some contexts there are additional constraints on the step that can be selected, for example, that it must be (or must not be) a descendant of the step on which the binding occurs. In this example, the input to the XInclude step named “expand” comes from the “result” port of the step named “otherstep”. ]]> It is a static error if the specified port does not exist. Specified by here document An document is specified by here document if it is contained in the body of the element that binds it. In this example, the stylesheet input to the XSLT step named “xform” comes from the content of the input element. ... ]]> Here documents are considered “quoted”, they are not interpolated or available to the pipeline processor in any way except as a document flowing through the pipeline.

The scope of a port name is the component on which it is defined. The names of all input and output ports on any component must be unique. The scope of step names is the flow graph. Should we say that components have names and remove steps from the equation all together?

In order for step container components (such as p:for-each and p:choose) to function as “black boxes” the way other components do, it is necessary to make sure that they are wholly self-contained. This means that any inputs, outputs, and parameters that are used within the subpipeline must be declared: ]]> However, in practice, forcing authors to declare all of the inputs and parameters to each component is tedious and error-prone. What's more, the pipeline processor can actually determine what additional declarations are necessary simply by examining the source declarations on the steps contained within the step. Therefore, we allow the syntactic shortcut of simply referring to in-scope sources and parameters directly. ]]> These shortcuts are purely syntactic. The processor is required to behave as if appropriate, unique declarations with the corresponding remapping of names had been present. The WG is still considering whether or not authors will be allowed to write pipelines using the fully-qualified syntax. This draft does not actually allow p:declare-input on language constructs so the fully qualified example given above is not syntactically valid.

An element from the XProc namespace may have any attribute not from the XProc namespace, provided that the expanded-QName of the attribute has a non-null namespace URI. These attributes are called extension attributes. The presence of an extension attribute must not cause the flow graph to differ from the flow graph that any other conformant XProc processor might produce. They must not cause the processor to fail to signal an error that a conformant processor is required to signal. This means that an extension attribute must not change the effect of any XProc element except to the extent that the effect is implementation-defined or implementation-dependent. A processor which encounters an extension attribute that it does not recognize must behave as if the attribute was not present.

Outside the context of a here document, any element not in the XProc namespace is an extension element. The presence of an extension element must not cause the flow graph to differ from the flow graph that any other conformant XProc processor might produce. They must not cause the processor to fail to signal an error that a conformant processor is required to signal. This means that an extension element must not change the effect of any XProc element except to the extent that the effect is implementation-defined or implementation-dependent. Inside the context of a here document, all content is considered quoted so neither extension elements nor XProc elements are said to occur. A processor which encounters an extension element that it does not recognize must behave as if neither the element nor its attributes, nor any of its content was present.

This section describes the XML vocabulary that instantiates a pipeline.

A p:pipeline represents a [[pipeline]]. Its children declare the inputs, outputs, and parameters that the pipeline exposes and the subpipeline that constitutes its definition. If specified, the name must be unique across all available pipelines. If a p:pipeline occurs as the child of a p:pipeline-library element, it must be named. … ]]>

A p:declare-input declares an input port. The port attribute defines the name of the port. It is a static error two define two ports with the same name. An input declaration can indicate if a sequence of documents is allowed to appear on the declared port. If sequence is specified with the value “yes”, then a sequence is allowed. If the sequence is not specified, or has the value “no”, then it is a dynamic error for a sequence of more than one document to appear on the declared port. The declaration may be accompanied by a binding (or default binding) for the input. This binding can be accomplished by source: by URI: or by here document: If a by source binding is used, the port selected must be an output port on a step which is not a descendant of the step on which the p:declare-input appears or it must be a port declared with p:declare-input on some ancestor (e.g., an enclosing p:choose or p:for-each) of the step. If a binding is provided, a select expression may also be provided. If provided, the specified XPath select expression is used to filter the document(s) that are read. Each matching node or set of nodes is wrapped in a document and provided to the input port. See .

A p:declare-parameter declares a parameter. Components must declare the parameters that they accept. The name attribute must be a single asterisk (*), a QName, or a string of the form *:NCName or NCName:*. If name is a QName, then the declaration may include a default value using either content or a select attribute as per p:param.

A p:declare-output identifies an output port. The port attribute defines the name of the port. It is a static error to declare two ports with the same name. An output declaration can indicate if a sequence of documents is allowed to appear on the declared port. If sequence is specified with the value “yes”, then a sequence is allowed. If the sequence is not specified, or has the value “no”, then it is a dynamic error if the component produces a sequence of more than one document on the declared port. On a step container, the declaration must be accompanied by a binding for the output. This binding can be accomplished by source: by URI: or by here document: If a by source binding is used, the port selected must be an output port on an step which is a descendant of the step on which the p:declare-output appears.

A p:step represents a component in a pipeline. The component attribute identifies the component to be instantiated. It is a static error if the name is not unique in the current scope, if the specified component is not known to the processor, or if the specified inputs, outputs, and parameters do not match the component's signature.

A p:input identifies input for a component. Inputs identify their source in exactly one of three ways, by source: by URI: or by here document: It is a static error if more than one of source, uri or here document is specified. The port attribute identifies the input port of the component that will read from the specified source. It is a static error if the name given does not match the name of an input port for the component or if more than one input is specified for any given port. The select expression, if specified, applies the specified XPath select expression to the document(s) that are read. Each matching node or set of nodes is wrapped in a document and provided to the input port. In other words, ]]> provides a single document, but ]]> provides a sequence of zero or more documents, one for each matching html:div in http://example.org/input.html. ]]> provides a sequence of zero or more documents, one for each matching html:div in the document (or each of the documents) that is read from the portname port of the step named stepname.

A p:param associates a particular value with a parameter. The value of the parameter can be specified in several ways; as the content of the p:param element: In this case, the (XPath 1.0) string value of the content becomes the value of the parameter. With a value attribute: Or as an XPath expression. In this case, the context in which the expression is evaulated must be specified in exactly one of three ways, by source: by URI: or by here document: The (XPath 1.0) string value of the result of evaluating the expression becomes the value of the parameter.

An p:import-param provides a set of in-scope parameters to a component. All in-scope parameters which match the name are made available to the component as if they had been specified with individual p:param elements. The name attribute must be a single asterisk (*), a QName, or a string of the form *:NCName or NCName:*.

A p:for-each represents a [[for-each]]. Exactly one input must be declared and it must include a binding for the port it declares. If outputs are declared, they must also include a binding. The processor will provide each document read through that binding to the subpipeline that the p:for-each contains, one at a time. For each declared output, the processor will collect all the documents that are produced for that output from all the iterations, in order, into a sequence. The result of the p:for-each is that set of document sequences. shows an example of a p:for-each in action. ]]> The //chapters of the DocBook document are selected. Each chapter is transformed into HTML and XSL FO using an XSLT step. The resulting HTML and FO documents are aggregated together and appear on the html and fo ports, respectively, of the chapters step. It is a static error if there is not exactly one p:declare-input child of p:for-each, if the declared input does not specify a binding, or if it specifies a binding to a step inside the p:for-each. It is a static error if any declared output does not specify a binding.

A p:viewport represents a [[viewport]]. Exactly one input must be declared and it must include both a binding and a select expression. Exactly one output must be declared and it must include a binding. The processor will provide a document that contains each set of nodes that matches the specified select expression through the input binding to the subpipeline that the p:viewport contains, one at a time. What appears on the output from the p:viewport will be a copy of the input document except that where each matching node or set of nodes appears, the result of applying the subpipeline to those nodes will be output. It is a dynamic error if the input source is a sequence of more than one document or if the output from any iteration is a sequence of more than one document. shows an example of a p:for-each in action. ]]> The //h:div[@class='enc']s of the document are selected. Each selected div is encrypted and the resulting encrypted version replaces the original div. The result of the step is a copy of the input document with each selected div encrypted. It is a static error if there is not exactly one p:declare-input child and exactly one p:declare-output child of p:viewport, if the declared ports do not specify a binding, or if the input port specifies a binding to a step inside the p:viewport.

A p:choose represents a [[choose]]. Where p:when specifies a conditional branch. And p:otherwise specifies the default branch. The test expression is evaluated for each p:when in turn. The first p:when for which the expression evaluates to “true” is selected, all other p:when elements (and the p:otherwise) are ignored. If no p:when has a test expression which evaluates to “true”, then the p:otherwise is selected and all p:when elements are ignored. The context for each test expression is the document specified by the source attribute on the corresponding p:when. If no source attribute is specified on the p:when, then the context is the document specified by the source attribute on the p:choose. It is a static error if no source is specified in either place. All of the p:when branches and the p:otherwise must declare the same number of output ports with the same names. It is a static error if they do not. The result of the p:choose is the result of the selected branch. It is a dynamic error if no p:when is selected and no p:otherwise is specified.

A p:group is a wrapper for a subpipeline. The result of a p:group is its declared outputs.

A p:try represents a [[try/catch]]. Where p:group is any pipeline group and p:catch surrounds the error recovery behavior: The p:try component evaluates the p:group. If the group evaluates without signaling an error, that is the result of the p:try component. However, if any component in that group signals an error, then the group is abandoned (any accumulated output is discarded) and the p:catch subpipeline is evaluated. In that case, the result of the p:catch is the result of the p:try component. If any component in the p:catch subpipeline signals an error, the p:try fails. Within the p:catch block, the special input port !#error is defined. The document(s) on that port constitute the error messages received from the component which failed. Note that the order of the messages on that port is undefined. Note also that the failure of one component may cause others to fail and the component which signaled the error may not be the only or even the first component that failed. Both the p:group and the p:catch must declare the same number of output ports with the same names. It is a static error if they do not.

A p:declare-component provides the name and signature of an implementation-dependent component. It declares the inputs, outputs, and parameters of the component. We need to make some provision for identifying an external binding, even if it's implementation defined. We'll need some sort of mechanism for declaring multiple implementations too. It is a static error if a pipeline step refers to a component that is not recognized by the processor. It is not an error to declare such a component, only to use it. Exactly one input declaration of a p:declare-component may use the name “*” to indicate that the component accepts an arbitrary number of inputs.

A p:pipeline-library contains one or more component declarations and/or pipelines. It declares components that pipelines can import. … … … … ]]>

An p:import loads a pipeline or pipeline library, making it available in the current context. An import statement loads the specified URI and makes any pipelines declared within it available to the current pipeline. It is a dynamic error if the URI cannot be retrieved or if, once retrieved, it does not point to a p:pipeline-library or p:pipeline. If it points to a p:pipeline, it is a dynamic error if the pipeline does not have a name.

Errors in a pipeline can be divided into two classes: static errors and dynamic errors.

A static error is one which can be detected before pipeline evaluation is even attempted. Examples of static errors include cycles, incorrect specification of inputs and outputs, and reference to unknown components. Static errors are fatal and must be detected before any components are evaluated.

A A dynamic error is one which occurs while a pipeline is being evaluated. Examples of dynamic errors include references to URIs that cannot be resolved, components which fail, and pipelines that exhaust the capacity of an implementation (such as memory or disk space). If a component fails due to a dynamic error, failure propogates upwards until either a [[try]] is encountered or the entire pipeline fails. In other words, outside of a [[try]], component failure causes the entire pipeline to fail.

XML Core Req XML Processing Model Requirements Dmitry Lenkov, Norman Walsh, editors. W3C Working Group Note 05 April 2004 Infoset XML Information Set (Second Edition) John Cowan, Richard Tobin, editors. W3C Working Group Note 04 February 2004. XML 1.0 Extensible Markup Language (XML) 1.0 (Fourth Edition) Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, et. al. editors. W3C Recommendation 16 August 2006. XML 1.1 Extensible Markup Language (XML) 1.1 (Second Edition) Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, et. al. editors. W3C Recommendation 16 August 2006. This appendix describes the XML vocabulary that components are expected to use to identify messages on their error ports. T.B.D. This appendix describes the standard components that must be supported by any conforming processor. T.B.D.