xproc 2006-08-19 XML Norman Walsh Sun Microsystems, Inc. Norman.Walsh@Sun.COM This document is a shell where ideas, points of consensus, and early draft text is being collected. It does not necessarily represent the consensus of the Working Group. This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/. More boilerplate goes here…

An XML Pipeline describes a sequence of operations to be performed on a collection of input documents. Pipelines take zero or more XML documents as their input and produce zero or more XML documents as their output. Components in the pipeline may read or write non-XML resources as well. Each operation in a pipeline is performed by a component. Like pipelines, components take zero or more XML documents as their input and produce zero or more XML documents as their output. The inputs to a component come from the web, from the pipeline document, from the inputs to the pipeline itself, or from the outputs of other components in the pipeline. The outputs from a component are either consumed by other components or are outputs of the pipeline as a whole. (Outputs may also be ignored.) This specification defines a standard component library, . Pipeline implementations may support additional components as well. is a graphical representation of a simple pipeline that performs XInclude processing and validation on a document.

A simple, linear XInclude/Validate pipeline

This is a pipeline that consists of two components, XInclude and Validate. The pipeline itself has two inputs, “document” and “schema”. How these inputs are connected to XML documents is implementation-defined. The XInclude component reads the pipeline input “Document” and produces a result document. The Validate component reads the pipeline input “Schema” and the output from the XInclude component and produces a result document. The result of the validation is the result of the pipeline, “Result Document”. How pipeline outputs are connected to XML documents is implementation-defined. is a more complex example.

A transform and serialize pipeline

The heart of this example is the conditional. The standard “choose” component evaluates an XPath expression over a test document. If the effective boolean value of the expression is true, then that branch of the pipeline is evaluated. If no expressions are true, then the “otherwise” branch is evaluated. Note that the “Serialize” and “FO to PDF” components produce no output, consequently this pipeline produces no output. This pipeline transforms the input document with the input stylesheet and, if the result is an XSL-FO document, generates PDF. If the result is not XSL-FO, the pipeline assumes that it is XHTML and runs Tidy over it before serializing it.

A pipeline is an acyclic, directed graph of components connected together by inputs and outputs. A pipeline is itself a component and must satisfy the constraints on components. The result of evaluating a pipeline is the result of evaluating the components that it contains. A pipeline must behave as if it evaluated each component each time it occurs. Unless otherwise indicated, implementations must not assume that components are functional (that is, that their output depends only on their explicit inputs and parameters). A subpipeline is any collection of connected components. The distinction between pipelines and subpipelines is simply that a pipeline is a component and can stand alone. “Subpipeline” is just a convenient name for a user-specified collection of components that occurs inside another component and has no independent identity.

Components are the basic computational units of a pipeline. A component is a unit of XML processing, such as XInclude or transformation. Components may perform arbitrary amounts of computation but they are indivisible from the point of view of the pipeline that instantiates them. Components have “ports” into which inputs and outputs are connected. Each component has a number of input ports and a number of output ports, all with unique names. A component may have zero input ports and/or zero output ports. (All components have a standard port for reporting errors that does not have to be, and cannot be, declared.) Components may have an arbitrary number of parameters. All of the input ports of a component must be connected to inputs. It is a static error if a component has an input port which is not connected. Unconnected output ports are allowed, any documents produced on those ports are simply discarded. The signature of a component is the set of inputs, outputs, and parameters that it is declared to accept. The instantiation of a component matches its signature if and only if it specifies an input for each declared input and it specifies no inputs that are not declared, it specifies no outputs that are not declared, it specifies a parameter for each parameter that is declared to be required, and it specifies no parameters that are not declared. In other words, every input and required parameter must be specified and only inputs, outputs, and parameters that are declared may be specified. Outputs and optional parameters do not have to be specified.

Although components are free to read and write non-XML resources, what flows between components as inputs and outputs are exclusively XML documents or sequences of XML documents. Each XML document (or document in a sequence) must be a well formed or document. The inputs and outputs may be implemented as sequences of characters, events, or object models, or any other representation the implementation chooses. Is support for XML 1.1 optional? It is a dynamic error if a non-XML resource is produced on a component output or arrives on a component input. What about the cases where it's impractical to test for this error? An implementation may make it possible for a component to produce non-XML output, as the final components in demonstrate, but those results cannot flow through the pipeline. Similarly, one can imagine a component that takes no inputs, reads a non-XML file from a URI, and produces an XML output. But the non-XML file cannot be an input to the component or pipeline.

A parameter is a QName/value pair. The value of a parameter must be a string. If a document, node, or other value is given, its string value is computed and that string is used.

The components of a pipeline are the nodes of a component graph. The inputs and outputs of the components are the arcs of that graph. Consider two components in such a graph, “component A” and “component B”. Components A and B are connected if any output from one is connected to any input of the other, either directly or indirectly. With respect to connected components, we can speak of one component being either before or after another. Component A is before component B if component B is a subpipeline of component A, either directly or indirectly, or if any output from component A is connected to any input of component B, either directly or indirectly. after is the converse of before. It is static error if a component is either before or after itself. In other words, the component graph must be acyclic.

This section describes the core language constructs of XProc.

A [[pipeline]] is a user-defined pipeline component. It has a number of declared input and output ports and parameters. Viewed from the outside, it is a black box which performs some calculation on the inputs and produces the outputs. From the pipeline author's perspective, the computation performed by the pipeline is described in terms of a subpipeline which reads the pipeline's inputs and produce the pipeline's outputs. For example, a pipeline might accept a document and a stylesheet as input; perform XInclude, validation, and transformation over its inputs; and produce a sequence of formatted documents as its output.

A [[for-each]] component processes a sequence of documents, applying a subpipeline to each individual document in turn. The result of the [[for-each]] is the aggregation of the results produced by processing each individual document. If the [[for-each]] subpipeline declares multiple outputs, each output is the aggregation of the results produced on that output by each iteration. For example, a [[for-each]] might accept a sequence of DocBook chapters as its input, process each chapter in turn with XSLT, and produce a sequence of formatted chapters as its output.

A [[viewport]] component processes a single document, applying a subpipeline to one or more subsections of the document. The result of the [[viewport]] is a copy of the original document with the selected subsections replaced by the results of applying the subpipeline to them. For example, a [[viewport]] might accept an XHTML document as its input, apply encryption to selected div elements within that document, and return an XHTML document that is the same as the original except that each selected div has been replaced by its encrypted result.

A [[choose]] component selects exactly one of a set of possible subpipelines based on the evaluation of XPath expressions. If no expressions evaluate to “true”, a default subpipeline is selected. After a subpipeline is selected, it is evaluated as if only it had been present. The result of the [[choose]] is the result of the selected subpipeline. For example, a [[choose]] might test a schema and apply XML Schema validation to an input document if the schema is an XML Schema document, apply RELAX NG validation if the schema is a RELAX NG grammar, or perform no validation otherwise. Each subpipeline is associated with a separate XPath expression that is evaluated in the context of a document. The context document can be different for different XPath expressions. In order to ensure that the result of the [[choose]] is consistent irrespective of the subpipeline chosen, each subpipeline must declare the same number of outputs with the same names. It is a static error if two subpipelines in a choose declare different outputs. It is a dynamic error if no subpipeline is selected by the [[choose]].

A [[try]] component isolates a subpipeline, preventing any errors that arise within it from being exposed to the rest of the pipeline. A [[try]] component begins with two subpipelines: an initial subpipeline and a [[catch]] subpipeline. It evaluates the initial subpipeline and, if no errors occur, the results of that pipeline are the results of the component. However, if any errors occur, it abandons the first subpipeline, discarding any output that it may have generated, and evaluates the [[catch]] subpipeline. In this case, the results of the [[catch]] subpipeline are the results of the [[try]] component. For example, a pipeline might attempt to process a document by dispatching it to some web service. If the web service succeeds, then those results are passed to the rest of the pipeline. However, if the web service cannot be contacted or reports an error, the [[catch]] component can provide some sort of default for the rest of the pipeline. In order to ensure that the result of the [[try]] is consistent irrespective of whether the [[try]] subpipeline provides its output or the [[catch]] subpipeline does, both the [[try]] and [[catch]] subpipeline must declare the same number of outputs with the same names. It is a static error if the two subpipelines declare different outputs. In order to support corrective action in the [[catch]] subpipeline, components inside the [[catch]] have access to the (aggregation of the) error output of the components that were in the [[try]] subpipeline. In evaluating the [[try]] subpipeline, failure of one component may cause other components to fail. In addition, some components that fail may not produce output on their error ports and some components that succeeded may produce such output. This pipeline language places no constraints on the order of error messages provided to the [[catch]] subpipeline, nor does it attempt to gaurantee that such output will be avialable in all cases. The error documents that appear should conform to .

A pipeline document may declare additional components. These may be implementation-defined components or may be defined through some implementation-dependent extension mechanism. Each declared component must have a name and a signature. It is a static error if a pipeline refers to a component that is not recognized by the processor.

This section describes the syntactic elements necessary to instantiate a pipeline.

At the highest level, a pipeline is a collection of steps. Each step has a unique name and instantiates a particular component. The component has a signature and that signature has to be satisified by the inputs and parameters specified on the step. Recall that a pipeline is not required to consume all the outputs of a component, but it must identify all the inputs and required parameters. Components that contain subpipelines are represented naturally using XML hierarchy by placing steps inside other steps.

A step can bind a document or a sequence of documents to a port in three ways: by source, by URI, or by providing it “here”, as the content of the element establishing the binding. A document must be specified in exactly one of these ways. Specified by URI A document is specified by URI if it refers to it with a URI. The href attribute is used for this purpose. In this example, the input to the Identity step named “otherstep” comes from “http://example.com/input.xml”. ]]> It is a dynamic error if the processor attempts to retrieve the specified URI and fails. (For example, if the resource does not exist or is not accessible with the users authentication credentials.) Specified by source A document is specified by source if it refers to a specific port on another step. The source attribute is used for this purpose. The specified port must either be declared on some ancestor or it must be an output port of some other step. In some contexts there are additional constraints on the step that can be selected, for example, that it must be (or must not be) a descendant of the step on which the binding occurs. A source specification identifies a specific port with the name of the step on which that port occurs and the name of the port exposed by the component that that step instantiates. Syntactically, this is achieved with a compound name of the form “stepname!portname”. In this example, the input to the XInclude step named “expand” comes from the “result” port of the step named “otherstep”. ]]> The output port of the XInclude component is named “result”, so if another step wishes to take the output of the “expand” step as one of its inputs, it can refer to it with the compound name “expand!result”. As a special case, if the “stepname” is absent, the declared ports of any ancestor steps are considered. The corresponding port of the first such ancestor that declares a port with the specified name is selected. A source specification identifies a source if the specified port is either a declared input on some ancestor or an output of some other step. A source specification identifies a sink if the specified port is a declared output on some ancestor or an input of some other step. It is a static error if the specified port does not exist. Specified by here document An document is specified by here document if it is contained in the body of the element that binds it. In this example, the stylesheet input to the XSLT step named “xform” comes from the content of the input element. ... ]]> Here documents are considered “quoted”, they are not interpolated or available to the pipeline processor in any way except as a document flowing through the pipeline.

The scope of a port name is the component on which it is defined. The names of all input and output ports on any component must be unique. The scope of step names is the component graph. Should we say that components have names and remove steps from the equation all together?

In order for user-defined components (such as p:for-each and p:choose) to function as “black boxes” the way other components do, it is necessary to make sure that they are wholly self-contained: ]]> However, in practice, forcing authors to declare all of the inputs and parameters to each component is tedious and error-prone. What's more, the pipeline processor can actually determine what additional declarations are necessary simply by examining the source declarations on the steps contained within the step. Therefore, we allow the syntactic shortcut of simply referring to in-scope sources and parameters directly. ]]> These shortcuts are purely syntactic. The processor is required to behave as if appropriate, unique declarations with the corresponding remapping of names had been present.

This section describes the XML vocabulary that instantiates a pipeline.

A p:pipeline instantiates a [[pipeline]]. It declares the inputs, outputs, and parameters that the pipeline exposes and contains the subpipeline that constitutes its definition. If specified, the name must be unique across all available pipelines. If a p:pipeline occurs as the child of a p:pipeline-library element, it must be named. … ]]>

A p:declare-input declares an input port. The preceding syntax summary doesn't show any of the allowed binding attributes. The port attribute defines the name of the port. It is a static error two define two ports with the same name. The declaration may be accompanied by a binding (or default binding) for the input. This binding can be accomplished by source, by URI, or by here document. If a by source binding is used, the port selected must be an output port on an step which is not a descendant of the step on which the p:declare-input appears or it must be a port declared with p:declare-output on some ancestor of the step. The select expression, if specified, applies the specified XPath select expression to the document(s) that are read. Each matching node or set of nodes is wrapped in a document and provided to the input port. See . An input declaration can indicate if a sequence of documents is allowed to appear on the declared port. If sequence is specified with the value “yes”, then a sequence is allowed. If the sequence is not specified, or has the value “no”, then it is a dynamic error for a sequence of more than one document to appear on the declared port.

A p:declare-output identifies an output port. The preceding syntax summary doesn't show any of the allowed binding attributes. The port attribute defines the name of the port. It is a static error to declare two ports with the same name. The declaration must be accompanied by a binding for the output. This binding can be accomplished by source, by URI, or by here document. If a by source binding is used, the port selected must be an output port on an step which is a descendant of the step on which the p:declare-output appears. An output declaration can indicate if a sequence of documents is allowed to appear on the declared port. If sequence is specified with the value “yes”, then a sequence is allowed. If the sequence is not specified, or has the value “no”, then it is a dynamic error if the component produces a sequence of more than one document on the declared port.

A p:step instantiates a component in a pipeline. The component attribute identifies the component to be instantiated. It is a static error if the name is not unique in the current scope, if the specified component is not known to the processor, or if the specified inputs, outputs, and parameters do not match the component's signature.

A p:input identifies input for a component. Inputs identify their source in exactly one of three ways, by source: by URI: or by here document: It is a static error if more than one of source, uri or here document is specified. The port attribute identifies the input port of the component that will read from the specified source. It is a static error if the name given does not match the name of an input port for the component or if more than one input is specified for any given port. The select expression, if specified, applies the specified XPath select expression to the document(s) that are read. Each matching node or set of nodes is wrapped in a document and provided to the input port. In other words, ]]> provides a single document, but ]]> provides a sequence of zero or more documents, one for each matching html:div in http://example.org/input.html. ]]> provides a sequence of zero or more documents, one for each matching html:div in the document (or each of the documents) that is read from the portname port of the step named stepname.

A p:param associates a particular value with a parameter. The specified XPath expression is evaluated and its string value becomes the value of the parameter. The value of the parameter may also be specified as the content of the p:param element. The string value of the content becomes the value of the parameter.

An p:import-param provides a set of in-scope parameters to a component. All in-scope parameters which match the name are made available to the component as if they had been specified with individual p:param elements. The name attribute must be a single asterisk (*), a QName, or a string of the form *:NCName or NCName:*.

A p:for-each instantiates a [[for-each]]. Exactly one input must be declared and it must include a binding for the port it declares. If outputs are declared, they must also include a binding. The processor will provide each document read through that binding to the subpipeline that the p:for-each contains, one at a time. For each iteration, the processor will collect any outputs that appear on the declared outputs and aggregate them together. The result of the p:for-each is that set of aggregated outputs. shows an example of a p:for-each in action. ]]> The //chapters of the DocBook document are selected. Each chapter is transformed into HTML and XSL FO using an XSLT step. The resulting HTML and FO documents are aggregated together and appear on the html and fo ports, respectively, of the chapters step. It is a static error if there is not exactly one p:declare-input child of p:for-each, if the declared input does not specify a binding, or if it specifies a binding to a step inside the p:for-each. It is a static error if any declared output does not specify a binding.

A p:viewport instantiates a [[viewport]]. Exactly one input must be declared and it must include both a binding and a select expression. Exactly one output must be declared and it must include a binding. The processor will provide a document that contains each set of nodes that matches the specified select expression through the input binding to the subpipeline that the p:viewport contains, one at a time. What appears on the output from the p:viewport will be a copy of the input document except that where each matching node or set of nodes appears, the result of applying the subpipeline to those nodes will be output. It is a dynamic error if the input source is a sequence of more than one document or if the output from any iteration is a sequence of more than one document. shows an example of a p:for-each in action. ]]> The //h:div[@class='enc']s of the document are selected. Each selected div is encrypted and the resulting encrypted version replaces the original div. The result of the step is a copy of the input document with each selected div encrypted. It is a static error if there is not exactly one p:declare-input child and exactly one p:declare-output child of p:viewport, if the declared ports do not specify a binding, or if the input port specifies a binding to a step inside the p:viewport.

A p:choose instantiates a [[choose]]. Where p:when specifies a conditional branch. And p:otherwise specifies the default branch. The test expression is evaluated for each p:when in turn. The first p:when for which the expression evaluates to “true” is selected, all other p:when elements (and the p:otherwise) are ignored. If no p:when has a test expression which evaluates to “true”, then the p:otherwise is selected and all p:when elements are ignored. The context for each test expression is the document specified by the source attribute on the corresponding p:when. If no source attribute is specified on the p:when, then the context is the document specified by the source attribute on the p:choose. It is a static error if no source is specified in either place. All of the p:when branches and the p:otherwise must declare the same number of output ports with the same names. It is a static error if they do not. The result of the p:choose is the result of the selected branch. It is a dynamic error if no p:when is selected and no p:otherwise is specified.

A p:group is a wrapper for a subpipeline. The result of a p:group is its declared outputs.

A p:try instantiates a [[try/catch]]. Where p:group is any pipeline group and p:catch surrounds the error recovery behavior: The p:try component evaluates the p:group. If the group evaluates without signaling an error, that is the result of the p:try component. However, if any component in that group signals an error, then the group is abandoned (any accumulated output is discarded) and the p:catch subpipeline is evaluated. In that case, the result of the p:catch is the result of the p:try component. If any component in the p:catch subpipeline signals an error, the p:try fails. Within the p:catch block, the special input port !#error is defined. The document(s) on that port constitute the error messages received from the component which failed. Note that the order of the messages on that port is undefined. Note also that the failure of one component may cause others to fail and the component which signaled the error may not be the only or even the first component that failed. Both the p:group and the p:catch must declare the same number of output ports with the same names. It is a static error if they do not.

A p:declare-component provides the name and signature of an implementation-dependent component. It declares the inputs, outputs, and parameters of the component. We need to make some provision for identifying an external binding, even if it's implementation defined. We'll need some sort of mechanism for declaring multiple implementations too. It is a static error if a pipeline step refers to a component that is not recognized by the processor. It is not an error to declare such a component, only to use it. The input and parameter declarations of a p:declare-component may use the name “*” to indicate that the component accepts an arbitrary number of inputs, outputs, or parameters.

A p:pipeline-library contains one or more component declarations and/or pipelines. It declares components that pipelines can import. … … … … ]]>

An p:import loads a pipeline or pipeline library, making it available in the current context. An import statement loads the specified URI and makes any pipelines declared within it available to the current pipeline. It is a dynamic error if the URI cannot be retrieved or if, once retrieved, it does not point to a p:pipeline-library or p:pipeline. If it points to a p:pipeline, it is a dynamic error if the pipeline does not have a name.

Errors in a pipeline can be divided into two classes: static errors and dynamic errors.

A static error is one which can be detected before pipeline evaluation is even attempted. Examples of static errors include cycles, incorrect specification of inputs and outputs, and reference to unknown components. Static errors are fatal and must be detected before any components are evaluated.

A A dynamic error is one which occurs while a pipeline is being evaluated. Examples of dynamic errors include references to URIs that cannot be resolved, components which fail, and pipelines that exhaust the capacity of an implementation (such as memory or disk space). If a component fails due to a dynamic error, failure propogates upwards until either a [[try]] is encountered or the entire pipeline fails. In other words, outside of a [[try]], component failure causes the entire pipeline to fail.

XML Core Req XML Processing Model Requirements Dmitry Lenkov, Norman Walsh, editors. W3C Working Group Note 05 April 2004 Infoset XML Information Set (Second Edition) John Cowan, Richard Tobin, editors. W3C Working Group Note 04 February 2004. XML 1.0 Extensible Markup Language (XML) 1.0 (Fourth Edition) Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, et. al. editors. W3C Recommendation 16 August 2006. XML 1.1 Extensible Markup Language (XML) 1.1 (Second Edition) Tim Bray, Jean Paoli, C. M. Sperberg-McQueen, et. al. editors. W3C Recommendation 16 August 2006. This appendix describes the XML vocabulary that components are expected to use to identify messages on their error ports. T.B.D. This appendix describes the standard components that must be supported by any conforming processor. T.B.D.