[Editorial Draft] Versioning XML Languages

1.1 Terminology

Extensibility is a property that enables evolvability of software. It is perhaps the biggest contributor to loose coupling in systems as it enables the independent and potentially compatible evolution of languages. Languages are defined to be [Definition: Extensible if instances of the language can include terms from other vocabularies. ].

An XML Namespace is a convenient container for collecting terms that are intended to be used together within a language or across languages. It provides a mechanism for creating globally unique names.

A language has a vocabulary that may be drawn from one or more XML Namespaces (or none). [Definition: A vocabulary is a set of terms]The syntactic structure of the language is constrained by the use of DTDs, XML Schema, other schema languages or narrative constraints expressed in the relevant language specification.

In general, the intended meaning of a vocabulary term is scoped by the language in which the term is found. However, there is some expectation that terms drawn from an XML Namespace have a consistent meaning across all languages in which they are used.

For our purposes, [Definition: a language is an identifiable set of vocabulary terms that has defined constraints.] For example, the elements and attributes of XHTML 1.0 or the names of built-in functions in XPath 2.0. Languages may or may not be defined by a schema in any any particular schema language. By language, we just mean the set of elements and attributes, or components, used by a particular application.

[Definition: An instance is a realization of a language]. Documents are instances of a language. They must have a root element in XML.

[Definition: Content is data that is part of an instance of a language.] Content is part of a document. Content has one or more components.

[Definition: a Component is a realization of a term in a language.] XML elements and attributes are components.

In this finding, we describe interaction between applications and languages in terms of senders and receivers. [Definition: A sender creates or produces an instance and sends it to another application for processing.] [Definition: A receiver consumes an instance that it obtained from a sender.]

These terms and their relationships are shown below

In the examples mentioned above, the stylesheet processor is a receiver of the XML document that it is processing (the sender isn't mentioned); in the Web services context the roles of sender and receiver may alternate as messages are passed back and forth.

Note that most Web service specifications provide definitions of inputs and outputs. By our definitions of compatibility, a Web service that updates its output schema is considered a new sender. A service that updates its input schema is a new receiver.

Versioning is an issue that effects almost all XML applications eventually. Whether it's a processor styling documents in batch to produce PDF files or Web services engaged in financial transactions, applications may receive versions of a language that they aren't expecting.

As languages evolve, it becomes possible to speak of both backwards backwards and forwards compatability. We base our definitions of backwards and forwards compatibility on the [FOLDOC] definitions. There are two aspects of compatibility that are called out: software compatibility and schema compatibility. While it is often the case that they are directly related, sometimes they are not.

[Definition: A language change is backwards compatible if newer processors can process all instances of the old language. ] A softwareAn example is a word processorprocessing software at version 5 being able to read and process version 4 documents. A schema example is a schema at version 5 being able to validate version 4 documents. In the case of Web services, this means that new Web services receivers, ones designed for the new version, will be able to process all instances of the old language. This means that a sender can send an old version of a message to a receiver that understands the new version and still have the message successfully processed.

[Definition: A language change is forwards compatible if older processors can process all instances of the newer language.] An example is a word processing software at version 4 being able to read and process version 5 documents. A schema example is a schema at version 4 being able to validate version 5 documents. In the case of Web services, this means that existing Web service receivers, designed for a previous version of the language, will be able to process all instances of the new language. This means that a sender can send a newer version of a message to an existing receiver and still have the message successfully processed.

In broad terms, backwards compatibility means that newer senders can continue to use existing services, and forwards compatibility means that existing senders can use newer services

The cost of changes that are not backward or forward compatible is often very high. All the software that uses the language must be updated to the newer version. The magnitude of that cost is directly related to whether the system in question is open or closed.

[Definition: A closed system is one in which all of the senders and receivers are more-or-less tightly connected and under the control of a single organization.] Closed systems can often provide integrity constraints across the entire system. A traditional database is a good example of a closed system: all of the database schemas are known at once, all of the tables are known to conform to the appropriate schema, and all of the elements in the each row are known to be valid for the schema to which the table conforms.

From a versioning perspective, it might be practical in a closed system to say that a new version of a particular language is being introduced into the system at such and such a time and all of the data that conforms to the previous version of the schema will be migrated to the new schema.

[Definition: An open system is one in which some senders and receivers are loosely connected or are not controlled by the same organization. The internet is a good example of an open system.]

In an open system, it's simply not practical to handle language evolution with universal, simultaneous, atomic upgrades to all of the software components. Existing senders and recievers outside the immediate control of the organization that's publishing a changed language will continue to use the previous version for some (possibly long) period of time.

Finally, it's important to remember that systemsentire projects evolve over time and have different requirements at different stages in their life cycle. During development, when the first version of a language is under active development, it may be valuable to persue a much more aggressive, draconian versioning strategy. After a system is in production and there is an expectation of stability in the language, it may be necessary to proceed with more caution. Being prepared to move forward in a backwards and forwards compatible manner is the strongest argument for worrying about versioning at the very beginning of a project.

1.2 Why Worry About Extensibility and Versioning?

As documents, or messages, are exchanged between applications, they are processed. Most applications are designed to discriminate between valid and invalid inputs. In order to have any sort of interoperability, a language must be defined or described in some normative way so that the terms "invalid" and "valid" have meaning.

There are a variety of tools that might be employed for this purpose (DTDs, W3C XML Schema, RELAX NG, Schematron, etc.). These tools might be augmented with normative prose documentation or even some application-specific validation logic. In many cases, the schema language is the only validation logic that is available.application.

It is almost unheard of for a single version of a language to be deployed without requiring some kind of augmentation. Invariably, the original language designer did not include certain terms and constraints. In fact, good designers should not try to define all the possible terms and constraints. This is sometimes called "boiling the ocean". Knowing that a language will not be all things to all people, a language designer can allow parties to extend instances of the language or the language itself. Typically the tools will allow the language designer to specify where extensions in the instance and extensions in the language are allowed. Of note, we do not call extending an instances of a language a new version. This scopes our discussion of versioning to changes in a language, not changes to instaces.

Whether you've deployed ten services, or a hundred, or a million, if you change a language in such a way that all those services will consider instances of the new language invalid, you've introduced a versioning problem with real costs.

Once a language is used outside of its development environment, there will be some cost associated with changing it: software, user expectations,will have to andchange, or some users documentation may have to bechange, or some documentation will updated to accomodate the change. Once a language is used in environments outside of a single realmimmediate of control, any changes madethat you make will introduce multiple versions of the language.

1.3 Why Extend languages?

The primary motivation to allow instances of a language to be extended is to decentralize the task of designing, maintaining, and implementing extensions. It allows senders to change the instances without going through a centralized authority. It means that changes can occur at the sender or receiver without the language owner approving of them. Consider the effort that the HTML Working Group put into modularity of HTML. Without some decentralized process for extension, every single variant of HTML would have to be called something else or the HTML Working Group would have to agree to include it in the next revision of HTML.

Allow Extensibility: Languages designers SHOULD create extensible languages.

1.4 Why Do Languages Change?

There are many reasons why a different version of a language may be needed. A few of them include:

1. Bugs may need to be fixed. Production use may reveal defects or oversights that need to be fixed. This may involve changes to components of the language or changes to the semantics of existing components.

2. Changing requirements may motivate changes in the schema design. For example, a callback may be added to a service that performs some processing so that it is able to notify the caller when processing has completed.

3. Different flavors of a schema may be desirable. For example, the XHTML 1.0 Recommendation defines strict, transitional, and frameset schemas. All three of those schemas purport to define the same namespace, but they describe very different languages.

   And additional schemas may be defined by other specifications, such as the XHTML Basic Recommendation.

Whatever the cause, over time, different versions of the language exist and designing applications to deal with this change in a predictable, useful way requires a versioning strategy.

1.5 How Do Languages Change?

At the most basic level, languages can change in only a few ways:

• Elements: New elements can be added, existing elements can be removed, or the acceptable number of occurences of an element can change. In addition, the content of an element could change from element only content to mixed content, or vice versa.

  For elements with simple content, the type or range of values that are acceptable can change.

• Attributes: New attributes can be added, existing attributes can be removed, or the type or range of values that are acceptable can change.

• Semantics: The meaning of a an existing element or attribute can change.

Of course, the difference between two versions of a language can be an arbitrary number of these changes.

One of the most important aspects of a change is whether or not it is backwards or forwards compatible. For example:

• Adding new optional elements or attributes is both backwards and forwards compatible.

• Increasing the allowed number of occurrences of an element is backwards compatible. Similarly, extending the allowed range of an element with simple content or an attribute is a backwards compatible change.

• Decreasing the allowed number of occurrences of an element is forwards compatible. Decreasing the allowed range of an element or attribute is forwards compatible.

1.6 Kinds of Languages

Ultimately, there are different kinds of languages. The versioning approaches and strategies that are appropriate for one kind of language may not be appropriate for another. Among the various kinds of vocabulares, we find:

• Just Names: some languages don't actually identify elements or attributes, they're just lists of names. Using QNames to identify words in the WordNet database, for example, or the names of functions and operators in XPath2 are examples of "just name" languages.

• Standalone: languages designed to be used more-or-less by themselves, for example XHTML, DocBook, or The TEI.

• Containers: languages designed to be used as a wrapper or frameworktransport layer for some other language or payload, for example SOAP or WSDL.

• Container Extensions: languages designed to extend or augment a particular class of container. Specifications that extend SOAP by defining SOAP header blocks,headers, for example, to provide security, asynchrony or reliable messaging are examples of container extension languages.

  There are a couple types of extension languages, element extension and attribute extension.

  • Element Extension. Languages that are elements. SOAP, etc. are element extensions.

  • Attribute or type Extensions. Langages that are types or attributes. These languages must exist in the context of an element. Sometimes called "parasite" languages as they require a "host" element. XLink is an example.

• Mixtures: languages designed for, or often used for, encapsulating some semantics inside another language. For example, MathML might be mixed inside of another language.

This is by no means an exhaustive list. Nor are these categories completely clear cut. MathML can certainly be used standalone, for example, and languages like SVG are a combination of standalone, containers, and mixtures.

4.1 An Example

Throughout this finding, we'll motivate our discussion of versioning with an ongoing example. Suppose that you have designed a Web service for handling purchase orders. Processing begins when a purchase order is sent to the service. The order is processed and notification of shipment is sent back.

Because it can take an arbitrary amount of time to process an order before the shipped message can be sent (consider, for example, items on back order), it's useful to make this process asynchronous. To do this, the original sender includes a "call back" address where the shipped message should be delivered when it is ready.

Initially, the purchase order processor acts as a receiver. But when the shipment has been processed, it acts as a sender, delivering a message to the callback service that was provided.

Ultimately, we'll describe the evolution of this callback service in two ways: one that is backwards compatible and one that is not.

In order to be schema agnostic, we will use the examplotron [examplotron] schematypographic conventions to convey the semantics of our examples. Examplotronexamples: An uses examples to indicate the type and uses additional attributes to refine the content model. optional

Ironically, given the purpose of this finding, examplotron doesn't currently support extensibility points. Wepoint The are in discussion with Eric van der Vlist aboutan addingxsi:type this functionality. For the time being, we've added the [...] notation to indicate arbitrary extensibility of attributes and elements. attribute.

5.1 Defining an Extensible Callback

A number of examples illustrate the application of these and subsequent rules.

<wscb:Callback xmlns:wscb="http://example.com/callback/"
     xmlns:eg="http://examplotron.org/0/">
     ...
   <wscb:callbackLocation eg:content="xsd:anyURI" eg:occurs="."/>
   ...
</wscb:Callback>

AWe observe that a major motivation for using XML Namespaces is to allow extensions to a language to be madenamespaces by any party. The example above accomplishsaccomplished that goal. An extension can be defined by any new specification simply by normative reference to the original Callback specification and a definition for the new element or attribute content.element. No permission should be needed from the authors of the specification to make such an extension.

7.1 Backwards Incompatible Changes

There are a fewtwo distinct types of backwards incompatible change:

1. A required information item is added.

2. The semantics of an existing information item are changed.

The maximum number of allowable items is reduced. This change does not guarantee incompatibility. Instance documents where the maximum number allowable is still greater than or equal to the number of occurrences will still be vaild. If the maximum number of allowable items is reduced below the previous versions minimum number, then incompatibility is guaranteed.

Imagine that the semantics of callback location change; perhaps a conversation identifier is now required as part of the location. In this case, the must ignore rule won't help as ignoring the conversation identifier will result in invalid callbacks being dispatched.

Backwards incompatibility can easily be achieved by changing the namespace name or any of the element names. For example:

<wscb:Callback xmlns:wscb="http://example.com/conversationCallback/"
     xmlns:eg="http://examplotron.org/0/">
     
...

   <wscb:callbackLocation eg:content="xsd:anyURI" eg:occurs="."/>
   <wscb:conversationId eg:content="xsd:anyURI" eg:occurs="."/>
   
...

</wscb:Callback>

Notice that the namespace name has been changed from /callback to /conversationCallback. This new message type will be rejected as unrecognized by older systems. Deploying this callback request is an example of a "big bang" change.

7.2 Namespace content changes

Only the owner of a namespace can change (ie. version) the meaning of elements and attributes in that namespace.

Only Namespace Owners Change Namespace: The namespace name owner is the only entity that is allowed to change the meaning of names in a namespace.

There is a school of thought that says that every extension should be placed in a separate namespace; that after publication, no new names should be added to a language in the original namespace. If you hold that point of view then you may not feel that an extensibility element is necessary or desirable.

Another school of thought says that the maintainers of the language have a right to add new names to a namespace as they see fit. There are certain advantages associated with adding new names in the same namespace.

1. It reduces the number of namespaces needed to describe instances of the document. There are significant convenience advantages to using defaulted namespaces for document creation and manipulation.

2. It provides a clear separation between extensions by the language designers and extensions by third parties.

3. There may be additional benefits in code generation and reuse if single namespace or a small set of namespaces can completely describe the language.

A namespace name owner will use the lifecycle of the namespace as one of the factors in determining whether to revise the namespace or not. Typically, the changes during development are not compatible changes. The author of namespaces that are under development will typically followwant a "big bang" approach. This helps reduce the number of potentially buggy or immature implementations that are deployed. A W3C specification is a good illustrative example. It will probably change namespace names for each working draft. The Candidate Recommendation, Proposed Recommendation and Recommendation namespaces names should only be changed if compatibility is not achieved.

9.1 Identifying and Controlling Languages

Extensibility in XML Schema is commonly accomplished using the wildcard element, and the ##any namespace or a combination of the ##other namespace and the ##targetnamespacecurrent namespace in some designated "extensibility" element.

Using the callback example, our first attempt at designing an extensible schema for extensibility might look something like this:

<xs:complexType name="CallbackType">
  <xs:sequence>
    <xs:element name="callbackLocation" type="xs:anyURI"
                minOccurs="1" maxOccurs="1"/>
    <xs:any namespace="##any" processContents="lax"
                minOccurs="0" maxOccurs="unbounded"/>
  </xs:sequence>
  <xsl:anyAttribute/>
</xs:complexType>

At first, this seems to accomplish what we want: it allows a callbackLocation followed by any number of extension elements. But a problem arises when we want to extend this schema. Suppose, for example, that we want to add an optional expires element. Although we can send a expires to receivers using the old schema, we want a new schema that allows new receivers to validate the expires element explicitly. It appears that the following would work:

<xs:complexType name="CallbackType">
  <xs:sequence>
    <xs:element name="callbackLocation" type="xs:anyURI"
                minOccurs="1" maxOccurs="1"/>
    <xs:element name="expires" type="xs:integer"
                minOccurs="0" maxOccurs="1"/>
    <xs:any namespace="##any" processContents="lax"
                minOccurs="0" maxOccurs="unbounded"/>
  </xs:sequence>
  <xsl:anyAttribute/>
</xs:complexType>

But that schema fragment is invalid. There are constraints about namespace options and optional elements that are explained later in 9.3 Determinism. In order to create valid extension schemas that allow extension in the same namespace, we must create a container to hold them.

This brings us to the first rule for enabling a must ignore versioning strategy in XML languages defined by W3C XML Schema:

Any Namespace - XML Schema edition: The language MUST provide an extensibility element that allows for extension in the current namespace and a wildcard that allows for extension in another namespace.

For extensions in the same namespace, create an extension type. For example:

 <xs:element name="extension" id="extension">
   <xs:annotation>
     <xs:documentation>An extension container for elements in the current namespace.
     </xs:documentation>
   </xs:annotation>
   <xs:complexType mixed="true">
     <xs:sequence minOccurs="1" maxOccurs="unbounded">
       <xs:any processContents="lax" namespace="##targetnamespace"/>
     </xs:sequence>
     <xs:anyAttribute namespace="##any" processContents="lax"/>
   </xs:complexType>
 </xs:element>

This example is similar to the <xs:appinfo> element in W3C XML Schema itself, which can contain extensions.

For extensions in another namespace, use <xs:any namespace="##other"> at appropriate points in your content model. W3C XML Schema determinism requirements impose restrictions on where these extensions may appear. Generally, it is easiest of the extensions are allowed at the end of the content model.

Full Extensibility - XML Schema edition: All XML Elements MUST allow any attributes and allow any elements at the end of their content models. XML Elements MAY allow for element extensibility elsewhere in their content models.

Consider the following definition for a SOAP header lock for our callback example.

<s:complexType name="CallbackType">
   <s:sequence>
      <s:element name="callbackLocation"
                 type="s:anyURI"
                 minOccurs="1" maxOccurs="1"/>
      <s:element name="Extension"
                 type="wscb:ExtensionType"
                 minOccurs="0" maxOccurs="1"/>
      <s:any processContents="lax"
                 namespace="##other"
                 minOccurs="0" maxOccurs="unbounded"/>
   </s:sequence>
   <s:anyAttribute/>
</s:complexType>

<s:complexType name="ExtensionType">
   <s:sequence>
      <s:any processContents="lax"
             namespace="##targetnamespace"
             minOccurs="1" maxOccurs="unbounded"/>
   </s:sequence>
   <s:anyAttribute/>
</s:complexType>

This design has the serendipitous advantage that it separates third party extensions (in a different namespace) from compatible changes in the same namespace. If both extensions in the same namespace and in other namespaces are provided in the same document, there is no confusion about ordering of the extensions. This strict ordering of extensions guarantees that validation can occur. In particular, a node that has the newer language can use the newer schema, and all the binding and mapping tools that are available.

In both cases, lax validation is used to enable validation to succeed with or without a schema for the extension elements.

Use of an Extension element for extensions in the same namespace and namespace="##other for extensions in different namespaces avoids determinism issues discussed later.

9.2 Versioning

The callback element with a expires using XML Schema is

<wscb:Callback xmlns:wscb="http://example.com/callback/">
     <wscb:callbackLocation>
         http://example.com/foo/CallbackService
     </wscb:callbackLocation>
     <wscb:extension>
         <wscb:expires>60</wscb:expires>
     </wscb:extension>
</wscb:Callback>

Here is the callback with expires example in an XML Schema.

<s:complexType name="CallbackType">
   <s:sequence>
      <s:element name="callbackLocation" type="s:anyURI"
                  maxOccurs="unbounded"/>
      <s:element name="extension">
          <s:complexType>
              <s:sequence>
                  <s:element name="expires" type="s:integer"
                         minOccurs="0" maxOccurs="unbounded"/>
                  <s:element name="extension" type="wscb:ExtensionType"
                         minOccurs="0" maxOccurs="1"/>
                  <s:any processContents="lax" minOccurs="0"
                         maxOccurs="unbounded" namespace="##other"/>
              </s:sequence>
          </s:complexType>
      </s:element>
      <s:any processContents="lax" minOccurs="0" maxOccurs="unbounded"
             namespace="##other"/>
   </s:sequence>
   <s:anyAttribute/>
</s:complexType>

9.3 Determinism

XML DTDs and W3C XML Schema have a rule that requires schemas to have deterministic content models. The meaning of deteriminism can be seen in this example from the XML 1.0 specification:

"For example, the content model ((b, c) | (b, d)) is non-deterministic, because given an initial b the XML processor cannot know which b in the model is being matched without looking ahead to see which element follows the b."

The addition of wildcards in W3C XML Schema (not present in XML DTDs) leads to some schemas that we might like to express, but that aren't allowed.

• Wildcards with ##any, where minOccurs does not equal maxOccurs, are not allowed before an element declaration.

• The element before a wildcard with ##any must have a maxOccurs value that equals its minOccurs value. If these are different, then the optional occurrences could match either the element definition or the wildcard. As a result of this rule, the minOccurs must also be greater than zero.

• Types in any namespace derived by extension that add element definitions after a wildcard with ##any must be avoided.

• Types in a different namespace derived by extension that add element definitions after a wildcard with ##other must be avoided. This rule and the previous are roughly equivalent for explanation purposes: In these two cases, violating these constraints with an instance of the added element definition could match either the wildcard or the derived element definition. This is effectively the first bulleted rule applied to derivation. The problems of determinism make derivation and wildcarding almost mutually exclusive, the exception being extension in the same namespace and wildcards with ##other.

Be Deterministic: Use of wildcards must be deterministic. Location of wildcards, namespace of wildcard extensions, minOccurs and maxOccurs values are constrained, and type restriction is controlled.

We'll look at these alternatives in more detail below.

It is worth noting that the structural limitations introduced by W3C XML Schema's handling of determinism are a consequence of W3C XML Schema's design and are not an inherent limitation of schema-based structures. In RELAX NG, for example, none of these determinism constraints apply.

9.4 Containers and XML Schema

It should be noted that W3C XML Schema does not permit validation of containers with required or optional extensions. Effectively, W3C XML Schema cannot specify that a particular extension is required or optional in a wildcard element. It does have the process contents attribute that controls validation of content, but there is no option that says a particular extension is required. Languages such as WSDL SOAP binding extension have been created to allow the specification of a container and required extensions in containers like SOAP. The WSDL design for SOAP bindingis to come extension provides a new schema that merges the SOAP schema with the SOAP extension schemas.

9.5 Alternative Extensibility and Versioning Techniques

There are a variety of alternative extensibility and versioning techniques.

<s:schema xmlns=http://example.com/2003/02/soap/callback/
          xmlns:wscb="http://example.com/2003/02/soap/callback/">
   <s:element name="CallbackExtended"
              type="wsdb:CallbackExtendedType"/>
   <s:complexType name="CallbackType">
      <s:sequence>
         <s:element name="callbackLocation" type="s:anyURI"
               maxOccurs="unbounded"/>
         <s:any processContents="lax" minOccurs="0"
               maxOccurs="unbounded" namespace="##other"/>
      </s:sequence>
      <s:anyAttribute/>
   </s:complexType>
   <s:complexType name="CallbackExtendedType">
      <s:complexContent>
      <s:extension base="wscb:CallbackType">
         <s:sequence>
            <s:element name="conversationID" type="s:anyURI"
                   minOccurs="1" maxOccurs="1"/>
            <s:any processContents="lax" minOccurs="0"
                   maxOccurs="unbounded"/>
         </s:sequence>
         <s:anyAttribute/>
      </s:extension>
    </s:complexContent>
    </s:complexType>
</s:schema>

<CallbackExtended xmlns="http://example.com/2003/02/soap/callback/">
   <callbackLocation>
       http://example.com/foo/CallbackService
   </callbackLocation>
   <conversationID>
       http://example.com/convid/5
   </conversationID>
</CallbackExtended>