W3C

[Editorial Draft] Extending and Versioning Languages Part 2: XML Languages

Draft TAG Finding 12 December 2006

This version:
http://www.w3.org/2001/tag/doc/versioning-part2-20061212.html ( xml )
Latest version:
http://www.w3.org/2001/tag/doc/versioning-part2
Editor:
David Orchard, BEA Systems, Inc. <David.Orchard@BEA.com>

Abstract

@@rewrite

This document is the requested breakout of xml specific discussion of extensibility. It is heavily XML schema based, but only because of scheduling. OWL/RDF and RelaxNG sections wil be added.

Status of this Document

This document has been developed for discussion by the W3C Technical Architecture Group. It does not yet represent the consensus opinion of the TAG.

Publication of this finding does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time.

Additional TAG findings, both approved and in draft state, may also be available. The TAG expects to incorporate this and other findings into a Web Architecture Document that will be published according to the process of the W3C Recommendation Track.

Please send comments on this finding to the publicly archived TAG mailing list www-tag@w3.org (archive).

Table of Contents

1 Introduction
    1.1 XML Terminology
2 Version Identification Strategies using XML Namespaces
    2.1 Version Strategy: all components in new namespace(s) for each version (#1)
    2.2 Version Strategy: all new components in new namespace(s) for each compatible version (#2)
    2.3 Version Strategy: all new components in new or existing namespace(s) for each compatible version (#3)
    2.4 Version Strategy: all new components in existing or new namespace(s) for each version and a version identifier(#4)
    2.5 Version Strategy: all components in existing namespace(s) for each version and a version identifier(#5)
3 Indicating Incompatible changes
    3.1 Type changes
4 Schemas for Version Identification Strategies
    4.1 Version Strategy: all components in new namespace(s) for each version (#1)
    4.2 Version Strategy: all new components in new namespace(s) for each compatible version (#2)
    4.3 Version Strategy: All new components in existing or new namespace(s) for each compatible version(#3)
    4.4 Version Strategy: all new components in existing or new namespace(s) for each version and a version identifier(#4)
5 Indicating Incompatible changes
    5.1 Must Understand
    5.2 Type extension
    5.3 Substitution Groups
6 Determinism
7 Other technologies
8 Conclusion
9 References
10 Acknowledgements


1 Introduction

Extending and Versioning XML Languages Part 1 described extending and versioning languages. Part 2 focuses on XML and includes schema language specific aspects of extending and versioning XML. The choices, decisions, and strategies described in Part 1 are augmented with xml and schema instances herein.

1.1 XML Terminology

There are many different systems for exchanging texts in languages, such as SQL, Java, XML, ECMAScript, C#. We will briefly describe some key refinements to our lexicon for XML. An XML language has a vocabulary that may use terms from one or more XML Namespaces (or none), each of which has a namespace name. [Definition: An XML language is an identifiable set of vocabulary terms with defined XML syntactic and semantic constraints. ] By XML language, we mean the set of elements and attributes, or instances, used by a particular application. The Name Language - consisting of name, given, family terms - has a namespace for the terms. We use the prefix "namens" to refer to that namespace. The Name Language could consist of terms from other vocabularies, such as Dublin Core or UBL. These terms each have their own namespaces, illustrating that a language can comprise vocabularies from multiple namespaces. An XML Namespace is a convenient container for collecting terms that are intended to be used together within a language or across languages. It provides a mechanism for creating globally unique names.

We shall use the term instance when speaking of sequences of characters (aka text) in XML. [Definition: An instance is a specific, discrete Text in XML format.] Documents are instances of a language. In XML, they must have a root element. A name text might have a name element as the root element. Alternatively, the name vocabulary may be used by a language such as purchase orders. The purchase order texts may contain name elements. Thus instances of a language are always part of a text and also may be the entire text. XML instances (and all other instances of markup languages) consist of markup and content. In the name example, the given and family elements including the end markers are the markup. The values between the start and end markers are the content. An instance has an information model. There are a variety of data models within and without the W3C, and the one standardized by the W3C is the XML infoset.

The XML related terms and their relationships are shown below

UML diagram of XML terms

A stylesheet processor is a consumer of the XML text that it is processing (the producer isn't mentioned); in the Web services context the roles of producer and consumer alternate as messages are passed back and forth. Note that most Web service specifications provide definitions of inputs and outputs. By our definitions, a Web service that updates its output schema is considered a new producer. A Web service that updates its input schema is a new consumer.

2 Version Identification Strategies using XML Namespaces

There are a large variety of version identification designs. They range from many namespaces to only 1 namespace for all versions of a language. A few of the most common are listed below and described in more detail later.

  1. all components in new namespace(s) for each version

    ie version 1 consists of namespaces a + b, version 1.1 consists of namespaces c + d; or version 1 consists of namespace a, version 1.1 consists of namespace b.

  2. all new components in new namespace(s) for each compatible version

    ie version 1 consists of namespaces a + b; version 1.1 consists of namespaces a + b + c; version 2.0 consists of namespaces d + e.

  3. all new components in existing or new namespace(s) for each compatible version

    ie version 1 consists of namespace a, version 1.1 consists of namespace a, version 2 consists of namespace b; or version 1 consists of namespace a, version 1.1 consists of namespace a + b.

  4. all new components in existing or new namespace(s) for each version and a version identifier

    ie version 1 consists of namespace a + b + version attribute “1_, version 2 consists of namespace c + d + version attribute “2_.

  5. all components in existing namespace(s) for each version (compatible and incompatible) and a version identifier

    ie version 1 consists of namespace a + version attribute “1.0_, version 1.1 consists of namespace a + version attribute “1.1_, version 2.0 consists of namespace a + version attribute "2.0".

Whatever the design chosen, the language designer must decide the component name, namespace name, and any version identifier for new and all existing components. The trade-offs between the decisions relate to the importance of:

Elaborating on these designs is illustrative.

2.1 Version Strategy: all components in new namespace(s) for each version (#1)

The following names would be valid:

Example 1: All components in new namespace(s) instances
<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/2">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/3">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/3/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/3">
  <given>Dave</given>
  <family>Orchard</family>
  <middiffdomain:middle xmlns:middiffdomain="http://www.example.com/mid/1">Bryce</middiffdomain:middle>
</name>

The 2nd example shows all the components in the same new namespace. The 3rd and 4th example show an additional middle element in 2 different namespace names. The 3rd example comes from a namespace name that is in the same domain as the name element’s new namespace name. One reason for 2 namespaces is to modularize the language. The 4th example shows a namespace name from a different domain for the middle. It is probable that the mid:middle was created by the name author, and the middiffdomain:middle was created by a 3rd party.

This strategy has the impact of potentially and generally resulting in incompatible changes. When an older consumer receives the new texts in the new namespace, most of the software will break, such as performing schema validation without the new schema. Achieving forwards compatibility requires careful selection of technologies, such as XPath expressions that are namespace agnostic. The effect of the change being an incompatible change is the design goal of some systems that have adopted this strategy

2.2 Version Strategy: all new components in new namespace(s) for each compatible version (#2)

In this strategy, the following names would be valid:

Example 2: New components in new namespace(s) instances
<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <middiffdomain:middle xmlns:middiffdomain="http://www.example.com/mid/1">Bryce</middiffdomain:middle>
</name>

The 2nd and 3rd example show an additional middle element in 2 different namespace names. The first middle, the 2nd example, comes from a namespace name that is in the same domain as the name element’s namespace name. The 3rd example shows a complete different namespace name for the middle. It is probable that the mid:middle was created by the name author, and the middiffdomain:middle was created by a 3rd party.

2.3 Version Strategy: all new components in new or existing namespace(s) for each compatible version (#3)

In this strategy, the following names would be valid:

Example 3: New components in new or existing namespace(s) instances
<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <middiffdomain:middle xmlns:middiffdomain="http://www.example.com/mid/1">Bryce</middiffdomain:middle>
</name>

The 2nd example shows the use of the optional middle name in the name namespace. The 3rd and 4th example show an additional middle element in 2 different namespace names. The first middle, the 3rd example, comes from a namespace name that is in the same domain as the name element’s namespace name. The 4th example shows a complete different namespace name for the middle. It is probable that the mid:middle was created by the name author, and the middiffdomain:middle was created by a 3rd party.

2.4 Version Strategy: all new components in existing or new namespace(s) for each version and a version identifier(#4)

Using a version identifier, the name instances would change to show the version of the name they use, such as:

Example 4: New components in existing or new namespace(s) with version identifier instances
<name xmlns="http://www.example.org/name/1" version="1.0">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1" version="1.1">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/1" version="1.1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1" version="1.0">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/1" version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1">Bryce</mid:middle>
</name>

<name xmlns="http://www.example.org/name/2" version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

The last two examples show that the middle is now a mandatory part of the name. This is indicated by just the version number or a new namespace plus version number.

A significant downside with using version identifiers is that software that supports both versions of the name must perform special processing on top of XML and namespaces. For example, many components “bind_ XML types into particular programming language types. Custom software must process the version attribute before using any of the “binding_ software. In Web services, toolkits often take SOAP body content, parse it into types and invoke methods on the types. There are rarely “hooks_ for the custom code to intercept processing between the “SOAP_ processing and the “name_ processing. Further, if version attributes are used by any 3rd party extensions—say mid:middle has a version—then the schema cannot refer to the correct middle.

2.5 Version Strategy: all components in existing namespace(s) for each version and a version identifier(#5)

Using a version identifier, the name instances would change to show the version of the name they use, such as:

Example 5: New components in existing namespace(s) with version identifier instances
<name xmlns="http://www.example.org/name/1" version="1.0">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<name xmlns="http://www.example.org/name/1" version="1.1">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.example.org/name/1" version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
  <middle>Bryce</middle>
</name>

The 2nd example shows that the middle is an optional part of the name. The last example shows that the middle is a mandatory part of the name.

A downside with using new namespace names is that some tools, like XPath, can be harder to use in the face of new namespace names. Software that extracts the given and family name based upon the expanded name will often break if a new namespace name is used.

3 Indicating Incompatible changes

Given adoption of the Must Ignore rule, it is often the case that the creator of an extension or a new version wants to require that the consumer understand the extension, overriding the Must Ignore rule. The previous section showed how a version author could use new namespace names, element names, or version numbers to indicate an incompatible change. An extension author does not have these mechanisms available for indicating an incompatible or mandatory extension. A language provider that wants to allow extension authors to indicate incompatible extension must provide a mechanism for indicating that consumers must understand the extension.

Good Practice

Provide Must Understand Rule: Container languages SHOULD provide a “Must Understand_ model for indicating extensions that override a default Must Ignore Rule.

This rule and the Must Ignore rule work together to provide a stable and flexible processing model for extensions.

Must Understand flag

Arguably the simplest and most flexible over-ride technique is a Must Understand flag that indicates whether the item must be understood. The SOAP, WSDL, and WS-Policy attributes and values for specifying understand are respectively: soap:mustUnderstand=_1_, wsdl:required=_1_, wsp:Usage=_wsp:Required_. SOAP is probably the most common case of a container that provides a Must Understand model. The default value is 0, which is effectively the Must Ignore rule.

A language designer can re-use an existing Must Understand model by constraining their language to an existing Must Understand model. A number of Web services specifications have done this by specifying that the components are SOAP header blocks, which explicitly brings in the SOAP Must Understand model.

A language designer can design a Must Understand model into their language. A Must Understand flag allows the producer to insert extensions into the container and use the Must Understand attribute to over-ride the must Ignore rule. This allows producers to extend instances without changing the extension element’s parent’s namespace, retaining backwards compatibility. Obviously the consumer must be extended to handle new extensions, but there is now a loose coupling between the language’s processing model and the extension’s processing model. A Must Understand flag is provided below:

Example 6: New components in new namespace(s) with Must Understand
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.example.org/name/1" 
      xmlns:name="http://www.example.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="given" type="xs:string"/>
      <xs:element name="family" type="xs:string"/>
      <xs:any namespace="##other" processContents="lax" 
            minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute ref="name:mustUnderstand"/>
    <xs:anyAttribute/>
  </xs:complexType>

  <xs:attribute name="mustUnderstand" type="xs:boolean"/>
</xs:schema>

An example of an instance of a 3rd party indicating that a middle component is an incompatible change:

Example 7: New components in existing or new namespace(s) instance with Must Understand
<name xmlns="http://www.example.org/name/1">
  <given>Dave</given>
  <family>Orchard</family>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1"
                name:mustUnderstand="true">
      Bryce
  </mid:middle>
</name>

Specification of a Must Understand flag must be treated carefully as it can be computationally expensive. Typically a processor will either: perform a scan for Must Understand components to ensure it can process the entire text, or incrementally process the instance and is prepared to rollback or undo any processing if an not understood Must Understand is found.

There are other refinements related to Must Understand. One example is providing an element that indicates which extension namespaces must be understood, which avoids the scan of the instance for Must Understand flags.

It is also possible to re-use the SOAP processing model with it's mustUnderstand.

Example 8: Using SOAP Must Understand
<soap:envelope>
  <soap:body>
    <name xmlns="http://www.example.org/name/1">
    <given>Dave</given>
    <family>Orchard</family>
   </name>
  </soap:body>
</soap:envelope>

<soap:envelope>
  <soap:header>
  <mid:middle xmlns:mid="http://www.example.org/name/mid/1"
                soap:mustUnderstand="true">
      Bryce
  </mid:middle>
  </soap:header>
  <soap:body>
    <name xmlns="http://www.example.org/name/1">
    <given>Dave</given>
    <family>Orchard</family>
   </name>
  </soap:body>
</soap:envelope>

3.1 Type changes

The various schema languages have a variety of defined mechanisms to indicate extensions that are incompatible. For example, XML Schema provides Type Extension and Substitution Groups. These mechanisms are described in Part 2.

4 Schemas for Version Identification Strategies

4.1 Version Strategy: all components in new namespace(s) for each version (#1)

Using XML Schema 1.0, the name owner might like to write a schema such as:

Example 9: All components in new namespace(s) schema
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

The next version of the schema, with middle name added, might look like

Example 10: New components in new namespace(s) with Type Extension schema
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/2" 
      xmlns:name2="http://www.openuri.org/name/2"> 

  <xs:import namespace="http://www.openuri.org/name/1"/>

  <xs:complexType name="name">
    <xs:complexContent>
      <xs:extension base="name:name">
        <xs:sequence>
          <xs:element name="middle" type="xs:string" minOccurs="0"/>
        </xs:sequence>
      </xs:extension>
    </xs:complexContent>
  </xs:complexType>
</xs:schema>

This schema is illegal because the middle in the 2nd name namespace and the wildcard with ##other are non-deterministic. More on non-determinism in the next strategy. An alternative is to not have the wildcard at all, and rely upon subtyping for extension. But this prevents any kind of compatible evolution as both sides must have the new schema to understand the type. There language designer has to choose between allowing compatible extensibility/versioning OR incompatible extensibility when subtyping is used.

Because of the type extension determinism problem, the language designer cannot re-use the existing name definition. They must create a new schema without any reference to the previous schema.

Example 11: All components in new namespace(s) schema, no-reuse of v1 schema
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/2" 
      xmlns:name="http://www.openuri.org/name/2"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:element name="middle" type="xs:string" minOccurs="0"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

The new namespace for all components does not allow compatible evolution by the language designer, unless they choose to put new components in a new namespace which is strategy #2. Additionally, the version 2 schema cannot re-use the existing type definition.

4.2 Version Strategy: all new components in new namespace(s) for each compatible version (#2)

We previously saw how re-use by importing and extending schemas with wildcards is not possible. In this strategy, the schema designer attempts to insert the new extension in the existing schema definition, like:

Example 12: Illegal New components in new namespace(s) schema
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:element ref="mid:middle" minOccurs="0"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/mid/1" 
      xmlns:mid="http://www.openuri.org/name/mid/1"> 
  <xs:element name="middle" type="xs:string"/>
</xs:schema>

However, the determinism constraint of XML Schema, described in more detail later, prevents this from working. The problem arises in a version when an optional element is followed by a wildcard. In this example, this occurs when an optional element is added and extensibility is still desired. This is an ungentle introduction to the difference between extensibility and versioning. An optional middle name added into a subsequent version is a good example. Consumers should be able to continue processing if they don’t understand an additional optional middle name, and we want to keep the extensibility point in the new version. We can't write a schema that contains the optional middle name and a wildcard for extensibility. The previous schema schema is roughly what is desired using wildcards, but it is illegal because of the determinism.

The author has 4 options for the v2 schema for name and middle, listed below and detailed subsequently:

  1. optional middle, extensibility retained, but name type does not refer to middle;

  2. optional middle, extensibility is lost, name type refers to middle;

  3. required middle, extensibility retained, name type refers to middle but compatibility is lost (essentially strategy #1);

  4. no update to the Schema

If they leave the middle as optional and retain the extensibility point, the best schema that they can write is:

Example 13: New components in new namespace(s) schema V2, no change to name type
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/mid/1" 
      xmlns:mid="http://www.openuri.org/name/mid/1"> 
  <xs:element name="middle" type="xs:string"/>
</xs:schema>

This is not a very helpful XML Schema change. The problem is that they cannot insert the reference to the optional mid:middle element in the name schema and retain the extensibility point because of the aforementioned Non-Determinism Constraint.

The core of the problem is that there is no mechanism for constraining the content of a wildcard. For example, imagine that ns1 contains foo and bar. It is not possible to take the SOAP schema—an example of a schema with a wildcard - and require that ns1:foo element must be a child of the header element and ns1:bar must not be a child of the header element using just W3C XML Schema constructs.   Indeed, the need for this functionality spawned some of the WSDL functionality.

They could decide to lose the extensibility point (option #2), such as

Example 14: New components in new namespace(s) schema V2, no extensibility
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"
      xmlns:mid="http://www.openuri.org/name/mid/1">

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:element ref="mid:middle" type="xs:string" minOccurs="0"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/mid/1" 
      xmlns:mid="http://www.openuri.org/name/mid/1"> 
  <xs:element name="middle" type="xs:string"/>
</xs:schema>

This does lose the possibility for forwards-compatible evolution.

The final option, #3, is adding a required middle. They must indicate the change is incompatible. A new namespace name for the name element can be created. This is essentially strategy #1, new namespace for all components.

The downsides of the 3 options for new components in new namespace name(s) design have been described. Additionally, the design can result in specifications and namespaces that are inappropriately factored, as related constructs will be in separate namespaces.

4.3 Version Strategy: All new components in existing or new namespace(s) for each compatible version(#3)

It is possible to create Schemas with additional optional components. This requires re-using the namespace name for optional components and special schema design techniques. The re-using namespace rule is:

Good Practice

Re-use namespace names Rule: If a backwards compatible change can be made to a specification, then the old namespace name SHOULD be used in conjunction with XML’s extensibility model.

It is important to note that that a new namespace name is not required whenever a specification evolves - strategies #1 and #2 - but rather a new namespace name can be required only if an incompatible change is made. Strategy #1 uses a new namespace for all existing components and any additions, Strategy #2 uses a new namespace for all additions. Strategy #3 re-uses namespaces for compatible extensions.

Good Practice

New namespaces to break Rule: A new namespace name is used when backwards compatibility is not permitted, that is software MUST break if it does not understand the new language components.

Earlier examples showed that it is not possible to have a wildcard with ##any (or even ##targetnamespace) following optional elements in the targetnamespace. The solution to this problem is to introduce an element in the schema that will always appear if the extension appears. The content model of the extensibility point is the element + the extension. There are two styles for this. The first was published in an earlier version of this Finding in December 2003. It uses an Extensibility element with the extensions nested inside. The second was published in July 2004, then updated on MSDN. It uses a Sentry or Marker element with extensions following it.

A name type with extension elements is

Example 15: New components in existing or new namespace(s) with Extension Type Schema version 1
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:element name="Extension" type="name:ExtensionType" 
                 minOccurs="0" maxOccurs="1"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>

  <xs:complexType name="ExtensionType">
    <xs:sequence>
      <xs:any processContents="lax" minOccurs="1" 
              maxOccurs="unbounded" namespace="##targetnamespace"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType> 
</xs:schema>

Because each extension in the targetnamespace is inside an Extension element, each subsequent target namespace extensions will increase nesting by another layer. While this layer of nesting per extension is not desirable, it is what can be accomplished today when applying strict XML Schema validation. It seems to at least this author that potentially having multiple nested elements is worthwhile if multiple compatible revisions can be made to a language. This technique allows validation of extensions in the targetnamespace and retaining validation of the targetnamespace itself.

The previous schema allows the following sample name:

Example 16: New components in existing or new namespace(s) with Extension Type instances
<name xmlns="http://www.openuri.org/name/1">
  <first>Dave</first>
  <last>Orchard</last>
  <Extension>
    <middle>Bryce</middle>
  </Extension>
</name>

The namespace author can create a schema for this type

Example 17: New components in existing or new namespace(s) with Extension Type Schema version 2
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <s:element name="Extension" type="name:middleExtensionType" 
                 minOccurs="0" maxOccurs="1"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>

  <xs:complexType name="middleExtensionType">
    <xs:sequence>
       <xs:element name="middle" type="xs:string"/>
       <xs:element name="Extension" type="name:middleExtensionType" 
                   minOccurs="0" maxOccurs="1"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType> 

  <xs:complexType name="ExtensionType">
    <xs:sequence>
      <xs:any processContents="lax" minOccurs="1" 
              maxOccurs="unbounded" namespace="##targetnamespace"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType> 
</xs:schema>

The advantage of this design technique is that a forwards and backwards compatible Schema V2 can be written. The V2 schema can validate documents with or without the middle, and the V1 schema can validate documents with or without the middle.

Further, the re-use of the same namespace has better tooling support. Many applications use a single schema to create the equivalent programming constructs. These tools often work best with single namespace support for the “generated_ constructs. The re-use of the namespace name allows at least the namespace author to make changes to the namespace and perform validation of the extensions.

An obvious downside of this approach is the complexity of the schema design. Another downside is that changes are linear, so 2 potentially parallel extensions must be nested rather than parallel.

4.4 Version Strategy: all new components in existing or new namespace(s) for each version and a version identifier(#4)

Using a version identifier, the name instances would change to show the version of the name they use, such as:

Example 18: New components in existing or new namespace(s) with version identifier instances
<name xmlns="http://www.openuri.org/name/1" version="1.0">
  <first>Dave</first>
  <last>Orchard</last>
</name>

<name xmlns="http://www.openuri.org/name/1" version="1.0">
  <first>Dave</first>
  <last>Orchard</last>
  <middle>Bryce</middle>
</name>

<name xmlns="http://www.openuri.org/name/1" version="1.1">
  <first>Dave</first>
  <last>Orchard</last>
  <pref1:middle xmlns:mid1="http://www.openuri.org/name/mid/1">Bryce</pref1:middle>
</name>

<name xmlns="http://www.openuri.org/name/1" version="1.0">
  <first>Dave</first>
  <last>Orchard</last>
  <pref2:middle xmlns:mid2="http://www.example.org/name/mid/1">Bryce</pref2:middle>
</name>

<name xmlns="http://www.openuri.org/name/1" version="2.0">
  <first>Dave</first>
  <last>Orchard</last>
  <pref1:middle xmlns:mid1="http://www.openuri.org/name/mid/1">Bryce</pref1:middle>
</name>

The last example shows that the middle is now a mandatory part of the name. As with Design #2, the schema for the optional middle cannot fully express the content model. A schema for the mandatory middle is

Example 19: New components in existing or new namespace(s) with version identifier schema v2, incompatible change
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"
      xmlns:mid="http://www.openuri.org/name/mid/1">

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:element name="middle" type="xs:string" minOccurs="0"/>
      <xs:element ref="mid:middle"/>
      <xs:any namespace="##other" processContents="lax" 
              minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>
</xs:schema>

<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/mid/1" 
      xmlns:mid="http://www.openuri.org/name/mid/1"> 
  <xs:element name="middle" type="xs:string"/>
</xs:schema>

A significant downside with using version identifiers is that software that supports both versions of the name must perform special processing on top of XML and namespaces. For example, many components “bind_ XML types into particular programming language types. Custom software must process the version attribute before using any of the “binding_ software. In Web services, toolkits often take SOAP body content, parse it into types and invoke methods on the types. There are rarely “hooks_ for the custom code to intercept processing between the “SOAP_ processing and the “name_ processing. Further, if version attributes are used by any 3rd party extensions—say mid:middle has a version—then the schema cannot refer to the correct middle.

5 Indicating Incompatible changes

5.1 Must Understand

A schema for a Must Understand flag is provided below:

Example 20: New components in existing or new namespace(s) with Extension Type Schema and Must Understand
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema"
      targetNamespace="http://www.openuri.org/name/1" 
      xmlns:name="http://www.openuri.org/name/1"> 

  <xs:complexType name="name">
    <xs:sequence>
      <xs:element name="first" type="xs:string"/>
      <xs:element name="last" type="xs:string"/>
      <xs:element name="middle" type="xs:string" minOccurs="0"/>
      <s:element name="Extension" type="name:ExtensionType" 
                 minOccurs="0" maxOccurs="1"/>
      <xs:any namespace="##other" processContents="lax" 
            minOccurs="0" maxOccurs="unbounded"/>
    </xs:sequence>
    <xs:attribute ref="name:mustUnderstand"/>
    <xs:anyAttribute/>
  </xs:complexType>

  <xs:complexType name="ExtensionType">
    <xs:sequence>
      <xs:any processContents="lax" minOccurs="1" 
             maxOccurs="unbounded" namespace="##targetnamespace"/>
    </xs:sequence>
    <xs:anyAttribute/>
  </xs:complexType>

  <xs:attribute name="mustUnderstand" type="xs:boolean"/>
</xs:schema>

An example of an instance of a 3rd party indicating that a middle component is an incompatible change:

Example 21: New components in existing or new namespace(s) instance with Must Understand
<name xmlns="http://www.openuri.org/name/1">
  <first>Dave</first>
  <last>Orchard</last>
  <pref2:middle xmlns:mid2="http://www.example.org/name/mid/1"
                name:mustUnderstand="true">
      Bryce
  </pref2:middle>
</name>

5.2 Type extension

Another option for indicating mandatory requirements is allowing extension authors to use other schema mechanisms for extending the main type, such as type extension. The language designer allows for type extension, and they must specify that type extensions must be understood. Strategy #1 showed that it is impossible to write a schema that extends an existing type that has a wildcard with a new namespace'd component. Type extension does not enable extension authors to indicate incompatible extensions.

5.3 Substitution Groups

Another mechanism for extending a type in XML Schema is substitution groups. Substitution groups enable an element to be declared as substitutable for another. This can only be used for incompatible extensions as the consumer must understand the substitution type. Substitution groups require that elements are available for substitution, so the name designer must have provided a name element in addition to the name type.

Substitution groups do allow a single extension author to indicate that their changes are mandatory. The limitations are that the extension author has now taken over the type’s extensibility. A visual way of imagining this is that the type tree has now been moved from the language designer over to the extensions author. And the language designer probably does not want their type to be “hijacked_.

However, this is not substantially different than an extension being marked with a “Must Understand_. In either case—with the extensions higher up in the tree (sometimes called top-typing) or lower in the tree (bottom-typing)—a new type is effectively created.

The difference is that there can only be 1 element at the top of an element hierarchy. If multiple mandatory extensions are added, then the only way to compose them together is at the bottom of the type because that is where the extensibility is.

Substitution groups do not allow a language designer and an extension author to incompatibly change the language as they end up conflicting over what to call the name element. Thus substitution groups are a poor mechanism for allowing an extension author to indicate that their changes are incompatible. A Must Understand flag is a superior method because it allows multiple extension authors to mix their mandatory extensions with a language designer’s versioning strategy. Hence language designers should prevent substitution groups and provide a Must Understand flag or other model when they wish to allow 3rd parties to make incompatible changes.

In some cases, a language does not provide a Must Understand mechanism. In the absence of a Must Understand model, the only way to force consumers to reject a message if they don’t understand the extension namespace is to change the namespace name of the root element, but this is rarely desirable.

6 Determinism

This Finding has spent considerable material describing deterministic content models, and so it is worthy of describing the W3C XML Schema determinism rules in more detail. The reader is reminded that these rules are unique to W3C XML Schema and other XML Schema languages like RELAX NG do not use these rules and so do not suffer from the contortions one is forced through when using W3C XML Schema. XML DTDs and W3C XML Schema have a rule that requires schemas to have deterministic content models. From the XML 1.0 specification,

“For example, the content model ((b, c) | (b, d)) is non-deterministic, because given an initial b the XML processor cannot know which b in the model is being matched without looking ahead to see which element follows the b._

The use of ##any means there are some schemas that we might like to express, but that aren’t allowed.

Good Practice

Be Deterministic rule: Use of wildcards MUST be deterministic. Location of wildcards, namespace of wildcard extensions, minOccurs and maxOccurs values are constrained, and type restriction is controlled.

As shown earlier, a common design pattern is to provide an extensibility point—not an element - allowing any namespace at the end of a type. This is typically done with <xs:any namespace=_##any_>.

Determinism makes this unworkable as a complete solution in many cases. Firstly, the extensibility point can only occur after required elements in the original schema, limiting the scope of extensibility in the original schema. Secondly, backwards compatible changes require that the added element is optional, which means a minOccurs=_0_. Determinism prevents us from placing a minOccurs=_0_ before an extensibility point of ##any. Thus, when adding an element at an extensibility point, the author can make the element optional and lose the extensibility point, or the author can make the element required and lose backwards compatibility.

7 Other technologies

The W3C XML Schema Working has heard and taken to heart many of these concerns. They have plans to remedy some of these issues in XML Schema 1.1 [21]. They currently are looking at a “weak wildcard_ model, which solves some but not all of the problems. There is no public Working Draft of a Schema 1.1 with improved extensibility or versioning at the time of writing this Finding.

A simple analysis of doing compatible extensibility and versioning using RDF and OWL is available [21]. In general, RDF and OWL offer superior mechanisms for extensibility and versioning. RDF and OWL explicitly allow extension components to be added to components. And further, the RDF and OWL model builds in the notion of “Must Ignore Unknowns_ as an RDF/OWL processor will absorb the extra components but do nothing with them. An extension author can require that consumers understand the extension by changing the type using a type extension mechanism.  

RELAX NG is another schema language. It explicitly allows extension components to be added to other components as it does not have the non-determinism constraint.

8 Conclusion

This Finding describes a number of questions, decisions and rules for using XML, W3C XML Schema, and XML Namespaces in language construction and extension. The main goal of the set of rules is to allow language designers to know their options for language design, and ideally make backwards- and forwards-compatible changes to their languages to achieve loose coupling between systems.

9 References

FOLDOC
Free Online Dictionary of Computing. (See http://wombat.doc.ic.ac.uk/foldoc/.)
FlexXMLP
Flexible XML Processing Profile. (See http://www.upnp.org/download/draft-goland-fxpp-01.txt.)
MIME
RFC 1521, MIME. (See http://www.ietf.org/rfc/rfc1521.txt.)
HTML 2.0
RFC 1866, HTML 2.0. (See http://www.ietf.org/rfc/rfc1866.txt.)
WebDAV XMLIgnore post
Yaron GolandXML Ignore proposed for WebDAV (See http://lists.w3.org/Archives/Public/w3c-dist-auth/1997AprJun/0190.html.)
WebDAV
RFC 2518, WebDAV (See http://www.ietf.org/rfc/rfc2518.txt.)
HTML 4.0
HTML 4.0. (See http://www.w3.org/TR/1998/REC-html40-19980424/.)
TBL Mandatory Extensions
Berners-Lee. Web Architecture: Mandatory extensions. (See http://www.w3.org/DesignIssues/Mandatory.html.)
TBL Extensible languages
Berners-Lee. Web Architecture: Extensible languages. (See http://www.w3.org/DesignIssues/Extensible.html.)
TBL Evolution
Berners-Lee. Web Architecture: Evolvability. (See http://www.w3.org/DesignIssues/Evolution.html.)
Web Architecture: Extensible Languages
Berners-Lee and Connolly, ed. Web Architecture: Extensible Languages World Wide Web Consortium, 1998. (See http://www.w3.org/TR/1998/NOTE-webarch-extlang-19980210.)
HTML Document types
Connolly, ed. HTML Document dialects World Wide Web Consortium, 1996. (See http://www.w3.org/MarkUp/WD-doctypes.)
SOAP 1.2
W3C Recommendation, SOAP 1.2 Part 1: Messaging Framework (See http://www.w3.org/TR/SOAP/.)
WSDL 1.1
W3C Note, WSDL 1.1 (See http://www.w3.org/TR/WSDL/.)
WS-Policy 1.2
W3C Note, WS-Policy 1.2 (See http://www.w3.org/Submissions/WS-Policy/.)
XML 1.0
W3C Recommendation, XML 1.0 (See http://www.w3.org/TR/REC-xml.)
XInclude
W3C Working Draft, XML Inclusions (See http://www.w3.org/TR-Xinclude.)
XML Namespaces
W3C Recommendation, XML Namespaces (See http://www.w3.org/TR/REC-xml-names.)
XML Schema Part 2
W3C Recommendation, XML Schema, Part 2 (See http://www.w3.org/TR/xmlschema-2.)
XML Schema Wildcard Test Collection
XML Schema Wildcard Test collection (See http://www.w3.org/XML/2001/05/xmlschema-test-collection/result-ms-wildcards.htm.)
XFront Schema Best Practices
XFront Schema Best Practices (See http://www.xfront.com/BestPracticesHomepage.html.)
XML.com Schema Design Patterns
Dare ObasanjoXML.com Schema design patterns (See http://www.xml.com/pub/a/2002/07/03/schema_design.html.)
Dave Orchard writings on Extensibility and Versioning
Dave Orchard writings on extensibility and versioning (See http://www.pacificspirit.com/Authoring/Compatibility.)

10 Acknowledgements

The author thanks the many reviewers that have contributed to the article, particularly David Bau, William Cox, Ed Dumbill, Chris Ferris, Yaron Goland, Hal Lockhart, Mark Nottingham, Jeffrey Schlimmer, Cliff Schmidt, and Norman Walsh.