[Editorial Draft] Extending and Versioning Languages: Strategies

1.1 Why Do Languages Change?

There are many reasons why a different version of a language may be needed. A few of them include:

1. Bugs may need to be fixed. Production use may reveal defects or oversights that need to be fixed. This may involve changes to texts of the language or changes to the information of existing texts.

2. Changing requirements may motivate changes in the schema design. For example, a middle, prefix, suffix, common may be added to a name structure.

3. Different flavors of a schema may be desirable. For example, the XHTML 1.0 Recommendation defines strict, transitional, and frameset schemas. All three of those schemas purport to define the same namespace, but they describe very different languages.

   And additional schemas may be defined by other specifications, such as the XHTML Basic Recommendation.

Whatever the cause, over time, different versions of the language exist and designing applications to deal with this change in a predictable, useful way requires a versioning strategy.

Whether ten, a hundred, or a million resources have been deployed, if a language is changed in such a way that all those resources will consider texts of the new language invalid, a versioning problem with real costs has been introduced.

1.2 How Do Languages Change?

One of the most important aspects of a language change is whether texts in the revised language are backwards or forwards compatible with the unrevised language .

Some typical backwards- and forwards-compatible changes:

• adding optional components ( in XML, this is generally elements and/or attributes)

• adding optional content, for example extending an enumeration

Some typical forwards-compatible changes:

• Decreasing the maximum allowed number of occurrences of a component but not to less than the minimum

• Decreasing the allowed range of a component

Some typical backwards-compatible changes:

• Increasing the maximum allowed number of occurrences of a component

• Increasing the allowed range of a component

Some typical incompatible changes:

• changing the information conveyed by existing components

• adding required components

• removing required components

• restricting a components content model, such as changing a choice to a sequence in XML

• adding new components that changes the semantics of existing components

1.3 Kinds of Languages

Ultimately, there are different kinds of languages. The versioning approaches and strategies that are appropriate for one kind of language may not be appropriate for another. Among the various kinds of languages, we find:

• Just Names: some languages don't actually have a syntax or grammar; they're just lists of names. Using names to identify words in the WordNet database, for example, or the names of functions and operators in XPath2 are examples of "just name" languages.

• Markup: SGML, HTML, and the non-SGML variants of HTML are all markup languages. XML languages are described in Versioning - XML

• Non-markup Text: languages designed with a text format. These may be programming languages such as Java or ECMAScript, or data formats like CSS or Comma Separated Values. Typically these are intended for humans to author and or view.

• binary: languages that are not in a text format. These may be image formats like GIF, JPEG, or even binary encoded XML.

This is by no means an exhaustive list. Nor are these categories exclusive. Many languages have mixed modes. For example, XQuery has a text mode and an XML mode.

Languages may be composed of different languages, including different kinds of languages. If a language is composed of other languages, then languages are considered nested. There may be different versioning strategies for each nested language, and they all combine together into the overall versioning strategy.

2.1 Why Have a Strategy?

Different kinds of languages and different versioning strategies expose different problems. Attempting to deploy a system that provides no versioning mechanism puts the burden of version "discovery" on consumers and is often impractical in anything except a closed system.

At the other end of the spectrum is the "big bang" approach which is also problematic.

"Big bang" is a very coarse-grained approach to versioning. It establishes a single version identifier, either a version number or namespace name, for an entire text.

The semantics of the "big bang" are that applications decide on the basis of the text version whether or not they know how to process that text. If the version isn't recognized, the entire text is rejected. Typically, when introducing a new version using the big bang approach, all of the software that produces or consumes the texts is updated in a sweeping overhaul in which the entire system is brought down, the new software deployed and the system is restarted. This big bang approach to versioning is practical only in circumstances where there is a single controlling authority, and even in that case, it carries with it all manner of problems. The process can take a considerable amount of time, leaving the system out of commission for hours if not days. This can result in significant losses if the system is a key component of a revenue generating business process and the cost of coordinating the system overhaul can also be quite costly as well.

The "big bang" approach is appropriate when the new version is radically different from its predecessor. But in many cases, the changes are incremental and often a consumer could, in practice, cope with the new version. For example, it might be that there are many messages that don't use any features of the new version or perhaps it is appropriate to simply ignore components that are not recognized.

Recall our Name example and consider a producer and a consumer exchanging name messages. Imagine that some future version of the name language defines a new "middle" component. Because producers and consumers are distributed, it may happen that an old consumer, one unprepared for a middle component, encounters a message with a middle component sent by a newer producer.

If big bang versioning is used, old systems will reject the new message. However, if the versioning strategy allowed the old consumer to simply ignore unrecognized content, it's quite possible that other components of the system could simply adapt to the previous behavior. In effect, the old system would ignore the middle component and its descendents so it would "see" a message that looks just like the old message it is expecting.

For the producer, the result would be that the request is fulfilled, though perhaps not quite the way it may have hoped. For example, a request that results in a name response may return a name without the middle. In many cases this may be better behavior than receiving an error. In particular, producers using the new language can be written to cope with the possibility that they will be speaking to older consumers.

If the new system needs to make sure that middle is respected, then it can change the language in an incompatible way to indicate that the new behavior is not considered backwards compatible.

Often, what is needed is some sort of middle ground solution.

2.2 Versioning Designs

For any given strategy, there are various designs that achieve the strategy, and some may be more appropriate than others. Among them we find:

3.1 What language form

Languages can be expressed in text, comma separated values, XML, SGML, binary, source code, and almost any kind of form. See the Architecture of the World-Wide Web section on data formats for more information - http://www.w3.org/TR/2004/REC-webarch-20041215/#formats.

3.2 Can 3rd parties extend the language?

It is sometimes desirable to prevent 3rd parties from extending languages, but it does happen. An example may be a tightly constrained security environment where distributed authoring is considered a "bug" rather than a feature.

3.3 Can 3rd parties extend the language in a compatible way?

If so, a substitution mechanism is required for forwards compatibility. If an older consumer has no mechanism for dealing with new content, then forwards-compatible evolution isn't possible. One simple substitution mechanism is simply ignoring the unrecognized components.

3.4 Can 3rd parties extend the language in an incompatible way?

If so, and if compatible extensions are also possible, then it must be possible to identify incompatible changes so that they can override the substitution mechanism used for extensible changes.

In environments where unrecognized components are accepted, a "must understand" component can be added to identify incompatible changes.

If compatible changes are not possible, then incompatible changes simply become the default. For example, WS-Security mandates that 3rd parties can only provide incompatible extensions. Unlike most languages, a security language has unique requirements where the consequences of accepted data can be severe. WS-Security accomplished this by specifying that all extensions are required to be understood and there is no substitution mechanism.

3.5 Can the designer extend the language in a compatible way?

As with 3rd parties compatible extensions, a substitution mechanism for the designer’s extensions is required for forwards compatibility.

3.6 Can the designer extend the language in an incompatible way?

In XML, the designer can always do this by using new namespace names, element names or version numbers. In other languages, this may not be possible because there is no mechanism for indicating the incompatible change.

3.7 Is the vocabulary a stand-alone language or an extension of another vocabulary?

A part of this question is whether the language depends on another language. That determines which, if any, facilities are provided for the containing language and what must be provided by a contained language.

SOAP is an example of a container language. The SOAP processing model applies uniformly to all headers, which may employ soap:mustUnderstand to identify incompatible changes, even though the contents of the SOAP headers are languages independent from SOAP.

3.8 What Schema language(s)?

Choosing a schema language or languages guides the language design in many ways. Some features, particularly extensibility, must be anticipated in the first version of a language in order to take advantage of the features of some schema languages.

In addition, various features may be incompatible across different languages. For example, writing a V2 compatible schema in W3C XML Schema requires special design, which is not required in a schema language such as RELAX NG. Some of the language design choices mandated by W3C XML Schema are discussed in other sections of this Finding.

3.9 Should extensions or versions be expressible in the Schema language?

The ability to write a schema for extensions or versions is directly affected by the schema design and the compatibility desires.

3.10 Requirements Summary

Every language design will make decisions about these requirements. These requirements can be expressed in a table form:

Requirement
Language form
Schema Lang
3rd party compatibly extend
3rd party incompatibly extend
Designer incompatibly extend
stand-alone

4.1 Schema language design choices or constraints.

If the language is intended to be capable of compatible extensibility, then a few specific schema design choices must be followed.

4.2 Substitution Mechanism.

Forwards compatibility can only be achieved by providing a substitution mechanism for Version 2 instances or Version 1 extensions to V1 without knowledge of V2.  A V1 consumer must be able to transform any instances, such as V1 + extensions, to a V1 instance in order to process the instance. The "Must Accept and Discard Unknowns" rule is a simple substitution mechanism. This rule says that any extensions are "ignored". Using it, a V1 + extensions text is transformed into a V1 text by discarding the extensions. Others substitution mechanisms exist, such as the fallback model in XSLT.

4.3 Component identification

The identification of components into language versions or extensions has a variety of general mechanisms related to namespaces. These are detailed in the Versioning section.

4.4 Identification of incompatible extensions

The identification of versions is covered by language identification, but 3rd parties cannot arbitrarily change versions or change namespaces. They may need a mechanism to indicate that an extension is an incompatible change. A couple of mechanisms are a "Must Understand" identifier (such as a flag or list of required namespaces) or requiring that extensions are in substitution groups.

4.5 Design Summary

Every language design will make a decision in these areas. These designs can also be expressed in a table form:

Design
Schema design
Substitution Mechanism
Component Identification
Incompatible Ext identification

5.1 Version Numbers

Having multiple versions naturally leads to the need to identify versions. Version identification has traditionally been done with a decimal separating the major versions from the minor versions, ie "8.1", "1.0". Often the definition of a "major" change is that it is incompatible, and the definition of a "minor" change is that it is forwards- and/or backwards - compatible. Usually the first broadly available version starts at "1.0". A compatible version change from 1.0 might be identified as "1.1" and an incompatible change as "2.0".

The version numbers can be contained in the texts, in the protocol messages containing in the text, or the address for the protocol messages. Some examples are shown below:

<name version="2.0">
  <given>Dave</given>
  <family>Orchard</family>
</name>

<span class="fn20">Dave Orchard</span>

urn:nameschemev2:given:Dave:family:Orchard

<?XML version="1.1"?>

GET /name/123456789  HTTP/1.1

GET /name/v2/123456789/ HTTP 1.1

It should be noted that associating version number changes with compatibility changes may be idealistic as there abundant cases where this system does not hold. New major version identifiers are often aligned with product releases, or incompatible changes identified as a "minor" change. A good example of an incompatible changed identified as a minor change is XML 1.1. XML 1.0 processors cannot process all XML 1.1 documents because XML 1.1 extended XML 1.0 where XML 1.0 does not allow such extension.

Unfortunately, version numbers often wind up looking very similar to the big bang approach. In many approaches, each language is given a version identifier, almost always a number, that's incremented each time the language changes. Although it's possible to design a system with version numbers that enables both backward and forward compatibility - for example XSLT - typically a version change is treated as if that the new language is not backwards compatible with the old language.

Some efforts, such as HTTP, try to have the best of both worlds by allowing for extensibility (in HTTP's case, via headers) as well as version numbers that explicitly identify when a new version is backwards compatible with an old version.

One argument in favor of version numbers is that they allow one to determine what is a 'new version' and what is an 'old version'. But in practice this is not necessarily true. For example, RSS has 0.9x, 1.x, and 2.x versions, all being actively developed in parallel. In effect the version numbers, even though they appear to be ordered, are simply opaque identifiers. Using version numbers does not gaurantee that version 1+x has any particular relationship to version 1.

Version numbers typically work best when versioning and extending a language is done in a centralized and linear manner. The makeup of each version can then be consistent and well described.

5.2 XML Namespaces

There are many cases where decentralized and non-linear versioning is desired. The desire for decentralized and non-linear versioning and extensibility was a large motivator for XML and for XML Namespaces. The self-describing and extensible nature of XML markup, and the addition of XML Namespaces, provides a framework for developing languages that can evolve in a decentralized manner. XML Namespaces [ XML Namespaces 1.0] provide a mechanism for associating a URI with an XML element or attribute name, thus specifying the language of the name. This also serves to prevent name collisions.

6.1 HTML