W3CArchitecture Domain XML | XML Schema

XML Schema Working Group

Framework for discussion of versioning

2 March 2005

W3C XML Schema Working Group

ed. C. M. Sperberg-McQueen



This is an attempt to define a conceptual framework for discussions of versioning in the context of XML Schema.
It is not yet finished, and the exposition is almost certainly not fit for reading by any readers who are feeling at all impatient. We hope to improve it as time goes on.

1. Motivating use cases

Note: a fuller account of some of these use cases may be found in another document of this Working Group. The descriptions there supersede those given here.

1.1. The ‘local customization’ / invoice use case

(Note: in some email threads this is sometimes referred to as the ‘Lee Buck’ use case.) The original description was posted to the XML Schema WG mailing list in October 1999; it is reproduced below. Note that the description predates the introduction of redefine into XML Schema 1.0.
CoolToys Inc. receives a notice from MegaRetail Inc. that henceforth they will accept only invoices that conform the the supplied schema “MegaInvoice.xsd”. Rather than just bolting on an emitter of MegaInvoices, CoolToys gets ‘religion’ and decides to standardize its invoice on MetaInvoice.xsd. They have an additional piece of information that they care about (in addresses as we'll see), but thanks to the wonders on refinement they are confident that they will be able to create CoolInvoice.xsd, refine the necessary archetype and be done.
Conceptually what they have to do is:
  1. Create a new schema named "CoolInvoice.xsd"
  2. Reference MegaInvoices.xsd
  3. Refine the type Address with SubAddress
  4. Require that anywhere address appears in MegaInvoice that SubAddress must appear instead.
At first blush it seems like refinement made this much easier, they can leverage the work MegaRetail did on MegaInvoicexsd and make a modest ‘delta’ schema which references MegaInvoice.xsd and makes a small change...
But that's not how it works... they will have to re-define every archetype that includes Address in its content model to restrict it to SubAddress... and then having done that, they'll need to change every archetype that refers to the archetypes that refer to Address, ..., all the way up the heirarchy. (And they won't even be able to use refinement for these changes.) By the time they are done all that they will have essentially created a shadow schema with virtually no association with the old one. And they may be left scratching their head as to what the value of refinement is!
ps: The above scenario might equally be cast as version 1 and version 2 of MegaInvoices.xsd

1.2. The ‘convenience store’ use case

Note: in some discussions, this use case has also been known as the ‘David Ezell’ use case.
Consider a convenience store, with an electronic cash register, gas pumps, and car wash.
The computer systems in the cash register, gas pumps, and car wash are built and sold by different vendors, but they must cooperate: the peripherals send transaction data to the cash register, and the cash register sends data of various kinds (price, prompt message text, rules for offering a discounted car wash to customers who purchase a certain amount of gasoline, etc.).
In order to provide for some interoperation among devices, vendor-neutral organizations define schemas which provide a sort of neutral common ground. Because the vendors compete for sales, it is to each vendor's advantage to add value by providing more useful information than the other vendor's product; this takes the form of adding new subelements or attributes to items in the common schema.
Because the devices must cooperate, dropped messages are a potential problem, and if one device rejects a message sent by another device, the result is likely to be a discussion between the vendors about whose device is at fault: did the gas pump emit a bad message, or did the cash register wantonly reject a perfectly legal data stream? In order to make such disputes relatively easy to resolve, it is helpful to have a written description of the set of messages the devices are required to accept. This description takes the form of a schema document: that is, the accept set is defined as the set of documents valid against a particular schema. (Either VF or VP will do, although VP may involve some further stipulations about how full or partial the validation has to be.)
Thus any approach to versioning which involves accepting partially valid documents would seem to involve abandoning the role of the schema as defining the contract between sender and recipient; that would leave the vendors without a convenient way to adjudicate disputes over dropped data and is thus unacceptable in practice.
For security reasons, the convenience store devices do not load new schemas dynamically; schemas may be changed as part of an upgrade or as part of system maintenance, but the upgrades to each device are handled by its vendor, not by the convenience store owner. Upgrades and maintenance are handled by each vendor on their own schedules; vendors may be understandably loath to spend time and effort changing the schemas in their deployed devices just to support an upgrade made by their competitor to a different device. In deploying a new version of the common schema (or a new set of value-added additions to it), therefore, a vendor cannot count on corresponding changes to the other devices.

1.3. Health records use case

Note: in some email discussions this has been known as the ‘Hoylen Sue’ use case, based on problems arising in the Titanium project at DSTC)[1]
Health record systems need to support many different types of health records. Different types of health records are used between different jurisdictions, organizations and even between different clinics. They also change over time as medical practice, regulations and legal requirements changes. A versioning mechanism is needed to manage these dimensions of change.
Clinical software that handles this dynamic open-world environment uses a two level data modelling approach. All software is designed to process data conforming to a generic reference schema. However, specialist applications would handle specific versions of the reference schema. In this text, the term "specialization" will be used to refer to these versions. These specializations constrain or restrict the reference model so that it is not generic, but specific to the clinical concept being represented.
Data is exchanged between a general practice (GP) clinic and a hospital. The GP software has the generic reference schema (G) and GP specific specializations of it (one of which is S1). The hospital software has the generic reference schema (G), hospital specific specializations of it (one of which is S2), and also has S1. Both S1 and S2 are versions of G. Instances of S1 and S2 need to be also schema valid according to G because they are specializations of it. For example, S1 could be a record of a clinical consultation and S2 a hospital pathology report.
A patient is referred to the hospital, and their GP doctor sends an extract of their health record to the hospital. The GP clinic sends an instance of S1 to the hospital. The hospital system validating it must reject it if it does not validate according to the rules in S1. It may satisfy the rules in G, but that is not sufficient because the hospital system has access to S1 and must validate using the extra constraints in S1.
The patient is discharged, and the hospital sends a discharge summary to the patient's GP doctor. The hospital sends an instance of S2 to the GP clinic. The GP system validating it must reject it if it violates the rules in G. Since the GP system does not have S2, it will not be able to apply the additional rules of S2. Although the GP system does not understand S2, it wants to store it in the patient's record and treat it generically.
A new clinical concept is developed in GP medicine, and a schema S3 (another specialization of G) is created to represent it. The GP software is configured with S3. When an instance of S3 is sent to the hospital, the hospital validates it using the rules in G because it does not have S3. Alternatively, the hospital may obtain a copy of the S3 schema (by an out-of-band means) and then it must use the rules in S3. Fetching the new schema is not always possible, so fall-back validation to G is a necessary feature of the versioning mechanism.

1.4. The ‘UBL’ use case

This is an abstraction and generalization of the schema development methods described in [Gregory / Gutentag 2002].
The schemas defining a particular language use a very specific discipline for type derivation, to allow software written for a particular version of the language to process data conforming to a later version, across minor (decimal-point) revisions of the language.
In a minor revision, all new types are derived by restriction or extension from types known in the base version. So if version 2.4 of the schema introduces a new type T1, it will be based on, and explicitly derived from, a type (say T2) present in version 2.3. Type T2 will in turn either be present in version 2.2 of the schema, or else it will be derived from some type T3, which is present there. And so on, back to the inventory of types defined in version 2.0 of the schema.
In each case, the semantics of the type derivation are such that an application which does not understand type T1 but does understand type T2 or T3 can process the instance successfully as if it were of the known type. That processing may not exploit all the advantages of the later type, but it will be correct as far as it goes.
At the root of the application-specific type hierarchy are a set of maximally loose type definitions which are designated as abstract and designed to allow virtually any subset of the defined children and attributes to be selected by a derived type.
All types are named.
When a new type cannot satisfy the successful-processing invariant, it cannot be introduced in a minor revision but must wait for a major revision.
Major revisions are not required to maintain these invariants: an application with hard-coded semantics from version 1.n will not necessarily understand data conforming to version 2.m, and so on.
Note: at present, this description resembles the description of a versioning technique more than a use case. The use case might be more properly described thus:
We have a application, which has full understanding of the elements, attributes, and types in some version n of a schema. The documents in question are intended to be legally binding, so the use of wildcards is strictly avoided by the schema designers. (Using wildcards would amount to asking users of the schema to undertake legal obligations without understanding them.)
When confronted with material valid against later versions of the schema (n + 1, n + 2, ...), the application must be in a position to
  • recognize situations in which it can process the later-version data correctly using its in-built knowledge of the semantics of version n — that is, the application must be in a position to recognize, given a type in schema version n + i, which version-n type it corresponds to, so that that version-n type can be used as a fallback.
  • recognize situations in which the in-built version-n semantics do not allow correct processing of the newer data, so that it can abort without doing further harm.
It may be assumed that the application is downstream from a schema-based validator, that the validator has access to the newer schema, and that the validator provides the application with access to the type hierarchy of the (later) schema. Since validity against a wildcard-free schema is a precondition of acceptance, the schema appropriate to the data must be acquired and used.

1.5. The ‘XML Schema’ use case

In discussing XML Schema 1.1, the XML Schema WG has rejected (or in some cases has been urged to reject, but has not yet finally decided to reject) various proposals on the grounds that they would create either a backward incompatibility (new software would process valid old data differently) or a forward incompatibility (old software would fail unacceptably on new data).
What kinds of mechanisms would, if they had been written into XML Schema 1.0, have permitted the Working Group to adopt these or similar proposals for change without unacceptable compatibility problems?
The goal of versioning for the XSD use case is to make it easier for the owners of the XSD language to make changes in the language without requiring that everyone involved in using XSD upgrade at the same time. This means new schemas should be processable in more or less acceptable ways by existing schema processors. This, in turn, means: the designers of version n+1 of XSD should be able to define constructs which will not be understood by old processors, without causing those old processors to fail. (Theoretically, they might wish to define new constructs which WILL cause existing processors to fail, but it is not obvious that there are any possible new constructs which should fail in that way. So in its current form, this discussion assumes that breaking changes are not part of the XSD use case.)
As examples, we consider several changes which a future WG designing version n+1 of XSD might wish to make; any resemblance to changes actually proposed for XSD is entirely the result of careful thought, but they are included here in order to illustrate a variety of examples, not as proposals for any particular version of XSD.
  1. New top-level element.
    The WG wishes to define a new top-level element called xsd:check. It is analogous to a table-level check clause in SQL, and contains a series of predicates (expressed by xsd:test elements), each of which must be true of the document as a whole. A version n+1 processor will know how to check the constraints it contains; a version n processor will not know how. The correct fallback behavior for a version n processor is to ignore the constraints and proceed without error (although it might issue a warning).
    A schema author might, however, prefer to say, in effect: if you can't process these constraints, die right now.
    A user might, in turn, wish to control whether the processor dies or soldiers on after encountering the unknown xsd:check element.
  2. New embedded element (I). Change of the syntax of content models. In addition to the 1.0 transfer syntax for content models, the WG wishes to allow a different XML transfer syntax. So where a version n processor expects an 'xsd:element', 'xsd:sequence', 'xsd:choice', or 'xsd:all' element, a version n+1 schema document may instead have a 'xsd:content' element.
    Since a version n processor will not understand how to interpret the xsd:content element, a schema author interested in co-existence with version n processors may wish to specify a version-n-style content model as a fallback. (If the main appeal of xsd:content is that it provides currently unavailable functionality, the v.n fallback will be only an approximation. If the main appeal is that xsd:content is easier to read or more elegant, it seems unlikely that authors will want to provide the fallback for schemas being edited. Once the schema is frozen, however, the old syntax could be added by hand or by machine for portability's sake.)
  3. New embedded element (II). Like the xsd:schema element itself, in v.n+1 the other top-level source declarations are also to be allowed to have xsd:check elements. The schema author may wish to provide fallbacks, where appropriate, using (say) key and keyref, which approximate the tests in the check clause and which can be performed by v.n processors.
    The proper behavior of a v.n processor is to perform the fallback validation if any is specified, and to ignore the check clause otherwise (possibly with a warning).
  4. Extension namespace labeling. Addition of an ‘extension-namespace-prefixes’ or ‘extension-namespaces’ attribute to the xsd:schema element.
    This attribute allows a schema author to declare that certain namespaces should be recognized as containing elements or attributes which are extensions to the XSD specification; these extensions may be recognized and processed by some but not by all conforming processors and should not cause an error. A v.n processor should ignore the attribute (although it might raise an error if it encounters actual extension elements in the schema document in places where it's not prepared to find unknown material).
Other sample changes are imaginable; this use case may need to be extended in future.
Solutions of this use case should say what versioning rules in XML Schema v.n will allow v.n schema-validity assessors to
  • ignore the ignorable xsd:check elements,
  • die on the xsd:content elements unless a fallback is given,
  • recognize and process the fallback constraints for un-understood xsd:check elements
  • recognize and process the fallback content models (for xsd:content elements)
  • ignore the extension-namespaces attribute
  • die on extension elements when necessary
  • ignore extension elements when desirable (preferably the judge here is to be the schema author)
  • allow the schema author to exert some (total?) control over when processors ignore ununderstood elements in the XSD or other namespaces, and when they die

1.6. The ‘XSLT’ use case

The XSLT 1.0 specification defines a particular versioning strategy: future versions of the specification will use the same namespace, and version attributes on the constructs of the language will indicate to a processor which version of the language applies to them.
Explicit fallback mechanisms (include a fallback element) are defined for processors implementing the defined semantics of the language.
Would XSLT or similar languages benefit by being able to describe these versioning mechanisms in a schema which defines the vocabulary? Would it be advantageous, for example, to have a schema-language construct which identifies a particular element (e.g. the one named xsl:fallback) as a fallback element? To have a schema-language construct which identified the version attribute as having a particular role?
Is there a finite repertoire of behaviors or semantic patterns which can usefully be captured in schema-language constructs? If not, is there a language in which such behaviors or patterns could be described?

1.7. The ‘MathML’ use case

What is the most convenient way to integrate new constructs (e.g. constructs which represent newly discovered mathematical constructs) into a specialized language like MathML? Is it possible to introduce new elements and attributes in such a way as to allow software which does not have hard-coded knowledge of them to do the right thing with them?

2. Proposed concepts and terminology

As an attempt at beginning the task of a conceptual analysis, we may think about the behavior of a program given a set of inputs. (This description speaks mostly in terms suitable for atomic or batch processes, not interactive ones, but it is hoped that this simplification is without fatal consequences.)
There are several ways we may profitably characterize inputs to a program / application / piece of software; the distinctions among them may help us describe versioning problems more precisely.

2.1. Universe of discourse

Let
U = the universe of all possible inputs.

2.2. Termination

In general, for some inputs a program will terminate without error; for others, the program will terminate with an error code (but gracefully), for others, the program will terminate gracelessly (or be terminated by the operating system), for others, the program will loop indefinitely. Let:
Z = the set of inputs for which the program terminates gracefully without returning an error code (the name Z may be thought of as meaning ‘with a return code of zero’)
E = the set of inputs for which the program terminates gracefully and returns an error code; ZE = ∅
G = the set of inputs for which the program terminates gracefully; we have G = ZE
B = the set of inputs for which the program terminates gracelessly (B may be thought of as denoting inputs which cause the program to ‘blow up’)
N = the set of inputs for which the program does not terminate at all
We have:
G, B, and N are pairwise disjoint.
Z and E are disjoint.
U = GBN

2.3. Domain, extended domain, and pathological cases

Any software written to implement a given process has, we postulate, a set of inputs for which it is able to calculate the correct result (or supposed to be able to do so). If you go to the developer and show a set of such inputs, together with an incorrect result, the developer will blush, and say 'Oops', and promise to fix it. Let us call these inputs the 'domain' of the application. (Thanks to Henry Thompson for suggesting this term.)
If we need to distinguish the set of inputs for which the program is supposed to be able to calculate the correct result from the set of inputs for which it can actually calculate the correct result, we may use the terms 'intended domain' and 'actual domain'. If the intended domain contains cases not in the actual domain, the program has a bug; if the converse is true, no harm is usually done, although it may be an indication that the program has been over-engineered.
Similarly, there are other inputs for which the program is not expected to calculate a useful or correct result. If you go to the developer with such input and say "The result was incorrect", the developer will say "Well, what did you expect?" and decline to view it as a bug. These inputs are outside the intended domain of the program.
For some inputs outside its (intended or actual) domain, the program will detect that the input is outside its domain, perhaps provide an error diagnostic of some more or less useful sort, and exit gracefully. This is precisely the set E defined above.
Let us use the term 'extended domain' for the set of inputs for which the program either calculates the correct and intended result, or provides an error diagnostic and a graceful exit.
Let
D = the domain of the program (inputs for which the program terminates with a correct answer)
DI = the intended domain of the program
DA = the actual domain of the program
X = the extended domain of the program
Note that D and DA are synonyms; the latter is used only to stress the contrast with DI.
We have:
DZ
X = DE
For each input in Z \ D, the program terminates without error but does not calculate a correct or useful result; for most software projects, such inputs mean there is a bug in the program.
For inputs outside the extended domain, the program may be taken wholly unawares, with the result that it may fail in some graceless or possibly spectacular way. In high-quality software, the developers' goal is perhaps usually to enlarge the extended domain until it covers the universe of all possible inputs (i.e. make X = U), so that no input at all will cause a graceless exit (B = ∅); if any input turns out to fall outside the extended domain, there is a bug in the software. In less ambitious software (particularly one-off programs written for private use), the developer may not care, and everything outside the domain may also fall outside the extended domain. The developer may also characterize all input outside the program's domain as "pathological cases". (Some good software is characterized by a determination to diagnose pathological cases and thus bring them into the extended domain.)
So, first of all, we can partition the set of all possible inputs into the domain D and its complement, or into the extended domain X and its complement.

2.4. Accept set, reject set

We can also distinguish the set of inputs which a program will accept and for which it will try to calculate a result (or more generally, which it will try to do something useful with), from the set of inputs a program will decline to try to process. Let us call the first set the program's 'accept set' and the second its 'reject set'. (These are presumably complements of each other.)
Let
A = inputs ‘accepted’ by the program
R = inputs rejected by the program.
We have
U = AR
AR = ∅
R = E (by their definitions)
A = ZBN
In common cases, the intention of the developers may be to accept all and only the inputs which are in the program's domain, and to reject everything else, namely everything outside the domain. If they are successul, we have A = D, R = U \ D, (and thus) B = N = (Z \ D) = ∅, (and further) U = X.
The difference between the intended and actual domains may be important here: if a program accepts everything in its intended domain, then any input which falls in its accept set but outside of its actual domain is liable to cause a failure. If a program erroneously accepts some input outside its intended domain (i.e. if A \ D is non-empty), then input which falls outside the intended domain but inside the actual domain will fail to raise an error condition, even though in theory it should have done so.

2.5. Valid, invalid, partially valid

If a schema or other formal characterization of the intended domain is available, then we may also distinguish valid input from other input. If the schema language supports a notion of partial validity (as does XML Schema), then we may also distinguish various kinds and degrees of validity. (See http://www.w3.org/XML/2001/06/validity-outcomes for a matrix.)
If for purposes of discussion we allow ourselves to limit consideration to XML input validated against an XSD schema, we can let
VF = inputs with [validity] = valid and [validation attempted] = full
VP = inputs with [validity] = valid and [validation attempted] = partial
V = inputs with [validity] = valid. We have V = VFVP.
IF = inputs with [validity] = invalid and [validation attempted] = full
IP = inputs with [validity] = invalid and [validation attempted] = partial
NP = inputs with [validity] = notKnown and [validation attempted] = partial
NN = inputs with [validity] = notKnown and [validation attempted] = none
To simplify life, we can treat inputs that haven't been validated at all as NN.
Validation with DTDs produces only two kinds of results: VF and IF: for DTD-based validation, VP = IP = NP = NN = ∅

2.6. Understanding, partial understanding, and misunderstanding

If we think of software as 'understanding' the inputs which fall into its domain, then one way to describe the challenging of versioning is to say that to achieve forward compatibility we must specify software so as to behave gracefully and usefully when it encounters data it understands only partially. This seems to mean two things:
  1. wherever possible to allow partially understood data to fall into the accept set whenever it also falls into the actual domain of the software, and
  2. to detect cases where partially understood data falls outside the actual domain, and to ensure that that data goes into the reject set, rather than into the accept set
When partially understood data fall inside the accept set but outside the actual domain, the result is less like partial understanding than like misunderstanding. The goal of point (a) is to maximize partial understanding and the goal of (b) is to avoid misunderstanding.
In some applications, the cost of misunderstanding is minor and the cost of a misguided rejection is high, so the simplest technique is to reject as little as possible. In other applications, the cost of misunderstanding might be so high that it swamps the cost of misguided rejections, so that the simplest technique is to accept nothing unless it is fully understood.

2.7. Some simple approximations

In some cases (it is not clear whether to call them simple cases, ideal cases, common cases, or what — surely they all have exceptions), the goals of those involved may include one or more of the following:
  • Make the set of valid documents a subset of the domain (E.g. develop a schema to detect in advance errors which might otherwise emerge only after expensive processing has already been done.) If we do this, we have V (or in some designs VF) ⊆ D; then if we set A = V, we are guaranteed that AD.
  • Make the domain a superset of the set of valid documents: DV. (E.g. given a schema, write software which will handle all valid documents; use the schema to remind yourself what you must be prepared for at any particular point in the input.)
  • Make the set of valid documents a superset of the domain, i.e. make VD. (E.g. develop a schema to weed out documents you cannot process, but never let the schema reject as invalid any input that might turn out to be in the domain. The application will run additional tests which cannot be expressed in the schema, or which could be but are not.)
  • Make the reject set a superset of the invalid set (R ⊃ (INNNP)). (I.e. reject anything that's not valid.)
In many cases, the purpose of validation is to aid application software in deciding whether to accept or reject the input, and the intention is to make the extended domain of the application contain every valid document (the combination of application plus validation then will behave like an application whose extended domain is the universe of all possible inputs and be more stable).
In different cases, however, the possible mismatches between the valid set and the domain or extended domain may have different tradeoffs:
  • inputs which fall inside the valid set but outside the extended domain (inputs in (V \ X)) may cause failures
  • inputs which fall inside the valid set and inside the extended domain, but outside the actual domain (i.e. inputs in ((VX) \ DA)), may cause wasted computation (if the non-schema validation is expensive)
  • inputs which fall outside the valid set but inside the actual domain (i.e. inputs in (D \ V)) may represent lost opportunities (the software could have handled them in fact, but declined to try because the inputs weren't schema valid)

2.8. Some versioning problems

It seems plausible that the distinctions introduced above ought to be usable to express versioning problems a bit more precisely than is sometimes the case. But a full development of this idea is left to a future version of this document. For now, we limit ourselves to illustrating a couple of assumptions we believe the draft TAG finding on versioning appears to make about the relations among some of these sets, and the possibility of different assumptions:
  • "The accept set is a subset of the valid set (i.e. no application accepts invalid or partially valid input)." In contrast, some members of the WG believe that a key tool for schema-based versioning is the ability of processors to accept partially valid documents.
  • "The actual domain is always a superset of the valid set" (if we can allow the v1 schema to accept v2 data, then the problem is solved; there isn't any need for the software to do anything in particular -- and specifically there is no danger that v2 inputs will ever fall outside the actual domain of v1 software, and no need to discuss how designers can go about ensuring that this is so). In contrast, we believe the schema-flexibility question is only part of the story, and not necessarily at all the most important, most interesting, or hardest part. Also important (and often much harder) is to define what default actions a version-N application can safely take on version-N+1 data, and how to tell when the default action isn't safe after all and should be avoided.

3. Versioning mechanisms

Several mechanisms have been suggested as potentially helpful in solving the versioning problems involved in the use cases described above. Some of those mechanisms are described below.

3.1. V-M1 Second-class wildcards

No UPA violations in cases of element/wildcard competition.

3.2. V-M2 Match info (element, wildcard)

Which (kind of) particle (wildcard, element, ...) did we match? Some WG members suggest labeling this “particle attribution”.

3.3. V-M3 Fallback from xsi:type value to declared element type

if an xsi:type in the instance names an unavailable type, allow the schema-validity assessor to use the declared type of the element instead. If the element is valid against a type legally derived from the declared type, then either the entire element instance will be valid against the declared type or a proper prefix of it will be valid; make it convenient to tell from the PSVI what prefix of the instance is valid (if any).

3.4. V-M4 Prefix validity

If a prefix of the children is valid, (so that assuming a second-class wildcard at the end would make things valid), specify in the PSVI that the element was prefix-valid (and indicate which prefix of the children was valid).

3.5. V-M5 ancestor prefixes (titration)

In a chain T1, T2, T3, ... Tn of correct complex-type derivations, an element instance valid against Tn invariably has a prefix (and subset of its attributes) valid against each Ti for 0 < in. Provide information in the PSVI to indicate (for each Ti) which prefix of the children (and which subset of the attributes) is valid against Ti.

3.6. V-M6 Version attribute

We immediately discovered two interpretations of this label:
  • a way to explicitly link multiple schemas (this was paraphrased or rephrased as: a way for a linear sequence of revisions by a single author to allow easy recognition of constructs which will not be understood by down-version implementations
  • a way to mark different parts of an XML instance as requiring different levels of understanding from the application code
These seem distinct, so we divided them.

3.7. V-M7 Schema version attribute

A way to explicitly link multiple schemas, or more elaborately a way for a linear sequence of revisions by a single author to allow easy recognition of constructs which will not be understood by down-version implementations

3.8. V-M8 Instance-based version attribute

A way to mark individual parts of an XML instance as requiring particular kinds of support from the processor.

3.9. V-M9 Allow augmentation of choice and all groups

Make it possible to add new members to choice-groups and all-groups, not just add new items at the end of a (top-level) sequence-group.

3.10. V-M10 Allow augmentation of enumerations

Make it possible to add new members to enumerations of legal values in a simple type.

3.11. V-M11 More robust code generation

What is sometimes reported by users as a versioning problem may be a problem not with their schemas, or with their schema language, but with their code.
The problem reported is frequently that the version-1.0 schema classifies documents which conform not to 1.0 but to 1.1 or a later version of the schema as invalid. In itself, however, this need not be a problem: there is no reason that applications cannot process partially valid or invalid documents. The problem appears in some cases to be that the application code dies when it sees a message which is not fully valid according to the 1.0 schema. This in turn reflects in some cases a decision at code generation time to generate code only for valid documents, rather than generating code which understands valid documents but also accepts partially valid and invalid documents.
In other words, the actual problem at the user level might well be soluble without any change to the 1.0 schema or the schema language, if code generators systematically generated code which gracefully handles invalid and partially valid input.
There may, of course, be ways to change the schema language so as to make it easier to generate code which is more robust in the face of invalid data.
(Note that although described in terms of “1.0” and “1.1” schemas, this reasoning obviously applies more generally to known schemas and different schemas which describe related or overlapping sets of documents.)

3.12. V-M12 A fallback element

(Note that the descriptions of XSLT and SVG below may be wrong in details; since the point is not to write conforming XSLT or SVG processors but to illustrate a possible mechanism for supporting versioning of vocabularies, it is hoped that any errors will be mostly harmless.)

3.12.1. XSLT

Variation among processors may be the result of (a) support for different versions of XSLT, (b) partial implementation supporting some but not all features of a particular version (e.g. during an iterative write-and-release development process), (c) proprietary or non-proprietary extensions to the language, (d) errors in processors and/or differences of opinion about the correct interpretation of the spec, leading to different behavior.
XSLT 1.0 defines two overlapping mechanisms for dealing with such variation: (1) the functions
  • element-available(QName) → boolean
  • function-available(QName) → boolean
  • system-property(QName) → any
allow stylesheets to interrogate the processor to find out whether a particular facility is available. In conjunction with conditional instructions like xsl:if and xsl:choose/xsl:when, they make it possible to avoid executing stylesheet instructions which would raise a dynamic error, or to select among different formulations to adapt to variation among processors. Note that all of the functions named can be used to handle both third-party extensions and new items (functions or XSLT instructions) in the XSLT namespace.
Michael Kay's book on XSLT gives this example of adjusting both to version changes in XSLT and to vendor extensions. The draft of XSLT 1.1 (later abandoned in favor of work on XSLT 2.0) allows temporary result trees to be processed by passing them to xsl:apply-templates as the value of its 'select' attribute; various vendor extensions do the same thing. The following template does the job for a 1.1 processor or any processor which supports any of several vendor extensions:
 <xsl:template name="process-tree-fragment"
  xmlns:msxml="urn:schemas-microsoft-com:xslt"
  xmlns:xt="http://www.jclark.com/xt"
  xmlns:saxon="http://icl.com/saxon">

  <xsl:param name="fragment"/>

  <xsl:choose>

   <xsl:when test="system-property('xsl:version') > '1.0'">
    <xsl:apply-templates mode="process-fragment" select="$fragment"/>
   </xsl:when>

   <xsl:when test="function-available('msxml:node-set')">
    <xsl:apply-templates mode="process-fragment"
select="msxml:node-set($fragment)"/>
   </xsl:when>

   <xsl:when test="function-available('xt:node-set')">
    <xsl:apply-templates mode="process-fragment"
select="xt:node-set($fragment)"/>
   </xsl:when>

   <xsl:when test="function-available('saxon:node-set')">
    <xsl:apply-templates mode="process-fragment"
select="saxon:node-set($fragment)"/>
   </xsl:when>

   <xsl:otherwise>
    <xsl:message terminate="yes">
     Cannot convert result tree fragment to node-set
    </xsl:message>
   </xsl:otherwise>
   
  </xsl:choose>
 </xsl:template>
A result tree fragment is passed in as the value of the parameter 'fragment'. If we can, we wish to call 'apply-templates' with the node set of that result-tree fragment as the set of nodes to process, which means we want to use an expression denoting that node set as the value of the 'select' attribute on the 'apply-templates' element. If the XSLT version number is greater than 1.0, then we just write '$fragment', since 1.1 made that legal. Otherwise, if the msxml:node-set function is available, we write 'msxml:node-set($fragment)', and similarly if the 'node-set' functions in the 'xt' or 'saxon' namespaces are available. If none is available, we issue an error message and die.
(2) The XSLT 1.0 spec defines a 'forward-processing' mode which can be described as follows. This is not an XSLT tutorial, so we have elided some details.
  • At any point in a stylesheet, there is an 'effective version' number. In the simple case, it's set using the 'version' attribute on the 'xsl:stylesheet' element; it can also be set for part of the stylesheet using the 'xsl:version' attribute on a literal result element. (Not, apparently, on individual templates or other constructs.) Normal usage will be that if a stylesheet uses features of XSLT N.M, it will have an effective version of N.M.
  • XSLT distinguishes four kinds of XML elements encountered in an XSLT stylesheet: top-level children of the 'stylesheet' element, literal result elements, XSLT instructions, and other elements in the XSLT namespace. Instructions are, roughly, the XSLT elements and extension elements which can occur in a template; other XSLT elements include the xsl:when and xsl:otherwise elements inside an xsl:choose instruction.
  • When a 1.0 processor encounters an unexpected element U, the action taken by a 1.0 processor (or, more declaratively, the 1.0 interpretation of the stylesheet) depends on the kind of unexpected element and the effective version at that point in the stylesheet:
    • If U is a top-level element, U is ignored.
    • If U is a literal result element, U is not unexpected (i.e. it cannot happen that we have an 'unexpected' literal result element).
    • If U is an instruction then:
      if the effective version is 1.0, an error is raised,
      else (the effective version is not 1.0), fallback processing occurs.
    • If U is another element in the XSLT namespace then an error is raised.
Fallback processing is quite simple:
If there is an xsl:fallback element appearing as a child of U, then all xsl:fallback elements appearing as children of U are instantiated; otherwise an error is raised.
In normal processing (i.e. when not falling back), the xsl:fallback element is silently ignored.
It may be seen that a stylesheet author thus has the ability to induce an XSLT 1.0 processor to do any of several things when it encounters an unknown instruction:
  • pass over it in silence. To do this, put an empty xsl:fallback element among its children:
           <xsl:new-feature>
             ...
             <xsl:fallback/>
           </xsl:new-feature>
    
  • raise an error (do not put a fallback element in, or write version="1.0" on the stylesheet element)
  • terminate. To do this, place an XSLT construct inside the fallback element which causes immediate termination, such as
           <xsl:new-feature>
             ...
             <xsl:fallback>
              <xsl:message>Your processor doesn't have 'new-feature'
    enabled?</xsl:message>
              <xsl:message>You've got to be kidding me.</xsl:message>
              <xsl:message terminate="yes">[Thud.]</xsl:message>
             </xsl:fallback>
           </xsl:new-feature>
    
  • perform some useful alternative processing. To do this, write the fallback using XSLT 1.0 constructs which approximate what is desired. E.g. let us suppose that XSLT 6.0 introduces a 'copy-to-output' instruction which copies a file directly to the XSLT output. We can do much the same using the 'document' function and 'copy-of', but perhaps 'copy-to-output' is typically much faster, so we prefer to use it if we can:
             <xsl:template>
              <div xsl:version="6.0">
               <xsl:copy-to-output href="boilerplate.xml">
                <xsl:fallback>
                 <xsl:copy-of select="document('boilerplate.xml')"/>
                </xsl:fallback>
               </xsl:copy-to-output>
              </div>
             </xsl:template>
    
Note that for some kinds of versioning- or extension-related changes in vocabulary, the stylesheet author has no control. If a new construct is introduced as a top-level child of the xsl:stylesheet element, a 1.0 processor will ignore the construct, period. If a new construct appears not as an instruction but in a more restricted context (e.g. within an existing construct like choose/when/otherwise), a 1.0 processor will raise an error.

3.12.2. SVG

SVG has some similar constructs.
The 'svg' element has 'baseProfile' and 'version' attributes. (How these interact with the other constructs described here will not be explained here.)
The SVG 'switch' element allows an SVG document to signal that certain content should be processed only if the processor supports a given named feature. The description in the SVG 1.1 spec is clear and concise:
SVG contains a 'switch' element along with attributes requiredFeatures, requiredExtensions and systemLanguage to provide an ability to specify alternate viewing depending on the capabilities of a given user agent or the user's language.
...
Attributes requiredFeatures, requiredExtensions and systemLanguage act as tests and return either true or false results. The 'switch' renders the first of its children for which all of these attributes test true. If the given attribute is not specified, then a true value is assumed.
For example:
  <svg:switch>
    <svg:g ... requiredFeatures=" 
        http://www.w3.org/TR/SVG11/feature#OpacityAttribute
         http://www.w3.org/TR/SVG11/feature#XlinkAttribute">
     ...
    </svg:g>
    <svg:g ... requiredExtension="http://example.org/xsvg#newFeature">
     ...
    </svg:g>
    <svg:g><!--* ultimate fallback *-->
     ...
    </svg:g>
  </svg:switch>
Note that since the test attributes default to 'true', omitting them all (as in the last 'g' element above) has the effect of an 'otherwise' clause.
SVG processors are required to tolerate and ignore elements and attributes in other namespaces; the svg:foreignObject element can be used to encapsulate extension elements in other namespaces which should be processed rather than ignored; it will typically be used in conjunction with a 'switch' element.

3.12.3. What's in it for us?

The facilities of XSLT and SVG suggest several other techniques we should consider:
  • a 'fallback' element analogous to that of XSLT, to fire when its parent is not a well understood construct
  • explicit identification of features / system properties, so that schema documents can test for them
  • some sort of conditional construct analogous to the 'choose/when' of XSLT (too general, probably, for us) or the 'switch' of SVG. (Like XSD, SVG is not a programming language and has no need of a fully general select statement.)
  • explicit policies enabling extension constructs (expressed as elements in other namespaces) at the top level and/or elsewhere in a schema document; the xsd:annotation element is clearly felt by many as a kind of ghetto and doesn't seem to get the job done, perhaps because of the term 'annotation'.
  • a way to induce a processor to terminate schema-validity assessment immediately, or to decline to start, so that schema authors can indicate when a down-level processor should NOT try to muddle through

3.13. V-M13 Auto-insertion of wildcards

Allow, or require, the automatic insertion of wildcards into content models, so that the resulting content model allows unrecognized elements at any point. A model declared as a sequence of my:a, my:b, my:c, for example:
  <xsd:sequence>
   <xsd:element ref="my:a"/>
   <xsd:element ref="my:b"/>
   <xsd:element ref="my:c"/>
  </xsd:sequence>
would automatically be transformed into something like the following:
  <xsd:sequence>
   <xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
   <xsd:element ref="my:a"/>
   <xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
   <xsd:element ref="my:b"/>
   <xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
   <xsd:element ref="my:c"/>
   <xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
  </xsd:sequence>
Note that in XML Schema 1.0 this latter content model is illegal because an initial my:a element could match either the element particle or the preceding wildcard. As a result, this proposal is frequently linked to proposals for weak wildcards.

3.14. V-M14 Typed wildcards

Allow a content model to contain particles which match a element with any name (generic identifier) at all, as long as the element has a particular type.

3.15. V-M15 Negative wildcards

Allow wildcards to exclude a specific set of namespaces (so that a wildcard could express the complement of the following choice:
   <xsd:choice>
    <xsd:any namespace="ns1" minOccurs="0" maxOccurs="unbounded"/>
    <xsd:any namespace="ns2" minOccurs="0" maxOccurs="unbounded"/>
    <xsd:any namespace="ns3" minOccurs="0" maxOccurs="unbounded"/>
   </xsd:choice>
Similarly, allow wildcards to match any elements in a particular namespace except those which match any of an enumerated list of local names. Make it convenient to express notions like “match anything in the current namespace except elements which appear explicitly in this content model”.
See also V-M19 below.

3.16. V-M16 Fallback to expected element type

If the element in the instance does not match any of the expected particles in a content model, and if there is only one element type expected, allow the schema processor to attempt to validate the instance element against the expected element declaration. So, for example, if the content model expects my:a, my:b, my:c, and the instance has my:a, my:b, my:x, then perhaps the instance conforms to a variant schema in which my:x is in the substitution group of my:c. If that is so, then the type of my:x should be derived from that assigned to my:c; if derived by restriction, then it should be legal against the type associated with my:c; if by extension, then a prefix of it should be legal against the type associated with my:c.

3.17. V-M17 Explicit relations among schema documents

Provide mechanisms for asserting relations between the languages recognized by the constructs in different schema documents, e.g. “the language recognized here is a subset of the language recognized by X”, or “... is a superset of ...”, or “... is a later version of ...”, etc.

3.18. V-M18 Explicit relations among schemas

As for the preceding, but make the facilities operate at the component level, rather than the source-declaration level.

3.19. V-M19 Wildcard that matches anything not declared in this schema

A specific variant of V-M15.

3.20. V-M20 Fallback attribute in instance

Allow an instance document to carry attributes (similar to xsi:type) which specify what declarations to use if the schema in use doesn't have the expected ones. The fallback attribute might specify an alternative element declaration, but perhaps will be more useful if it specifies a type definition. A sequence of type definition names might be provided, to be tried in sequence.
An element might then appear in the instance as
<my:e xsi:fallback="my:t1 my:t2 my:t3">...</my:e>
with the result that the schema processor would perform fallback processing as follows:
  1. If the schema in use has a declaration for element my:e, which assigns type my:t0 to it, then the instance element will be validated against the declaration of my:e and the definition of my:t0 in the usual way.
  2. Otherwise, if the schema in use has a declaration for element my:e, which assigns type my:t0 to it, but the schema lacks any definition of my:t0, then the instance element will be validated against the declaration of my:e and either the definition of my:t1, or that of my:t2, or that of my:t3, using the first one found.
  3. Otherwise, if the schema in use has no declaration for element my:e, then the instance element will be validated against either the definition of my:t1, or that of my:t2, or that of my:t3, using the first one found.

3.21. V-M21 Fallback validating using local declarations in preference to globals

In 1.0, associations between element instances and local element declarations are only made in the case where the parent element is locally valid (or more specifically only in cases where the parent's children match the parent's content model). Otherwise, either the element is marked invalid and its children are not validated at all, or (at processor option) the element is validated laxly, which means the children are validated if and only if they match some top-level element declaration.
Change this to allow / require fallback processing to use the local declarations in preference to global ones.

3.22. Two schemas

One possible approach would be to define two schemas, one which defines the set of fully-understood documents and one which is used to define the accept set. The former might use only fully-specified and fully-understood elements and attributes and have no wildcards; the latter might be derived from the former by an algorithmic process (e.g. by adding wildcards at specified locations).

3.23. Sandboxes

One way to allow extensions is to provide a wrapper element for extensions; the draft TAG finding provides a good account of this. Effectively, a version n processor simply ignores everything in the extensions element.
An important drawback is that after repeated revisions a schema and its instances may begin to look either rather onion-like (with a version-n+1 extension element nested inside the version-n extension element), or else like a statement with a chain of afterthoughts (with the version-n+1 extension element following the version-n extension element). Either way, generators of documents which must conform to later versions must keep track of which version something was added in (“Quick, does middlename belong in the v1-extensions element or in the v2-extensions element?”).

3.24. Namespace filtering

Some schema authors support extensibility by requiring that all extensions be in the extender's namespace, not that of the base schema. This can be done either by inserting ##other wildcards at appropriate places in the content models to allow such extensions everywhere, or by using these rules for validation:
  1. Traverse the document tree, removing everything which is not in the namespace defined by the base schema.
  2. Validate.

3.25. Double validation

One way to allow for arbitrary extensions:
  1. Use top-level elements. To be more precise: in any complex type which you wish to allow to be extended arbitrarily, do not use any local element declarations; use only references to top-level element declarations. (Syntactically, use <xsd:element ref="..."/>, not <xsd:element name="..."/>.)
  2. Use the run-time option which tells your schema processor to fall back to lax processing if it cannot validate an element's children against its type.
  3. Validate once. If the part of the document you care about is fully validated and valid, you are done. Otherwise, continue.
  4. Traverse the document tree, removing everything which is invalid or has [validity=notKnown].
  5. Validate a second time. If the part of the document you care about is fully validated and valid, accept the document; otherwise reject it.
Note that namespace filtering is a special case of this technique.

4. Evaluation of schemas

This section to be supplied.

5. Syntax

This section to be supplied.

6. Using versioning to solve the use cases

This section to be supplied.

7. Explicit relations between versions

This section to be supplied.

References

Gregory, Arofan, and Eduardo Gutentag. “XSD Type Derivation and the UBL Context.” Paper given at XML 2002, Baltimore, December 2002. http://www.idealliance.org/papers/xml02/dx_xml02/papers/05-05-06/05-05-06.html

Gregory, Arofan, and Eduardo Gutentag. “XML Schema Libraries and Versioning: The UBL Case.” Paper given at XML Europe 2003, London, May 2003. http://www.idealliance.org/papers/dx_xmle03/html/abstract/03-04-03.html

Gutentag, Eduardo, and Arofan Gregory. “XML-based Rules: Automating Business Context Categorization to Produce Semantic Interoperability: The rules-based ebXML approach.” Paper given at Extreme Markup Languages 2001, Montréal, August 2001. http://www.idealliance.org/papers/extreme03/html/2001/Gutentag01/EML2001Gutentag01.html

Maler, Eve. “Substitutability Requirements for Different Applications”. 7 April 1999. http://www.w3.org/XML/Group/1999/08/s14y-paper

Maler, Eve, Ashok Malhotra, Allen Brown, et al. “Refinement Task Force Report”. 15 September 1999. http://www.w3.org/XML/Group/1999/07/schema-refine/refinement-taskforce-report.htm

Mendelsohn, Noah. “Some thoughts on versioning (for f2f reading list?)” Email to XML Schema IG, 13 January 2004. http://lists.w3.org/Archives/Member/w3c-xml-schema-ig/2004Jan/0051.html

Orchard, David, and Normal Walsh, ed., “[Editorial Draft] Versioning XML Languages”. Proposed TAG Finding 16 November 2003. http://www.w3.org/2001/tag/doc/versioning.html

Orchard, David. “Web services = or != distributed objects?” (12 January 2004). http://www.pacificspirit.com/blog/2004/01/12/web_services_or_distributed_objects

Orchard, David. “Examining wildcards for versioning” (8 January 2004). http://www.pacificspirit.com/Authoring/Compatibility/ExaminingElementWildcardSiblings.html

Orchard, David. “Providing Compatible Schema Evolution” (19 January 2004). http://www.pacificspirit.com/Authoring/Compatibility/ProvidingCompatibleSchemaEvolution.html


Notes

[1] This description was drafted by Hoylen Sue, to whom thanks from the editor.