This is an attempt to define a conceptual framework for discussions
of versioning in the context of XML Schema.
1. Motivating use cases
Note: a fuller account of some of these use cases
may be found in
another document of this
Working Group. The descriptions there supersede those given
here.
1.1. The ‘local customization’ / invoice use case
(Note: in some email threads this is sometimes referred to as
the ‘Lee Buck’ use case.) The original description
was posted to the XML Schema WG mailing list in October 1999; it is
reproduced below. Note that the description predates the introduction of
redefine into XML Schema 1.0.
CoolToys Inc. receives a notice from MegaRetail Inc. that
henceforth they will accept only invoices that conform the the
supplied schema “MegaInvoice.xsd”. Rather than just
bolting on an emitter of MegaInvoices, CoolToys gets
‘religion’ and decides to standardize its invoice
on MetaInvoice.xsd. They have an additional piece of information that
they care about (in addresses as we'll see), but thanks to the wonders
on refinement they are confident that they will be able to create
CoolInvoice.xsd, refine the necessary archetype and be done.
Conceptually what they have to do is:
- Create a new schema named "CoolInvoice.xsd"
- Reference MegaInvoices.xsd
- Refine the type Address with SubAddress
- Require that anywhere address appears in MegaInvoice that SubAddress must
appear instead.
At first blush it seems like refinement made this much easier, they
can leverage the work MegaRetail did on MegaInvoicexsd and make a
modest ‘delta’ schema which references
MegaInvoice.xsd and makes a small change...
But that's not how it works... they will have to re-define every archetype
that includes Address in its content model to restrict it to SubAddress...
and then having done that, they'll need to change every archetype that
refers to the archetypes that refer to Address, ..., all the way up the
heirarchy. (And they won't even be able to use refinement for these
changes.) By the time they are done all that they will have essentially
created a shadow schema with virtually no association with the old one. And
they may be left scratching their head as to what the value of refinement
is!
ps: The above scenario might equally be cast as version 1 and version 2 of
MegaInvoices.xsd
1.2. The ‘convenience store’ use case
Note: in some discussions, this use case has also
been known as the ‘David Ezell’ use case.
Consider a convenience store, with an electronic cash register,
gas pumps, and car wash.
The computer systems in the cash register, gas pumps, and car wash
are built and sold by different vendors, but they must cooperate:
the peripherals send transaction data to the cash register, and
the cash register sends data of various kinds (price, prompt message
text, rules for offering a discounted car wash to customers who
purchase a certain amount of gasoline, etc.).
In order to provide for some interoperation among devices,
vendor-neutral organizations define schemas which provide a sort of
neutral common ground. Because the vendors compete for sales, it is to
each vendor's advantage to add value by providing more useful
information than the other vendor's product; this takes the form of
adding new subelements or attributes to items in the common schema.
Because the devices must cooperate, dropped messages are a
potential problem, and if one device rejects a message sent by another
device, the result is likely to be a discussion between the vendors
about whose device is at fault: did the gas pump emit a bad message,
or did the cash register wantonly reject a perfectly legal data
stream? In order to make such disputes relatively easy to resolve,
it is helpful to have a written description of the set of messages
the devices are required to accept. This description takes the
form of a schema document: that is, the accept set is defined
as the set of documents valid against a particular schema. (Either
VF or VP will do, although VP
may involve some further stipulations about how full or partial
the validation has to be.)
Thus any approach to versioning which involves accepting
partially valid documents would seem to involve abandoning the role
of the schema as defining the contract between sender and
recipient; that would leave the vendors without a convenient way to
adjudicate disputes over dropped data and is thus unacceptable in
practice.
For security reasons, the convenience store devices do not load new
schemas dynamically; schemas may be changed as part of an upgrade or
as part of system maintenance, but the upgrades to each device are
handled by its vendor, not by the convenience store owner. Upgrades
and maintenance are handled by each vendor on their own schedules;
vendors may be understandably loath to spend time and effort changing
the schemas in their deployed devices just to support an upgrade made
by their competitor to a different device. In deploying a new
version of the common schema (or a new set of value-added additions
to it), therefore, a vendor cannot count on corresponding changes
to the other devices.
1.3. Health records use case
Note: in some email discussions this has been known
as the ‘Hoylen Sue’ use case, based on
problems arising in the Titanium project at DSTC)[
1]
Health record systems need to support many different types
of health records. Different types of health records are
used between different jurisdictions, organizations and even
between different clinics. They also change over time as
medical practice, regulations and legal requirements
changes. A versioning mechanism is needed to manage these
dimensions of change.
Clinical software that handles this dynamic open-world
environment uses a two level data modelling approach. All
software is designed to process data conforming to a generic
reference schema. However, specialist applications would
handle specific versions of the reference schema. In this
text, the term "specialization" will be used to refer to
these versions. These specializations constrain or restrict
the reference model so that it is not generic, but specific
to the clinical concept being represented.
Data is exchanged between a general practice (GP) clinic and
a hospital. The GP software has the generic reference schema
(G) and GP specific specializations of it (one of which is
S1). The hospital software has the generic reference schema
(G), hospital specific specializations of it (one of which
is S2), and also has S1. Both S1 and S2 are versions of
G. Instances of S1 and S2 need to be also schema valid
according to G because they are specializations of it. For
example, S1 could be a record of a clinical consultation and
S2 a hospital pathology report.
A patient is referred to the hospital, and their GP doctor
sends an extract of their health record to the hospital.
The GP clinic sends an instance of S1 to the hospital. The
hospital system validating it must reject it if it does not
validate according to the rules in S1. It may satisfy the
rules in G, but that is not sufficient because the hospital
system has access to S1 and must validate using the extra
constraints in S1.
The patient is discharged, and the hospital sends a discharge
summary to the patient's GP doctor. The hospital sends an
instance of S2 to the GP clinic. The GP system validating it
must reject it if it violates the rules in G. Since the GP
system does not have S2, it will not be able to apply the
additional rules of S2. Although the GP system does not
understand S2, it wants to store it in the patient's record
and treat it generically.
A new clinical concept is developed in GP medicine, and a
schema S3 (another specialization of G) is created to
represent it. The GP software is configured with S3. When
an instance of S3 is sent to the hospital, the hospital
validates it using the rules in G because it does not have
S3. Alternatively, the hospital may obtain a copy of the S3
schema (by an out-of-band means) and then it must use the
rules in S3. Fetching the new schema is not always possible,
so fall-back validation to G is a necessary feature of the
versioning mechanism.
1.4. The ‘UBL’ use case
This is an abstraction and generalization of the schema
development methods described in
[
Gregory / Gutentag 2002].
The schemas defining a particular language
use a very specific discipline for type derivation,
to allow software written for a particular version of the
language to process data conforming to a later version,
across minor (decimal-point) revisions of the language.
In a minor revision, all new types are derived by
restriction or extension from types known in the base version.
So if version 2.4 of the schema introduces a new type
T1, it will be based on, and explicitly derived
from, a type (say T2) present in version 2.3.
Type T2 will in turn either be present in version 2.2 of
the schema, or else it will be derived from some type
T3, which is present there. And so on, back to the
inventory of types defined in version 2.0 of the schema.
In each case, the semantics of the type derivation are
such that an application which does not understand type
T1 but does understand type T2 or T3 can process the
instance successfully as if it were of the known type. That
processing may not exploit all the advantages of the later
type, but it will be correct as far as it goes.
At the root of the application-specific type hierarchy are
a set of maximally loose type definitions which are
designated as abstract and designed to allow virtually any
subset of the defined children and attributes to be selected
by a derived type.
All types are named.
When a new type cannot satisfy the successful-processing
invariant, it cannot be introduced in a
minor revision but must wait for a major revision.
Major revisions are not required to maintain these
invariants: an application with hard-coded semantics
from version 1.n will not necessarily understand data
conforming to version 2.m, and so on.
Note: at present, this description resembles the
description of a versioning technique more than a use
case. The use case might be more properly described thus:
We have a application, which has full understanding of
the elements, attributes, and types in some version n of
a schema. The documents in question are intended to be
legally binding, so the use of wildcards is strictly
avoided by the schema designers. (Using wildcards would
amount to asking users of the schema to undertake legal
obligations without understanding them.)
When confronted with material valid against later
versions of the schema (
n + 1,
n + 2, ...), the application
must be in a position to
- recognize situations in which it can process the
later-version data correctly using its
in-built knowledge of the semantics of version n — that is,
the application must be in a position to recognize, given a
type in schema version n + i, which version-n type it
corresponds to, so that that version-n type can be used as
a fallback.
- recognize situations in which the in-built version-n
semantics do not allow correct processing of the newer data,
so that it can abort without doing further harm.
It may be assumed that the application is downstream from a
schema-based validator, that the validator has access to the
newer schema, and that the validator provides the application
with access to the type hierarchy of the (later) schema.
Since validity against a wildcard-free schema is a precondition
of acceptance, the schema appropriate to the data
must be acquired and used.
1.5. The ‘XML Schema’ use case
In discussing XML Schema 1.1, the XML Schema WG has rejected
(or in some cases has been urged to reject, but has not yet
finally decided to reject)
various proposals on the grounds that they would create either
a backward incompatibility (new software would process valid old
data differently) or a forward incompatibility (old software
would fail unacceptably on new data).
What kinds of mechanisms would, if they had been written into
XML Schema 1.0, have permitted the Working Group to adopt these
or similar proposals for change without unacceptable compatibility
problems?
The goal of versioning for the XSD use case is to
make it easier for the owners of the XSD language to make changes
in the language without requiring that everyone involved in using
XSD upgrade at the same time. This means new schemas should be
processable in more or less acceptable ways by existing schema
processors. This, in turn, means: the designers of version n+1
of XSD should be able to define constructs which will not be
understood by old processors, without causing those old
processors to fail. (Theoretically, they might wish to define
new constructs which WILL cause existing processors to fail, but
it is not obvious that there are any possible new constructs which should fail
in that way. So in its current form, this discussion assumes that breaking changes
are not part of the XSD use case.)
As examples, we consider several changes
which a future WG designing version
n+1 of XSD might wish to
make; any resemblance to changes actually proposed for XSD is
entirely the result of careful thought, but they are included
here in order to illustrate a variety of examples, not as
proposals for any particular version of XSD.
New top-level element.
The WG wishes to define a new top-level element called xsd:check.
It is analogous to a table-level check clause in SQL, and
contains a series of predicates (expressed by xsd:test elements),
each of which must be true of the document as a whole. A version
n+1 processor will know how to check the constraints it contains;
a version n processor will not know how. The correct fallback
behavior for a version n processor is to ignore the constraints
and proceed without error (although it might issue a warning).
A schema author might, however, prefer to say, in effect: if you
can't process these constraints, die right now.
A user might, in turn, wish to control whether the processor dies
or soldiers on after encountering the unknown xsd:check element.
-
New embedded element (I). Change of the syntax of content
models. In addition to the 1.0 transfer syntax for content
models, the WG wishes to allow a different XML transfer syntax.
So where a version n processor expects an 'xsd:element',
'xsd:sequence', 'xsd:choice', or 'xsd:all' element, a version n+1
schema document may instead have a 'xsd:content' element.
Since a version n processor will not understand how to interpret
the xsd:content element, a schema author interested in
co-existence with version n processors may wish to specify a
version-n-style content model as a fallback. (If the main appeal
of xsd:content is that it provides currently unavailable
functionality, the v.n fallback will be only an approximation.
If the main appeal is that xsd:content is easier to read or more
elegant, it seems unlikely that authors will want to provide the
fallback for schemas being edited. Once the schema is frozen,
however, the old syntax could be added by hand or by machine for
portability's sake.)
-
New embedded element (II). Like the xsd:schema element
itself, in v.n+1 the other top-level source declarations are also
to be allowed to have xsd:check elements. The schema author may
wish to provide fallbacks, where appropriate, using (say) key and
keyref, which approximate the tests in the check clause and which
can be performed by v.n processors.
The proper behavior of a v.n processor is to perform the fallback
validation if any is specified, and to ignore the check clause
otherwise (possibly with a warning).
-
Extension namespace labeling.
Addition of an
‘extension-namespace-prefixes’ or
‘extension-namespaces’ attribute to
the xsd:schema element.
This attribute allows a schema author to declare that certain
namespaces should be recognized as containing elements or
attributes which are extensions to the XSD specification; these
extensions may be recognized and processed by some but not by all
conforming processors and should not cause an error. A v.n
processor should ignore the attribute (although it might raise an
error if it encounters actual extension elements in the schema
document in places where it's not prepared to find unknown
material).
Other sample changes are imaginable; this use case may need to be
extended in future.
Solutions of this use case should say what versioning rules in
XML Schema v.
n will allow v.
n schema-validity assessors to
- ignore the ignorable xsd:check elements,
- die on the xsd:content elements unless a fallback is given,
- recognize and process the fallback constraints for
un-understood xsd:check elements
- recognize and process the fallback content models (for
xsd:content elements)
- ignore the extension-namespaces attribute
- die on extension elements when necessary
- ignore extension elements when desirable (preferably the
judge here is to be the schema author)
- allow the schema author to exert some (total?) control
over when processors ignore ununderstood elements in the
XSD or other namespaces, and when they die
1.6. The ‘XSLT’ use case
The XSLT 1.0 specification defines a particular versioning
strategy: future versions of the specification will use
the same namespace, and version attributes on
the constructs of the language will indicate to a processor
which version of the language applies to them.
Explicit fallback mechanisms (include a fallback
element) are defined for processors implementing the
defined semantics of the language.
Would XSLT or similar languages benefit by being able to
describe these versioning mechanisms in a schema which
defines the vocabulary? Would it be advantageous, for example,
to have a schema-language construct which identifies a
particular element (e.g. the one named xsl:fallback)
as a fallback element? To have a schema-language construct
which identified the version attribute as having
a particular role?
Is there a finite repertoire of behaviors or semantic patterns
which can usefully be captured in schema-language constructs?
If not, is there a language in which such behaviors or patterns
could be described?
1.7. The ‘MathML’ use case
What is the most convenient way to integrate new constructs
(e.g. constructs which represent newly discovered mathematical
constructs) into a specialized language like MathML? Is
it possible to introduce new elements and attributes in such
a way as to allow software which does not have hard-coded
knowledge of them to do the right thing with them?
2. Proposed concepts and terminology
As an attempt at beginning the task of a conceptual analysis, we
may think about the behavior of a program given a set of
inputs. (This description speaks mostly in terms suitable
for atomic or batch processes, not interactive ones, but it is
hoped that this simplification is without fatal consequences.)
There are several ways we may profitably characterize inputs to a
program / application / piece of software; the distinctions among them
may help us describe versioning problems more precisely.
2.1. Universe of discourse
Let
U = the universe of all possible inputs.
2.2. Termination
In general, for some inputs a program will terminate without error;
for others, the program will terminate with an error code (but
gracefully), for others, the program will terminate gracelessly (or be
terminated by the operating system), for others, the program will loop
indefinitely. Let:
Z = the set of inputs for which the program
terminates gracefully without returning an error code (the
name Z may be thought of as meaning
‘with a return code of zero’)
E = the set of inputs for which the program
terminates gracefully and returns an error code;
Z ∩ E = ∅
G = the set of inputs for which the program
terminates gracefully; we have
G = Z
∪ E
B = the set of inputs for which the program
terminates gracelessly (B may be thought of as
denoting inputs which cause the program to ‘blow up’)
N = the set of inputs for which the program
does not terminate at all
We have:
G, B, and N are
pairwise disjoint.
Z and E are disjoint.
U = G ∪
B ∪ N
2.3.
Domain, extended domain, and pathological cases
Any software written to implement a given process has, we postulate, a
set of inputs for which it is able to calculate the correct result (or
supposed to be able to do so). If you go to the developer and show a
set of such inputs, together with an incorrect result, the developer
will blush, and say 'Oops', and promise to fix it. Let us call these
inputs the 'domain' of the application. (Thanks to Henry Thompson for
suggesting this term.)
If we need to distinguish the set of inputs for which the program is
supposed to be able to calculate the correct result from the set of
inputs for which it can actually calculate the correct result, we may
use the terms 'intended domain' and 'actual domain'. If the intended
domain contains cases not in the actual domain, the program has a bug;
if the converse is true, no harm is usually done, although it may be
an indication that the program has been over-engineered.
Similarly, there are other inputs for which the program is not
expected to calculate a useful or correct result. If you go to the
developer with such input and say "The result was incorrect", the
developer will say "Well, what did you expect?" and decline to view it
as a bug. These inputs are outside the intended domain of the
program.
For some inputs outside its (intended or actual) domain, the program
will detect that the input is outside its domain, perhaps provide an
error diagnostic of some more or less useful sort, and exit
gracefully. This is precisely the set E defined
above.
Let us use the term 'extended domain' for the set of inputs for which
the program either calculates the correct and intended result, or
provides an error diagnostic and a graceful exit.
Let
D = the domain of the program (inputs for which
the program terminates with a correct answer)
DI = the intended domain of the program
DA = the actual domain of the program
X = the extended domain of the program
Note that
D and
DA are synonyms;
the latter is used only to stress the contrast with
DI.
We have:
D ⊆ Z
X = D ∪ E
For each input in Z \ D,
the program terminates without error but does not calculate
a correct or useful result; for most software projects,
such inputs mean there is a bug in the program.
For inputs outside the extended domain, the program may be taken
wholly unawares, with the result that it may fail in some graceless or
possibly spectacular way. In high-quality software, the developers'
goal is perhaps usually to enlarge the extended domain until it covers
the universe of all possible inputs (i.e. make X =
U), so that no input at all will cause a graceless exit
(B = ∅); if any input turns out to fall
outside the extended domain, there is a bug in the software. In less
ambitious software (particularly one-off programs written for private
use), the developer may not care, and everything outside the domain
may also fall outside the extended domain. The developer may also
characterize all input outside the program's domain as "pathological
cases". (Some good software is characterized by a determination to
diagnose pathological cases and thus bring them into the extended
domain.)
So, first of all, we can partition the set of all possible inputs into
the domain D and its complement,
or into the extended domain X and its
complement.
2.4.
Accept set, reject set
We can also distinguish the set of inputs which a program will accept
and for which it will try to calculate a result (or more generally,
which it will try to do something useful with), from the set of inputs
a program will decline to try to process. Let us call the first set
the program's 'accept set' and the second its 'reject set'. (These
are presumably complements of each other.)
Let
A = inputs ‘accepted’ by the
program
R = inputs rejected by the program.
We have
U = A ∪ R
A ∩ R = ∅
R = E (by their definitions)
A = Z ∪ B ∪ N
In common cases, the intention of the developers may be to accept all
and only the inputs which are in the program's domain, and to reject
everything else, namely everything outside the domain.
If they are successul, we have
A = D,
R = U \ D,
(and thus) B = N = (Z \ D)
= ∅,
(and further) U = X.
The
difference between the intended and actual domains may be important
here: if a program accepts everything in its intended domain, then any
input which falls in its accept set but outside of its actual domain
is liable to cause a failure. If a program erroneously accepts some
input outside its intended domain (i.e. if A \ D is
non-empty), then input which falls outside the
intended domain but inside the actual domain will fail to raise an
error condition, even though in theory it should have done so.
2.5.
Valid, invalid, partially valid
If a schema or other formal characterization of the intended domain is
available, then we may also distinguish valid input from other input.
If the schema language supports a notion of partial validity (as does
XML Schema), then we may also distinguish various kinds and degrees of
validity. (See
http://www.w3.org/XML/2001/06/validity-outcomes for a
matrix.)
If for purposes of discussion we allow ourselves to limit
consideration to XML input validated against an XSD schema,
we can
let
VF = inputs with
[validity] = valid
and
[validation attempted] = full
VP = inputs with
[validity] = valid
and
[validation attempted] = partial
V = inputs with
[validity] = valid. We have V = VF ∪ VP.
IF = inputs with
[validity] = invalid
and
[validation attempted] = full
IP = inputs with
[validity] = invalid
and
[validation attempted] = partial
NP = inputs with
[validity] = notKnown
and
[validation attempted] = partial
NN = inputs with
[validity] = notKnown
and
[validation attempted] = none
To simplify life, we can treat inputs that haven't been validated
at all as
NN.
Validation with DTDs produces only two kinds of results:
VF and IF: for DTD-based validation,
VP = IP = NP = NN
= ∅
2.6.
Understanding, partial understanding, and misunderstanding
If we think of software as 'understanding' the inputs which fall into
its domain, then one way to describe the challenging of versioning is
to say that to achieve forward compatibility we must specify software
so as to behave gracefully and usefully when it encounters data it
understands only partially. This seems to mean two things:
-
wherever possible to allow partially understood data to fall
into the accept set whenever it also falls into the actual domain
of the software, and
-
to detect cases where partially understood data falls outside
the actual domain, and to ensure that that data goes into the
reject set, rather than into the accept set
When partially understood data fall inside the accept set but outside
the actual domain, the result is less like partial understanding than
like misunderstanding. The goal of point (a) is to maximize partial
understanding and the goal of (b) is to avoid misunderstanding.
In some applications, the cost of misunderstanding is minor and the
cost of a misguided rejection is high, so the simplest technique is to
reject as little as possible. In other applications, the cost of
misunderstanding might be so high that it swamps the cost of misguided
rejections, so that the simplest technique is to accept nothing unless
it is fully understood.
2.7.
Some simple approximations
In some cases (it is not clear whether to call them simple cases, ideal
cases, common cases, or what — surely they all have exceptions),
the goals of those involved may include one or more of the following:
-
Make the set of valid documents a subset of the domain (E.g. develop a
schema to detect in advance errors which might otherwise emerge only
after expensive processing has already been done.) If we do this, we
have V (or in some designs VF) ⊆
D; then if we set A = V,
we are guaranteed that A ⊆ D.
-
Make the domain a superset of the set of valid documents:
D ⊃ V. (E.g. given a schema, write
software which will handle all valid documents; use the schema to
remind yourself what you must be prepared for at any particular point
in the input.)
-
Make the set of valid documents a superset of the domain, i.e. make
V ⊃ D. (E.g. develop a schema to
weed out documents you cannot process, but never let the schema reject
as invalid any input that might turn out to be in the domain. The
application will run additional tests which cannot be expressed in the
schema, or which could be but are not.)
-
Make the reject set a superset of the invalid set (R
⊃ (I ∪ NN ∪
NP)). (I.e. reject anything that's not valid.)
In many cases, the purpose of validation is to aid application
software in deciding whether to accept or reject the input, and the
intention is to make the extended domain of the application contain
every valid document (the combination of application plus validation
then will behave like an application whose extended domain is the
universe of all possible inputs and be more stable).
In different cases, however, the possible mismatches between the valid
set and the domain or extended domain may have different tradeoffs:
-
inputs which fall inside the valid set but outside the extended domain
(inputs in (V \ X)) may cause failures
-
inputs which fall inside the valid set and inside the extended domain,
but outside the actual domain (i.e. inputs in ((V
∩ X) \ DA)), may cause
wasted computation (if the non-schema validation is expensive)
-
inputs which fall outside the valid set but inside the actual domain
(i.e. inputs in (D \ V)) may represent
lost opportunities (the software could have handled them in fact, but
declined to try because the inputs weren't schema valid)
2.8.
Some versioning problems
It seems plausible that the distinctions introduced above ought
to be usable to express versioning problems a bit more precisely than
is sometimes the case. But a full development of this idea is left
to a future version of this document. For now, we limit ourselves
to illustrating a couple of
assumptions we believe the draft TAG finding on versioning appears to make
about the relations among some of these sets, and the possibility of
different assumptions:
-
"The accept set is a subset of the valid set (i.e. no application
accepts invalid or partially valid input)."
In contrast, some members of the WG believe that a key tool
for schema-based versioning is the ability of processors to accept
partially valid documents.
-
"The actual domain is always a superset of the valid set" (if we can
allow the v1 schema to accept v2 data, then the problem is
solved; there isn't any need for the software to do anything in
particular -- and specifically there is no danger that v2 inputs
will ever fall outside the actual domain of v1 software, and no
need to discuss how designers can go about ensuring that this is so).
In contrast, we believe the schema-flexibility question is only
part of the story, and not necessarily at all the most important,
most interesting, or hardest part. Also important (and often
much harder) is to define what default actions a version-N application
can safely take on version-N+1 data, and how to tell when the
default action isn't safe after all and should be avoided.
3. Versioning mechanisms
Several mechanisms have been suggested as potentially helpful in
solving the versioning problems involved in the use cases described
above. Some of those mechanisms are described below.
3.1. V-M1 Second-class wildcards
No UPA violations in cases of
element/wildcard competition.
3.2. V-M2 Match info (element, wildcard)
Which (kind of) particle (wildcard,
element, ...) did we match? Some WG members suggest labeling this
“particle attribution”.
3.3. V-M3 Fallback from xsi:type value to declared element type
if an
xsi:type in the instance names an unavailable type,
allow the schema-validity assessor to use the declared type of the
element instead. If the element is valid against a type legally
derived from the declared type, then either the entire element
instance will be valid against the declared type or a proper prefix of
it will be valid; make it convenient to tell from the PSVI what prefix
of the instance is valid (if any).
3.4. V-M4 Prefix validity
If a prefix of the children is
valid, (so that assuming a second-class wildcard at the end would make
things valid), specify in the PSVI that the element was prefix-valid
(and indicate which prefix of the children was valid).
3.5. V-M5 ancestor prefixes (titration)
In a chain T1, T2, T3, ... Tn
of correct complex-type derivations, an element instance valid
against Tn invariably has a prefix (and subset of its attributes)
valid against each Ti for 0 < i ≤ n.
Provide information in the PSVI to indicate (for each Ti) which
prefix of the children (and which subset of the attributes) is valid
against Ti.
3.6. V-M6 Version attribute
We
immediately discovered two interpretations of this label:
- a way to explicitly link multiple schemas (this was paraphrased
or rephrased as: a way for a linear sequence of revisions by a single
author to allow easy recognition of constructs which will not be
understood by down-version implementations
- a way to mark different parts of an XML instance as requiring
different levels of understanding from the application code
These seem distinct, so we divided them.
3.7. V-M7 Schema version attribute
A way to explicitly link multiple schemas, or more elaborately a
way for a linear sequence of revisions by a single author to allow
easy recognition of constructs which will not be understood by
down-version implementations
3.8. V-M8 Instance-based version attribute
A way to mark individual parts of an XML instance as requiring
particular kinds of support from the processor.
3.9. V-M9 Allow augmentation of choice and all groups
Make it possible to add new members to choice-groups
and all-groups, not just add new items at the end
of a (top-level) sequence-group.
3.10. V-M10 Allow augmentation of enumerations
Make it possible to add new members to enumerations of
legal values in a simple type.
3.11. V-M11 More robust code generation
What is sometimes reported by users as a versioning problem
may be a problem not with their schemas, or with their schema
language, but with their code.
The problem reported is frequently that the version-1.0 schema
classifies documents which conform not to 1.0 but to 1.1 or a later
version of the schema as invalid. In itself, however, this need not
be a problem: there is no reason that applications cannot process
partially valid or invalid documents. The problem appears in some
cases to be that the application code dies when it sees a message
which is not fully valid according to the 1.0 schema. This in turn
reflects in some cases a decision at code generation time to
generate code only for valid documents, rather than generating code
which understands valid documents but also accepts partially valid
and invalid documents.
In other words, the actual problem at the user level might well be
soluble without any change to the 1.0 schema or the schema language,
if code generators systematically generated code which gracefully
handles invalid and partially valid input.
There may, of course, be ways to change the schema language so as
to make it easier to generate code which is more robust in the face
of invalid data.
(Note that although described in terms of “1.0” and “1.1”
schemas, this reasoning obviously applies more generally to known
schemas and different schemas which describe related or overlapping
sets of documents.)
3.12. V-M12 A fallback element
(Note that the descriptions of XSLT and SVG below may be wrong in
details; since the point is not to write conforming XSLT
or SVG processors but to illustrate a possible mechanism for
supporting versioning of vocabularies, it is hoped that any errors will
be mostly harmless.)
Variation among processors may be the result of (a) support for
different versions of XSLT, (b) partial implementation supporting some
but not all features of a particular version (e.g. during an iterative
write-and-release development process), (c) proprietary or
non-proprietary extensions to the language, (d) errors in processors
and/or differences of opinion about the correct interpretation of the
spec, leading to different behavior.
XSLT 1.0 defines two overlapping mechanisms for dealing with such
variation: (1) the functions
- element-available(QName) → boolean
- function-available(QName) → boolean
- system-property(QName) → any
allow stylesheets to interrogate the processor to find out whether a
particular facility is available. In conjunction with conditional
instructions like xsl:if and xsl:choose/xsl:when, they make it
possible to avoid executing stylesheet instructions which would raise
a dynamic error, or to select among different formulations to adapt to
variation among processors. Note that all of the functions named can
be used to handle both third-party extensions and new items (functions
or XSLT instructions) in the XSLT namespace.
Michael Kay's book on XSLT gives this example of adjusting both to
version changes in XSLT and to vendor extensions. The draft of XSLT
1.1 (later abandoned in favor of work on XSLT 2.0) allows temporary
result trees to be processed by passing them to xsl:apply-templates as
the value of its 'select' attribute; various vendor extensions do the
same thing. The following template does the job for a 1.1 processor
or any processor which supports any of several vendor extensions:
<xsl:template name="process-tree-fragment"
xmlns:msxml="urn:schemas-microsoft-com:xslt"
xmlns:xt="http://www.jclark.com/xt"
xmlns:saxon="http://icl.com/saxon">
<xsl:param name="fragment"/>
<xsl:choose>
<xsl:when test="system-property('xsl:version') > '1.0'">
<xsl:apply-templates mode="process-fragment" select="$fragment"/>
</xsl:when>
<xsl:when test="function-available('msxml:node-set')">
<xsl:apply-templates mode="process-fragment"
select="msxml:node-set($fragment)"/>
</xsl:when>
<xsl:when test="function-available('xt:node-set')">
<xsl:apply-templates mode="process-fragment"
select="xt:node-set($fragment)"/>
</xsl:when>
<xsl:when test="function-available('saxon:node-set')">
<xsl:apply-templates mode="process-fragment"
select="saxon:node-set($fragment)"/>
</xsl:when>
<xsl:otherwise>
<xsl:message terminate="yes">
Cannot convert result tree fragment to node-set
</xsl:message>
</xsl:otherwise>
</xsl:choose>
</xsl:template>
A result tree fragment is passed in as the value of the parameter
'fragment'. If we can, we wish to call 'apply-templates' with the
node set of that result-tree fragment as the set of nodes to process,
which means we want to use an expression denoting that node set as the
value of the 'select' attribute on the 'apply-templates' element. If
the XSLT version number is greater than 1.0, then we just write
'$fragment', since 1.1 made that legal. Otherwise, if the
msxml:node-set function is available, we write
'msxml:node-set($fragment)', and similarly if the 'node-set' functions
in the 'xt' or 'saxon' namespaces are available. If none is
available, we issue an error message and die.
(2) The XSLT 1.0 spec defines a 'forward-processing' mode which can be
described as follows. This is not an XSLT tutorial, so we have elided
some details.
- At any point in a stylesheet, there is an 'effective version'
number. In the simple case, it's set using the 'version' attribute
on the 'xsl:stylesheet' element; it can also be set for part of the
stylesheet using the 'xsl:version' attribute on a literal result
element. (Not, apparently, on individual templates or other
constructs.)
Normal usage will be that if a stylesheet uses features of XSLT
N.M, it will have an effective version of N.M.
- XSLT distinguishes four kinds of XML elements encountered in an
XSLT stylesheet: top-level children of the 'stylesheet' element,
literal result elements, XSLT instructions, and other elements in
the XSLT namespace. Instructions are, roughly, the XSLT elements
and extension elements which can occur in a template; other XSLT
elements include the xsl:when and xsl:otherwise elements inside
an xsl:choose instruction.
- When a 1.0 processor encounters an unexpected element U, the
action taken by a 1.0 processor (or, more declaratively, the 1.0
interpretation of the stylesheet) depends on the kind of
unexpected element and the effective version at that point in the
stylesheet:
-
If U is a top-level element, U is ignored.
-
If U is a literal result element, U is not unexpected (i.e.
it cannot happen that we have an 'unexpected' literal result
element).
-
If U is an instruction then:
if the effective version is 1.0, an error is raised,
else (the effective version is not 1.0), fallback processing
occurs.
-
If U is another element in the XSLT namespace then an error is
raised.
Fallback processing is quite simple:
If there is an xsl:fallback element appearing as a child of U,
then all xsl:fallback elements appearing as children of U are
instantiated; otherwise an error is raised.
In normal processing (i.e. when not falling back), the xsl:fallback
element is silently ignored.
It may be seen that a stylesheet author thus has the ability to induce
an XSLT 1.0 processor to do any of several things when it encounters
an unknown instruction:
- pass over it in silence. To do this, put an empty xsl:fallback
element among its children:
<xsl:new-feature>
...
<xsl:fallback/>
</xsl:new-feature>
- raise an error (do not put a fallback element in, or write
version="1.0" on the stylesheet element)
- terminate. To do this, place an XSLT construct inside the
fallback element which causes immediate termination, such as
<xsl:new-feature>
...
<xsl:fallback>
<xsl:message>Your processor doesn't have 'new-feature'
enabled?</xsl:message>
<xsl:message>You've got to be kidding me.</xsl:message>
<xsl:message terminate="yes">[Thud.]</xsl:message>
</xsl:fallback>
</xsl:new-feature>
- perform some useful alternative processing. To do this, write
the fallback using XSLT 1.0 constructs which approximate what
is desired. E.g. let us suppose that XSLT 6.0 introduces a
'copy-to-output' instruction which copies a file directly to
the XSLT output. We can do much the same using the 'document'
function and 'copy-of', but perhaps 'copy-to-output' is typically
much faster, so we prefer to use it if we can:
<xsl:template>
<div xsl:version="6.0">
<xsl:copy-to-output href="boilerplate.xml">
<xsl:fallback>
<xsl:copy-of select="document('boilerplate.xml')"/>
</xsl:fallback>
</xsl:copy-to-output>
</div>
</xsl:template>
Note that for some kinds of versioning- or extension-related changes
in vocabulary, the stylesheet author has no control. If a new
construct is introduced as a top-level child of the xsl:stylesheet
element, a 1.0 processor will ignore the construct, period. If a new
construct appears not as an instruction but in a more restricted
context (e.g. within an existing construct like
choose/when/otherwise), a 1.0 processor will raise an error.
SVG has some similar constructs.
The 'svg' element has 'baseProfile' and 'version' attributes. (How
these interact with the other constructs described here will not be
explained here.)
The SVG 'switch' element allows an SVG document to signal that certain
content should be processed only if the processor supports a given
named feature. The description in the SVG 1.1 spec is clear and
concise:
SVG contains a 'switch' element along with attributes
requiredFeatures, requiredExtensions and systemLanguage to provide
an ability to specify alternate viewing depending on the
capabilities of a given user agent or the user's language.
...
Attributes requiredFeatures, requiredExtensions and systemLanguage
act as tests and return either true or false results. The 'switch'
renders the first of its children for which all of these
attributes test true. If the given attribute is not specified,
then a true value is assumed.
For example:
<svg:switch>
<svg:g ... requiredFeatures="
http://www.w3.org/TR/SVG11/feature#OpacityAttribute
http://www.w3.org/TR/SVG11/feature#XlinkAttribute">
...
</svg:g>
<svg:g ... requiredExtension="http://example.org/xsvg#newFeature">
...
</svg:g>
<svg:g><!--* ultimate fallback *-->
...
</svg:g>
</svg:switch>
Note that since the test attributes default to 'true', omitting them
all (as in the last 'g' element above) has the effect of an
'otherwise' clause.
SVG processors are required to tolerate and ignore elements and
attributes in other namespaces; the svg:foreignObject element can be
used to encapsulate extension elements in other namespaces which
should be processed rather than ignored; it will typically be used in
conjunction with a 'switch' element.
3.12.3.
What's in it for us?
The facilities of XSLT and SVG suggest several other techniques we
should consider:
-
a 'fallback' element analogous to that of XSLT, to fire when
its parent is not a well understood construct
-
explicit identification of features / system properties, so that
schema documents can test for them
-
some sort of conditional construct analogous to the 'choose/when'
of XSLT (too general, probably, for us) or the 'switch' of SVG.
(Like XSD, SVG is not a programming language and has no need of
a fully general select statement.)
-
explicit policies enabling extension constructs (expressed as
elements in other namespaces) at the top level and/or elsewhere
in a schema document; the xsd:annotation element is clearly
felt by many as a kind of ghetto and doesn't seem to get the job
done, perhaps because of the term 'annotation'.
-
a way to induce a processor to terminate schema-validity assessment
immediately, or to decline to start, so that schema authors can
indicate when a down-level processor should NOT try to muddle
through
3.13. V-M13 Auto-insertion of wildcards
Allow, or require, the automatic insertion of wildcards into content
models, so that the resulting content model allows unrecognized elements
at any point. A model declared as a sequence of
my:a,
my:b,
my:c, for example:
<xsd:sequence>
<xsd:element ref="my:a"/>
<xsd:element ref="my:b"/>
<xsd:element ref="my:c"/>
</xsd:sequence>
would automatically be transformed into something like the following:
<xsd:sequence>
<xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
<xsd:element ref="my:a"/>
<xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
<xsd:element ref="my:b"/>
<xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
<xsd:element ref="my:c"/>
<xsd:any namespace="##any" minOccurs="0" maxOccurs="unbounded"/>
</xsd:sequence>
Note that in XML Schema 1.0 this latter content model is
illegal because an initial my:a element could
match either the element particle or the
preceding wildcard. As a result, this proposal is frequently linked
to proposals for weak wildcards.
3.14. V-M14 Typed wildcards
Allow a content model to contain particles which match a element
with any name (generic identifier) at all, as long as the element
has a particular type.
3.15. V-M15 Negative wildcards
Allow wildcards to exclude a specific set of namespaces (so that
a wildcard could express the complement of the following choice:
<xsd:choice>
<xsd:any namespace="ns1" minOccurs="0" maxOccurs="unbounded"/>
<xsd:any namespace="ns2" minOccurs="0" maxOccurs="unbounded"/>
<xsd:any namespace="ns3" minOccurs="0" maxOccurs="unbounded"/>
</xsd:choice>
Similarly, allow wildcards to match any elements in a particular
namespace except those which match any of an enumerated
list of local names. Make it convenient to express notions like
“match anything in the current namespace except elements which
appear explicitly in this content model”.
See also V-M19 below.
3.16. V-M16 Fallback to expected element type
If the element in the instance does not match any of the
expected particles in a content model, and if there is only one
element type expected, allow the schema processor to attempt to
validate the instance element against the expected element
declaration. So, for example, if the content model expects
my:a, my:b, my:c,
and the instance has
my:a, my:b, my:x,
then perhaps the instance conforms to a variant schema in which
my:x is in the substitution group of
my:c. If that is so, then the type of my:x
should be derived from that assigned to my:c; if
derived by restriction, then it should be legal against the
type associated with my:c; if by extension, then
a prefix of it should be legal against the type associated
with my:c.
3.17. V-M17 Explicit relations among schema documents
Provide mechanisms for asserting relations between the
languages recognized by the constructs in different schema
documents, e.g. “the language recognized here is a subset
of the language recognized by X”, or
“... is a superset of ...”, or
“... is a later version of ...”, etc.
3.18. V-M18 Explicit relations among schemas
As for the preceding, but make the facilities operate at the component
level, rather than the source-declaration level.
3.19. V-M19 Wildcard that matches anything not declared in this schema
A specific variant of V-M15.
3.20. V-M20 Fallback attribute in instance
Allow an instance document to carry attributes (similar to
xsi:type) which specify what declarations to use if the
schema in use doesn't have the expected ones. The fallback attribute
might specify an alternative element declaration, but perhaps will
be more useful if it specifies a type definition. A sequence of
type definition names might be provided, to be tried in sequence.
An element might then appear in the instance as
<my:e xsi:fallback="my:t1 my:t2 my:t3">...</my:e>
with the result that the schema processor would perform fallback
processing as follows:
- If the schema in use has a declaration for element
my:e, which assigns type my:t0 to it,
then the instance element will be validated against the declaration of
my:e and the definition of my:t0 in the
usual way.
- Otherwise, if the schema in use has a declaration for element
my:e, which assigns type my:t0 to it,
but the schema lacks any definition of my:t0, then
the instance element will be validated against the declaration of
my:e and either the definition of my:t1,
or that of my:t2,
or that of my:t3, using the first one found.
- Otherwise, if the schema in use has no declaration for element
my:e, then
the instance element will be validated against
either the definition of my:t1,
or that of my:t2,
or that of my:t3, using the first one found.
3.21. V-M21 Fallback validating using local declarations in preference to globals
In 1.0, associations between element instances and local
element declarations are only made in the case where the
parent element is locally valid (or more specifically only
in cases where the parent's children match the parent's
content model). Otherwise, either the element is marked
invalid and its children are not validated at all, or
(at processor option) the element is validated laxly, which
means the children are validated if and only if they match
some top-level element declaration.
Change this to allow / require fallback processing to use
the local declarations in preference to global ones.
3.22. Two schemas
One possible approach would be to define two schemas, one which
defines the set of fully-understood documents and one which
is used to define the accept set. The former might use
only fully-specified and fully-understood elements and
attributes and have no wildcards; the latter might be derived
from the former by an algorithmic process (e.g. by adding
wildcards at specified locations).
One way to allow extensions is to provide a wrapper element for
extensions; the draft TAG finding provides a good account of this.
Effectively, a version n processor simply ignores
everything in the extensions element.
An important drawback is that after repeated revisions a schema
and its instances may begin to look either rather onion-like
(with a version-n+1 extension element nested inside
the version-n extension element), or else
like a statement with a chain of afterthoughts (with the
version-n+1 extension element following the version-n
extension element). Either way, generators of documents which
must conform to later versions must keep track of which version
something was added in (“Quick, does middlename
belong in the v1-extensions element or in the
v2-extensions element?”).
3.24. Namespace filtering
Some schema authors support extensibility by requiring that all
extensions be in the extender's namespace, not that of the base
schema. This can be done either by inserting ##other wildcards
at appropriate places in the content models to allow such extensions everywhere,
or by using these rules for validation:
- Traverse the document tree, removing everything which is
not in the namespace defined by the base schema.
- Validate.
3.25. Double validation
One way to allow for arbitrary extensions:
- Use top-level elements. To be more precise: in any
complex type which you wish to allow to be extended arbitrarily, do not
use any local element declarations; use only references to top-level
element declarations. (Syntactically, use <xsd:element ref="..."/>,
not <xsd:element name="..."/>.)
- Use the run-time option which tells your schema processor to
fall back to lax processing if it cannot validate an element's
children against its type.
- Validate once. If the part of the document you care about
is fully validated and valid, you are done. Otherwise, continue.
- Traverse the document tree, removing everything which is
invalid or has [validity=notKnown].
- Validate a second time. If the part of the document you care
about is fully validated and valid, accept the document; otherwise
reject it.
Note that namespace filtering is a special case of this technique.