The WG discussed this issue during its call of 21 January 2011 and instructed the editors to remove the misguided references to override, and see what they can do to minimize confusion for readers who are surprised not to find them.

Adopted on the telcon 2011-02-04: to accept the proposal for 11179 as submitted, and bounce back the proposal for 11354 to the editors with instructions to prepare a proposal to define two schemas and two DTDs for schema documents, one for the raw and one for the cooked.

I'm beginning to believe that the WG needs to re-open this issue because our resolution was based on an inaccurate premise.  In particular, we were operating on the belief that the override transform is defined in such a way that eventually, after performing the transform for each document in the target set of a given override element, one would have a set of schema documents none of which contain any override elements.  It would be these schema documents which are subject to the schema-document constraints.

But as Michael Kay has pointed out in bug 12184, the override transformation is not guaranteed to eliminate all occurrences of the override element.  On the contrary, it may increase the number of override elements, since it transforms every include element into an override element.  This should not be a surprise, since it was made explicit in the design discussions for the override facility that the transform would turn cycles of includes into cycles of overrides.  But the shorthand description that 'the semantics of override can be reduced to the semantics of include' (which is true, I think, as far as it goes) led the WG into the false belief that 'the override transform transforms all overrides into includes'.

Since the transform as currently specified does not eliminate all override elements, we cannot as things stand plausibly eliminate override elements from the syntax overviews in the spec (which means that bug 11179 also needs to be revisited) or from the schema for schema documents.  

I think our options include at least:

1) Reverse our decision on this issue and bug 11179.  Accept that override elements must be handled in the schema for schema documents, in syntax overviews, and in discussions of where elements can appear.  Ensure that all constraints on schema documents make sense in the presence of override.

2) Respecify the override transform to change the way it propagates an override.  In particular, make it guarantee that a simple-minded process of applying the transform will ultimately produce a set of documents without any override elements.  

The second option has the unhappy property that almost every time I look at it, it seems to be within reach, and every time I try to sketch the transformation required, I find that it either doesn't eliminate overrides after all, or that it fails to provide the functionality required by our resolution of bug 6021.

3) Forbid cycles in override and include.

In the absence of cycles, there is an override-free set of schema documents corresponding to any override target set.

For the reasons just given, I'm (unilaterally) changing the status of this issue from needsDrafting to needsAgreement, and marking it as dependent on bug 12184.

I'm confused.  As I read the spec., it's not only true that Dold' may still contain <override> elts, it will _always_ contain as many <override>s as the sum of the number of <override>s and the number of <include>s it started out with.

Dold' is not I think what we meant when we talked about 'raw' and 'cooked' -- our conversation, and the revised resolution in comment 2, only makes sense if by 'cooked' what we _actually_ meant was, as it were, Dnew' == the result of

  The <override> element in schema document Dnew pointing to Dold is
  replaced by an <include> element pointing to Dold&#8242; 

notwithstanding the Note saying

  Note: It is not necessary to perform a literal replacement of the <override>
  element in Dnew with an <include> element; any implementation technique
  can be used as long as it produces the required result.

This leads me to ask a historical question:  When we introduced the explicit transform mechanism to handle both chameleon include and override, why did we add the "not D2 but D2'" and "not Dold but Dold'" Notes (and of course the normative bits which justify them)?  In what circumstances would D2 be non-conforming but D2' be conforming?  Likewise for Dold and Dold'?  Sorry if the answers are in the archive -- I have been unable to figure out how to look for them. . .

Henry Thompson writes:

    Dold' ... will _always_ contain as many <override>s as the
    sum of the number of <override>s and the number of <include>s 
    it started out with.

Yes.  The misconception that afflicted us on 4 February 2011 was
a result of confusing the proposition that "the semantics of override
reduce to the semantics of include" (true) with the proposition "the
transformation works by eliminating occurrences of override" (false).

HT also asks

    When we introduced the explicit transform mechanism to handle 
    both chameleon include and override, why did we add the "not D2 
    but D2'" and "not Dold but Dold'" Notes (and of course the
    normative bits which justify them)?  In what circumstances would D2 be
    non-conforming but D2' be conforming?  Likewise for Dold and Dold'?

Chameleon processing appears to have been changed in response to 
bug 2067, which was closed in February 2007 in New
Orleans, then closed again in July and August 2007 (having apparently 
been reopened after the New Orleans meeting), and re-opened and closed
a third time in May 2008.  

The override facility was first adopted as a resolution of bug 4767 and
then modified in connection with bug 6021.

That may help narrow searches through the decision record.

The wording proposals adopted at various times (and some other
proposals considered but not adopted) are pointed to from the editorial
pointers page at

  http://www.w3.org/XML/Group/2004/06/xsd-ed-pointers.html
  (member-only link)

Without reviewing the relevant parts of the decision record, I'll observe
that in each case, the transformation gives us two schema documents to
deal with, and so we must specify whether the constraints in the spec
apply to the transformation's input, to its output, or to both.  

In the case of override, it seems clear that errors in the transformation
output need to be detected and errors in the input need to be detected
only if they lead to errors in the output.  There might be a validity error 
in a declaration (perhaps both @ref and @name are specified on an 
element declaration), a conformance error (perhaps minExclusive is 
applied to a string type), or possibly other problems (can't think of 
any but don't want to bother trying to prove they couldn't exist) in a 
declaration overridden by a corrected declaration in the overriding 
document.   Since the overridden source declaration is overridden, 
it isn't used to try to create or identify any components, so its errors 
do not need to stand in the way of identifying the schema to be used
in validating the document.  

In the case of chameleon include, I am not sure off hand whether there 
is any logical relation between the validity or conformance of the input 
schema document and those of the output document; it may well be 
that if one is valid and conforming, we know that the other is, as well.  
If so, it suffices to check just the one.  If they are not logically tied
together, then it seems clear that it is the validity and conformance 
of the result of the transformation that we should be concerned with,
while those of the input document are at most of academic interest.

In some cases, of course, errors arise not from a schema document in
isolation but only when schema documents are combined.  If we have

  - schema document A declares element E (unqualified) as xsd:date
  - schema document B in namespace N declares element N:E2
    and performs chameleon inclusion of schema document C
  - schema document C declares element E (unqualified) as xsd:int

then the combination of A and untransformed C will not be a 
conforming schema (two conflicting declarations for element {}:E),
while the combination of A with chameleon(C,N) will be fine.

I conclude that there is no need for the input to the transformations to
be conforming schema documents, only for the output to be.

In this, transformations defined on schema documents differ from
transformations defined on schemas.  For a transformation defined
on a schema rather than on a schema document, the conformance of
the starting schema document is of critical importance, and there is
no second schema document to be concerned with, only a second
schema.  There is, in the nature of the case, no way to avoid having to
check two schemas for conformance to the spec: the input schema
must be checked as a pre-condition for the transformation, and the
output needs to be checked again, since in the general case there
is no guarantee that the transformations on schemas defined by XSD
preserve conformance either of individual components or of the
schema.

>"Since the overridden source declaration is overridden, 
it isn't used to try to create or identify any components, so its errors 
do not need to stand in the way of identifying the schema to be used
in validating the document.".

Oh dear, ghastly thought. Suppose someone writes

<xs:element name="abc" type="!!**!!**!!**!!"/>

and then overrides this. Is the processor actually not allowed to report an error?

I would resist such an interpretation.

>I think it depends on what you mean by "report an error".

Sorry if I'm using vernacular language. It's my lifelong habit, which persists despite several years in the Schema WG. What I mean by "reports an error" is that the schema processor says it can't use this so-called schema to validate anything.

>Writing schema documents which _must_ be overridden before they can be
used is not something we want to encourage.

Agreed.

>I think it's clear that we need to require Dold as well as Dold' to "correspond to a conforming schema."

That may be too strong a requirement. As a minimum, I think we want Dold to be valid against a schema which closely resembles the S4SD. We may also want it to satisfy all/some of the other XML representation constraints. (It's a shame we never did the work of expressing all these constraints in the S4SD using assertions etc.)

> [requiring Dold as well as Dold' to "correspond to a conforming schema."
> may be too strong a requirement.

Hmm.  Why?  That is, do you have an example in mind which is a) reasonable or likely to arise but b) would be ruled out by the above?

>Hmm.  Why?  That is, do you have an example in mind which is a) reasonable or
likely to arise but b) would be ruled out by the above?

I don't think I would want to insist, for example, that the definition of a union type in Dold is not circular, if the override resolves the circularity. This, I think would undermine the idea that xs:override operates essentially at the schema document level.

Responding to comment #8:

Think again about circular override. When A overrides B and B includes A, we can't require B to correspond to a schema, because it'll contain duplicate components, one copy from B itself, and one from B' as a result of A's override.

It'll also make it much harder to implement. As MK suggested, ideally override should be performed at the source level, so an implementation can swap the original element with the new one and keep going. Requiring "correspond to a schema" will need a lot more.

Now to comment #6:

I'm not convinced that we need to impose any constraint on Dold. Why can't I write a "template" schema document with illegal names in them (e.g. "$to_be_replaced") and override it? This looks quite useful to me. Of course, people can use it to hang themselves. Their choice.

<dezell> MSM: we reached phase 1, but I'm expecting to come back with a phase 2 that does something different.
<cmsmcq> and the phase 2 wording we will bring back will have a single set of schemas, no raw/cooked distinction

A wording proposal intended to resolve this issue is on the server at

  http://www.w3.org/XML/Group/2004/06/xmlschema-1/structures.b11179bis.html
  (member-only link)

Since we clarified the nature of the override pre-processing transformation and reversed course on this bug, there are no changes to make to resolve it, and the only thing in the change proposal directly linked to this issue is the change to the schema and DTD for schema documents.

RESOLVED: adopt the proposal linked to comment 12 as ammended in bug 11179.

The change agreed upon in the call of 13 May has now been integrated into the status quo document, as amended, so I'm marking this issue resolved.

Sandy, as the originator of the issue I ask you to check that the changes have been made correctly and close or reopen the issue as appropriate.  Thanks!