XProc CR Disposition of Comments

> The V2 processor sees an unbound, non-primary output port so that's ok.
>
> The V1 processor sees a binding to a non-existant port on p:xslt.
> That's a static error.

This is basically an outline of the problem I'm worried about. BTW, that
messages port sounds like a great idea. I think I realize why it's not in V1
(hard to implement for some XSLT processors?), but it does sound like
something really useful and something to seriously consider for V2... and it
is the best example yet as to why some existing steps may need to add new
input/output ports in the future.

> Instead of importing a new library, we could use a version attribute, I
suppose.

The exact version notation is somewhat irrelevant I think. The problem is in
the behavior when in forwards compatible mode... it's currently too limiting
(in general). As long as V1 processors aren't forced to download signatures
of steps by contacting W3C servers, the current notation would be fine.

> What you're proposing is a different and more complex approach.
> It requires not only a refactoring of the forwards-compatibility section
> but also the introduction of at least one new concept to the language
> (the "wildcard" acceptability of unknown port names).

Is it THAT hard to specify it? Since it applies only to p:* steps while in
forwards compatible mode, it seems to me like a single (or at worst - a few)
extra paragraph(s) at "2.13 Versioning Considerations" will pretty much
cover it. A note or something at p:import may also be necessary, and that's
all. I don't see what else (at other parts of the spec) that would need
changing.

> It's entirely possible that the approach you outline could be made to
> work, but the WG went a different direction. It's awfully late in the
> day to attempt to change this, I fear.
>
> The extent to which this dynamic behavior is important is probably
> based partly on how likely one thinks it is that new ports will be added.
> I think the WG considers it very unlikely that an existing step's
signature
> will need to be changed. We could be wrong, of course.
>
> The XProc design is predicated on the availability of a signature for
> each step; it might be possible to reimagine an XProc design which doesn't
> have this constraint, but that's not the direction that the WG went.

I'm sorry I'll have to put it in this way, but... You ARE wrong. Your own
example is good enough (and like I said, a great) use case showing that
future ports may be needed for existing standard steps, p:xslt being the
most likely candidate for such. One other example (again, if your own isn't
enough) is the use of a serialization input port instead of serialization
options to all steps that perform serialization (p:store, p:escape-markup,
etc.) or a serialization output port to all steps that may control
serialization if used stand alone (p:xslt, etc.).

But perhaps the best example is if a future version of an external
specification needs accommodation that can be achieved simply by the
addition of a few optional ports (and perhaps with a few new options, which
would be ignored by XProc 1.0 anyway). Again, p:xslt serves as the best
example. The "secondary" port is useless for a standard XSLT 1.0, but if
XProc was created before XSLT 2.0, it would have had to create p:xslt2 (as
was once the case...). If XQuery makes a similar addition, the spec
currently forces future versions of XProc to create p:xquery2 (assuming it's
XQuery 2 that has this change). As seen already in the debate at the times
of p:xslt and p:xslt2, hard coding the version in the type isn't entirely a
good idea, as it will quickly become outdated as newer versions of the
external spec (with no need for a new signature) arises, it creates clutter,
creates inconsistencies if only the "changed" versions are accommodated
(p:xquery and p:xquery3, but no p:xquery2) and it simply feels too dirty.

Even if V1 standard steps didn't needed to be extensible port wise, some of
the steps in V3 may need to extend steps that first appeared in V2. Due to
V1's limitation, they won't be able to do that, unless they sacrifice V1
support, or (as. Depending on the time frame between V1, V2 and V3, and the
number of implementations on each, sacrificing V1 may or may not be a good
design choice at the point of V3.

If by "awfully late" you mean that this change would have to push the spec
back to (LC?)WD... since this is a V1 of a language, if there's any issue
that is worthy enough to push it back to this state, I think forwards
compatibility is the one. V1 (of any language, especially one like XProc)
just can't afford to get this wrong.

> Instead of saying that unknown content in a step is a static error,
> we could say that it is a dynamic error.
> Statically, the content is ignored but it's a dynamic error to
> attempt to evaluate that step.

Uh... well, yes... this is exactly what I'd like... and I think this is what
I said... if not, know that this is what I meant when I say "dynamic error"
(I assume static checks happen before p:when evaluation, and therefore if
something is a static error, it occurs with or without p:when; I also assume
non-static checks are dynamic checks). Yes, please do that.

Regards,
Vasil Rangelov

-----Original Message-----
From: public-xml-processing-model-comments-request@w3.org
[mailto:public-xml-processing-model-comments-request@w3.org] On Behalf Of
Norman Walsh
Sent: Wednesday, October 07, 2009 3:24 PM
To: public-xml-processing-model-comments@w3.org
Subject: Re: Concerns about forwards-compatible mode

Vasil Rangelov <boen.robot@gmail.com> writes:
>>> I suggest that the error be made dynamic instead of static.
>>
>> The WG considered this, but it really isn't practical.
>> Computing the dependency graph for the pipeline is really something 
>> that an XProc processor needs to be able to do statically.
>
> OK. Then how about if in forwards compatible mode:
> 1. All input ports in use on p:* steps are added as "valid" ones (or 
> treated as being "aliases" to a certain "wildcard" port).
> 2. All input bindings to unknown ports on p:* steps are discarded (or 
> added as contents of the wildcard port that doesn't use them for
anything).
> 3. All references (by p:pipe) to unknown output ports on in scope p:* 
> steps create an empty binding (or cause a dynamic error? If 
> connections between steps must also be figured out statically, then I 
> suppose an empty binding would be a more appropriate solution).
>
> With these "assumptions", there won't even be a need to contact W3C 
> servers to get the new signatures. Signatures are still computed 
> statically - it's just that they are far more liberal than in normal 
> mode. Sure, the pipeline may produce nonsense if left unchecked, but 
> p:choose is still there to allow the pipeline author to write a 
> cross-version pipeline. This approach isn't much different than XSLT 
> 1.0's forward compatible mode (which basically says "ignore everything
unknown").

Imagine that we add a "messages" output port to the p:xslt step in V2.
Then you want to be able to run this step through both V1 and V2
processors:

<p:choose>
  <p:when test="p:system-property('p:version') &lt; 2.0">
    <p:output port="result" primary="true">
      <p:pipe step="xsltv1" port="result"/>
    </p:output>
    <p:output port="secondary">
      <p:pipe step="xsltv1" port="secondary"/>
    </p:output>
    <p:output port="messages"/>
      <p:empty/>
    </p:output>
    <p:xslt name="xsltv1">...</p:xslt>
  </p:when>

  <p:otherwise>
    <p:output port="result" primary="true">
      <p:pipe step="xsltv2" port="result"/>
    </p:output>
    <p:output port="secondary">
      <p:pipe step="xsltv2" port="secondary"/>
    </p:output>
    <p:output port="messages">
      <p:pipe step="xsltv2" port="messages"/>
    </p:output>
    <p:xslt name="xsltv2">...</p:xslt>
  </p:when>
</p:choose>

The V2 processor sees an unbound, non-primary output port so that's ok.

The V1 processor sees a binding to a non-existant port on p:xslt.
That's a static error.

In order to make that a dynamic error, the processor would have to mark the
step as malformed (so that attempts to run it would always fail dynamically)
but then just accept the binding to a bad port name.

You'd (presumably) only want this in forwards compatible mode, so we'd have
to change the marker. Instead of importing a new library, we could use a
version attribute, I suppose.

This only applies to steps in the p: namespace because those are the only
ones that can be run without an explicit declaration.

Sigh.

What you're proposing is a different and more complex approach. It requires
not only a refactoring of the forwards-compatibility section but also the
introduction of at least one new concept to the language (the "wildcard"
acceptability of unknown port names).

It's entirely possible that the approach you outline could be made to work,
but the WG went a different direction. It's awfully late in the day to
attempt to change this, I fear.

The extent to which this dynamic behavior is important is probably based
partly on how likely one thinks it is that new ports will be added. I think
the WG considers it very unlikely that an existing step's signature will
need to be changed. We could be wrong, of course.

The XProc design is predicated on the availability of a signature for each
step; it might be possible to reimagine an XProc design which doesn't have
this constraint, but that's not the direction that the WG went.

>> With respect to new, unknown elements in other contexts (at the top 
>> level, in the body of a step, etc.), it's hard to imagine how 
>> ignoring them could be the right thing in the context of XProc.
>
> Not ignoring them is fine, as long as it's a dynamic error and not a 
> static one and therefore, a p:choose can be wrapped around it, 
> allowing the pipeline to execute properly in different XProc versions.

That seems more tractable. Instead of saying that unknown content in a step
is a static error, we could say that it is a dynamic error.
Statically, the content is ignored but it's a dynamic error to attempt to
evaluate that step.

>>From this statement (and the one from the 10 Sep 2009 minutes), it's 
>>not
> clear what you mean the resolution for such elements is. Is it a 
> dynamic error (please say "yes")?

I think the status quo is "no".

                                        Be seeing you,
                                          norm

--
Norman Walsh <ndw@nwalsh.com> | A man must have grown old and lived
http://nwalsh.com/            | long in order to see how short life
                              | is.-- Schopenhauer

> "Toman_Vojtech@emc.com" <Toman_Vojtech@emc.com> writes:
> > "1. If an unknown step in the XProc namespace is encountered, it is
> > assumed to be correct."
> >
> > What does "correct" mean in this case? Because the step is 
> unknown, the
> > processor has no idea about how its signature (input ports, output
> > ports, options, ...) looks like. Knowing the input and 
> output ports (and
> > which ones are primary, if any) is essential to determine 
> the correct
> > evaluation order of steps in the pipeline.
> 
> I managed to convince myself that this wasn't actually a problem. Any
> attempt to evaluate an unknown step will raise an error, so unless the
> step occurs inside a p:when or the p:group of a p:try, the error can
> propogate upwards. In a p:when or p:try, you know the order because
> you have the other branches.

I am not sure myself, but perhaps you are right. I still have to think
about this. The biggest problem I have is that the V1 processor would
have to assume too many things when it encounters a V2 step. Consider
this pipeline:

<p:pipeline>
  <p:identity/>
  <p:v2-step/>
</p:pipeline>

Now, let's assume the pipeline is a valid V2 pipeline, with no static
errors. Then, in order to accept this pipeline statically, the V1
processor must:

1. Assume that p:v2-step has a primary input port - otherwise err:XS0005
(unconnected primary output port of p:identity) would be thrown
2. Assume that p:v2-step has a primary output port - otherwise
err:XS0006 (primary output port of the pipeline not connected and the
last step in the subpipeline does not have a primary output port)

We should be very careful here, and really look at different scenarious
where such kinds of assumptions may lead to unexpected behavior, or too
ambitious requirements on V1 implementations.

Regards,
Vojtech

Folks,

I've posted a new "alternate" draft which outlines a revised
versioning policy:

  http://www.w3.org/XML/XProc/docs/alternate/

The key changes:

1. Extra attributes and elements are not static errors.
2. Extra input ports are not errors.
3. Extra output ports are not errors.

The key non-changes:

Fowards-compatible processing still requires the V-previous processor
to download the step library that defines the V-current step
signatures.

Comments most welcome.

                                        Be seeing you,
                                          norm

-- 
Norman Walsh <ndw@nwalsh.com> | If you haven't found something strange
http://nwalsh.com/            | during the day, it hasn't been much of
                              | a day.--John A. Wheeler

Rereading the proposal I realized it says exactly the opposite. It says 
that missing version implies "the latest version".
Could you explain the rationale/value in that interpretation over having 
it imply "1.0" ?
Due to the inheritance of versions through the document (a really cool 
Idea IMHO) At the very top of the pipeline it would only take adding 1 
attribute to convert a pipeline to a future version.

David A. Lee
dlee@calldei.com  
http://www.calldei.com
http://www.xmlsh.org
812-482-5224



David A. Lee wrote:
> Cant the absence of the version attribute be implied to mean "1.0" ?
> That way any existing pipeline is implicitly "1.0"
> Making something required that wasnt at this stage of the game will 
> break existing pipelines.
>
> David A. Lee
> dlee@calldei.com  
> http://www.calldei.com
> http://www.xmlsh.org
> 812-482-5224
>
>
> Florent Georges wrote:
>> 2009/10/8 Norman Walsh wrote:
>>
>>   Hi,
>>
>>   
>>> I wonder if this would be better.
>>>     
>>
>>   I think introducing the version attribute is a good thing.  But why
>> don't making it required?  If a further version wants to introduce
>> slight differences for the same components (I know that can sound
>> weird, but think about XSLT 1.0 vs. 2.0), that won't be possible if we
>> have a lot of legacy pipelines without explicit version.
>>
>>   But if we know that existing pipelines have version="1.0" we will be
>> less limited.  For instance if XProc 2.0 wants, hum, let's say, just
>> by chance, to use XDM ;-)
>>
>>   Regards,
>>
>>   

Well, yes, it is. My point exactly. Unknown steps are also unknown elements.
If the p:import requirement is removed in favor of a version attribute, an
XProc 1.0 processor isn't going to know the signatures of any new steps.
Therefore, any new steps will HAVE TO BE treated as unknown elements. I
mean, for example:

<p:pipeline>
  <p:declare-step type="...">
    <p:unrecognized>...</p:unrecognized>
  </p:declare-step>
  <p:identity/>
</p:pipeline>

Is p:unrecognized an element like p:serialization or p:input that does
something new and fancy with p:declare-step, or is it a new atomic and/or
contained step that has no required options, reads the primary input port,
and outputs on the primary output port implicitly? Without having a step
signature, you don't know for sure (and even if you did, you don't know if
it's a contained step, or a p:serialization like element). So the only thing
you're left to do is ignore it or have a dynamic error if that newly
declared step is evaluated. Ignoring, I think, is the better option, but a
dynamic error is also fine.

Regards,
Vasil Rangelov

-----Original Message-----
From: public-xml-processing-model-comments-request@w3.org
[mailto:public-xml-processing-model-comments-request@w3.org] On Behalf Of
Toman_Vojtech@emc.com
Sent: Monday, October 12, 2009 10:38 AM
To: public-xml-processing-model-comments@w3.org
Subject: RE: Another take on versioning

> 2. Treatment of unknown steps
> 
> I think this is a wrong way to think about it (the word 
> "steps" I mean).
> There are some elements that don't have a formal 
> qualification. They aren't
> "atomic steps", they aren't "contained steps", they aren't 
> (always) direct
> child elements of steps either. They need a formal qualification, and
> unknown elements (regardless of their formal qualification in future
> versions) should be treated as these kinds of elements would 
> be treated.
> These elements currently include (among others?) 
> p:declare-step, p:pieline,
> p:import, p:log, p:serialization, p:xpath-context, p:iteration-source,
> p:viewport-source, p:inline, p:document, p:pipe, p:empty, p:input and
> p:output.

Isn't this already covered by the bullet 2?

"2. On any element in the XProc namespace, unrecognized attributes
(other than extension attributes) and unrecognized ***descendant***
elements are ignored. Any step which has either is marked as invalid. If
they occur on or in p:declare-step, all attempts to evaluate instances
of the declared type are also marked as invalid."

On the other hand, as it is stated now, bullet 2 may be too restrictive.
If I understand it correctly, the following pipeline would be marked as
invalid, but it should not be:

<p:pipeline>
  <p:declare-step type="...">
    <p:unrecognized>...</p:unrecognized>
  </p:declare-step>
  <p:identity/>
</p:pipeline>

The top-level pipeline contains an unrecognized descendant element, but
it actually does not use it at all.

Regards,
Vojtech

The proposal talks mainly about new or unrecognized steps, but what if
V2 introduces a new non-step construct, such as in the examples below:

<p:pipeline>
  <p:include href="..."/>

  <p:identity/>
</p:pipeline>

Worse, there may be new constructs that affect the dependency graph:

<p:identity>
  <p:input port="source">
    <p:big-pipe step="..." port="..."/>
  </p:input>
</p:identity>

or

<p:identity>
  <p:depends-on>
    <p:pipe step="..."/>
  </p:depends-on>
</p:identity>

The proposed strategy of marking the steps as invalid works well in the
first example, but not that well in the other two, because a V1
processor does not know what p:big-pipe and p:depends-on mean. Even if
you wrap the identity steps from in p:choose, a V1 processor can never
establish correct dependency graph.

I guess the only safe solution to this is to use "use-when" here.

Regards,
Vojtech

"Toman_Vojtech@emc.com" <Toman_Vojtech@emc.com> writes:
>> ============================================================
>> 
>> Proposal 1: Add a version attribute.
>> 
>> The version attribute is allowed on any element. On an element in the
>> XProc namespace, it is "version" in no namespace. On an element not in
>> the XProc namespace, it is "p:version" in the XProc namespace.
>
> Are you suggesting we could use p:version on, for instance, c:data,
> c:directory or c:request?
> I can probably see what you were aiming at (we may decide to add new
> elements to the c:request model in Vnext), but I don't really see how
> this can work in practice. It seems to me you would have to add some
> intelligence to steps that consume (and probably produce?) documents in
> the xproc-step namespace, so that they can filter them based on
> p:version.
>
> Personally, I wouldn't support "p:version" on non-XProc elements. I
> would just say that you have to use "p:use-when" in this case.

Fair enough. In that case, I propose that it's only allowed on
p:library, p:pipeline, and p:declare-step.

>> ============================================================
>>
>> Proposal 2: The version attribute is required on top-level
>> p:pipeline, p:declare-step, and p:library elements.
>
> +1 from me. Just a question: At the moment, we allow explicitly
> importing a standard XProc library using p:import. With the "version"
> attribute, shouldn't we make this forbidden? Because with the "version"
> attribute and the defaulting rules, the processor would always know what
> to do.

I'm not sure. We could forbid it. Yes, maybe that's the simplest thing
to do.

>> ============================================================
>> 
>> Proposal 3: Add a use-when attribute
>> 
>> The use-when attribute is allowed on any element. On an element in the
>> XProc namespace, it is "use-when" in no namespace. On an 
>> element not in
>> the XProc namespace, it is "p:use-when" in the XProc namespace.
>
> +1
>
> So, if you wan't to preserve p:use-when elements in your XML documents,
> you would either have to escape/unescape the documents (p:escape-markup,
> p:unescape-markup), or make them external, right?

You mean in documents inside p:inline, yes? I don't think they come
into play anywhere else. We could say that p:use-when processing is
suspended within p:inline elements. On reflection, I think we probably
should.

>> ============================================================
>> 
>> Proposal 3: Attempt to treate unknown XProc step types as 
>> dynamic errors
>> 
>> Pipelines that attempt to unconditionally evaluate unknown step types
>> are guaranteed to fail. It's possible that some new XProc steps may
>> have attributes or children that a previous processor could not be
>> expected to interpret.
>> 
>> In order to minimize the disruption such elements cause, while at the
>> same time making it as easy as possible to write pipelines that will
>> run in fowards-compatible mode on an earlier processor, we adopt the
>> following rules:
>> 
>> In forwards-compatible mode:
>> 
>> Steps in the XProc namespace which the processor does not recognize
>> are marked invalid. It is a dynamic error (err:XD00??) to attempt to
>> evaluate a step that is marked invalid.
>> 
>> Any subpipeline that directly contains a step which has been marked
>> invalid is also invalid.
>> 
>> If a p:when directly contains an unknown XProc step type, that entire
>> branch is marked invalid. No static analysis is performed on the steps
>> within that pipeline. It is a dynamic error (err:XD00??) if that
>> branch of the p:choose is selected at runtime.
>
> I think you *MUST* statically check the pipe bindings in the branch; not
> doing so may break the dependency graph. The steps in the when- or
> otherwise- branch may contain dependencies on steps outside p:choose,
> and not statically checking the branch may result in the p:choose being
> executed before the steps it actually depends on!

I don't think so. Only the invalid branch is being ignored. And you
can't ever run that branch so it can't matter as far as the order goes.

Consider:

  <p:choose name="choice">
    <p:when test="some-condition">
      <p:fribble>
        <p:input port="purple">
          <p:pipe step="ident" port="result"/>
        </p:input>
      </p:fribble>
    </p:when>
    <p:otherwise>
      <p:identity>
        <p:input port="source">
          <p:inline><doc/></p:inline>
        </p:input>
      </p:identity>
    </p:otherwise>
  </p:choose>

  <p:identity name="ident">
    <p:input port="source">
      <p:inline><doc/></p:inline>
    </p:input>
  </p:identity>

If you understood the fribble step, then you'd always run "ident" first.
Since you don't understand the fribble step, you might not. Suppose
you don't:

  - If some-condition is true, then the pipeline fails and it doesn't matter.
  - If some-condition is false, then you executed the branch that didn't use
    the dependency and it doesn't matter.

I suppose you might occasionally run pipelines that a V2.0 processor
would assert contain loops, but since you can never run the "looping"
path, I don't think it matters.

I don't see how a processor *can* do the dependency analysis in a pipeline
that contains a step it doesn't understand. It certainly can't be guaranteed
to build the *correct* graph, so why build one at all?

>> If a p:group inside a p:try directly contains an unknown XProc step
>> type, that group is marked invalid. No static analysis is performed on
>> the steps within that group. An attempt to evaluate the p:try will
>> immediately evaluate the p:catch with an empty errors port.
>
> Again, I think the same as with p:choose applies here and you MUST do
> some sort of static analysis in p:group in p:try and p:catch.

I'll wait and see if I've convinced you otherwise above :-)

>> If all of the branches of a p:choose or p:try are invalid, the
>> p:choose or p:try is treated as if it had no primary output port.
>
> Why?

So that if it's the last step in a subpipeline, you can't wind up with
a default binding. The alternative is to say that it is treated as if
it did have a primary output port, which would also be an error
sometimes. It's irrelevant which choice we make, but I think we have
to make one.

>> ============================================================
>> 
>> Proposal 4: Support binding defaults in forwards-compatible mode
>> 
>> In forwards-compatible mode:
>> 
>> It is not a static error to encounter an unknown XProc step type. Such
>> steps are assumed to have no primary input ports, no primary output
>> ports, and no parameter input ports. It is a static error if, under
>> these assumptions, a valid dependency graph cannot be constructed.
>> 
>> It is not a static error to encounter explicit bindings to unknown
>> ports on an XProc step type. Such bindings are assumed to be correct.
>> Dynamically, a step must ignore all inputs that appear on an unknown
>> input port and must produce an empty sequence of documents on unknown
>> output ports.
>
> +1
>
>> ============================================================
>
>> E. Dynamic errors 1
>> 
>> The following pipeline is statically valid to an XProc 1.0 processor
>> and will run without errors.
>> 
>>   <p:pipeline version="2.0">
>> 
>>     <p:choose>
>>       <p:when test="system-property('p:version') &gt; 1.0">
>>         <p:fribble/>
>>         <p:sink/>
>>       </p:when>
>>       <p:otherwise>
>>         <p:identity/>
>>       </p:otherwise>
>>     </p:choose>
>> 
>>   </pipeline>
>
>> 
>> The first p:when branch is marked invalid because it contains the
>> unknown (from the 1.0 perspective) step p:fribble. As a result, no
>> static analysis is performed on that subpipeline
>
> Again, I think you have to check the pipe bindings in the branch

Again, I'll see if I've convinced you otherwise :-)

>> and that subpipeline
>> does not contribute to the constraint that all the branches of a
>> p:choose must declare the same outputs.
>
> Hmm. Even if the when branch explicitly declared the outputs, or if the
> last step of the subpipeline is a known step?

If there are other branches that don't contain unknown steps, then they 
all have to be consistent and if one of them runs, the right thing will
happen. If the branch that contains the invalid step runs, the pipeline
will fail, so it doesn't matter.

This comes back to my earlier point that I don't think the processor
*can know* what the outputs are for a subpipeline that contains a step
it doesn't understand.

(If all the bindings are explicit, it can, but it can't know what the
implicit bindings should be, so in general it can't know.)

>> K. Static errors 2
>> 
>> This pipeline is statically *in*valid:
>> 
>>   <p:pipeline version="2.0">
>>     <p:choose>
>>       <p:when test="system-property('p:version') &gt; 1.0">
>>         <p:unknown/>
>>       </p:when>
>>       <p:otherwise>
>>         <p:unknown/>
>>       </p:otherwise>
>>     </p:choose>
>>