14 March 2001
This document does not represent the thinking of (individuals of) the
W3C XML Protocol Working Group. It is a draft describing
conformance issues regarding protocol specification. Comments about this
document should be sent to xml-dist-app@w3.org (archives).
The document also contains some comments resulting from a
discussion during the 25 April teleconference.
Last modified: $Date: 2001/11/16 21:22:53 $
Requirement R301a reads:
The XMLP specification must clearly identify conformance requirements in a way that enables the conformance of an implementation of the specification to be tested (see also the W3C Conformance requirements (W3C members only)).
Moreover, at the end of the Candidate Recommendation stage, the Working Group will have to prove to the Director that there exists implementations that are conformance to the specification. One way to achieve that is to demonstrate two interoperable implementations of each feature.
The problems raised are:
This document discusses the implications of requirement R301a, and lists proposed deliverables.
Both of those problems can be tackled by writing a test suite for XML Protocol implementations. This could be a separate document produced by the Working Group.
Each section of the specification would correspond to a series of tests. There must be simple tests focusing on each atomic feature. There should also be a set, obviously non-exhaustive, of more complex scenarios (serialization and deserialization of complex data structures, one way messages generation, RPC calls, various fault handling, etc) combining those atomic features.
An important aspect is to clearly identify what parts of the specification are mandatory for an implementation to be compliant, and what parts are optional. For example, RPC support might not be required for a minimal XML Protocol processor.
@@@ In order to encourage interoperability, maybe include in the spec: Module developers are encoraged to include conformance tests in their XML Protocol module specification.
In order for tests to be run, some communication with the tested processor will happen, which might not be a normative part of the specification. The test suite must clearly state what non-normative parts of the specification (e.g. an HTTP protocol binding) are used.
The test suite must clearly state that passing the tests is not synonym of certification by the W3C. Implementation not passing the conformance tests, on the other hand, would not be conformant to the XML Protocol specification. The only claim that could be made would be that a particular implementation is conformant to a particular version of the XML Protocol test suite.
In order for the test specification to evolve independently from the XML Protocol specification, and evolve even when XML Protocol has been released as a W3C Recommendation, the test specification must not be on the Recommendation track. It could either be released as a Note, or a Working Draft eventually becoming a Note (such as the requirements document), or some other place out of the Technical Reports environment.
At the Candidate Recommendation stage, interoperability would be demonstrated using the scenarios described in the test specification.
A good way to do that would be to implement a XML Protocol processor which would automatically validate those scenarios. Implementations could then be tested against this validator. The validator could then be provided as a public service.
If the Working Group does not have such a program to automatically test implementations against the test suite, running the scenarios described in the test suite between two implementations and checking the result "by hand" could prove interoperability.
Considering the size of the Working Group, it looks reasonable to form a subgroup whose task would be to write a test specification in order to comply to requirement R301a. This group would also be responsible to prepare the Candidate Recommendation interoperability tests, since they would be really close to the test suite.
If there is enough time and volunteers, it would also be a good idea to write some software which would run the tests described in the test specification, in order to ease the Working Group's task at the Candidate Recommendation stage, and to provide a useful service to the Web community.
26 April 2001
The following are notes are I took after the discussion during the 25 April telecon:
July 16, 2001
This part is really work in progress.
The SOAPBuilders community runs interoperability tests. Those tests and in particular the results for each implementations would be useful for the Candidate Recommendation stage exit criteria.
Links:
One of the entrance criteria for the Proposed Recommendation status is to demonstrate that:
The round 1 SOAP interoperability tests mainly test the understanding of different data types of the implementations using RPC calls over HTTP.
The round 2 of interoperability tests proposed is more complete. The round 2 SOAP interoperability tests is a more complete set of tests identical to the round 1 tests. The round 2 SOAP group "B" interoperability tests runs test on the understanding and manipulation of complex types by implementations. The round 2 header processing interoperability tests tests how implementations handle mandatory and optional headers that they do or don't understand, and whether they correctly ignore the headers when needed.
The SOAPBuilders test do test a subset of features of a SOAP/1.1 implementation. The SOAP Version 1.2 specification must be scanned to locate the untested features and what complementary tests are needed to have a full test suite.
For example, the processing model could be tested with the examples used during the June 2001 face-to-face, 5 June pm.
25 July 2001
I just realized that this document was becoming more of a journal than a reference document as it was at the beginning, which is fine.
A list of testable assertions extracted from the specification has been generated. The next steps, as far as I currently see them are (not sure about the order yet):
Having looked at what needs to be tested more closely, it seems that writing a validator is feasible with a reasonable amount of work.
31 July 2001
The SOAPBuilders tests (round 1 and 2) are now listed in the list of testable assertions document.
They do a pretty extensive test of the encoding part of the specification (A21, A22, A24) and of the RPC encoding part (A32). The round 2 tests also test the actor and mustUnderstand attribute (A2, A3, A5, A6), although they do not cover all the cases.
However, the tests probably need to be formalized a little: how to decide whether the test is successful, etc.
I have in mind the following:
Some assertions are not tested at all, e.g. when the SOAP node behaves as an intermediary (A7) or behavior when an incorrect version of a SOAP message is received (A8, A10, A18).
13 September 2001
Presentation of the status of conformance at the XML Protocol Working Group face-to-face (see minutes).
24 September 2001
Following the decision to go ahead with a test suite, I have converted the list of testable assertions into a more formal XML document. This should become the test suite in order to test implementations' features.
16 November 2001
I have updated the collection of testable assertions according to the latest editors' copy of the specification and created a snapshot for the face-to-face meeting.