This document defines a common Operational Framework for building
conformance test suites and tools for W3C specifications. It presents
operational and procedural guidelines for groups undertaking conformance
materials development. This document is one in a family of QA Framework
documents, which includes the other existing or in-progress specifications:
Overview & Process; Specification Guidelines; and, Technical
Guidelines.
This document is a Note, made available by the W3C Quality Assurance Activity (QA) for
discussion on the QA email list.
This version incorporates the discussions at the first QA face-to-face,
and supersedes the first, incomplete draft. Open issues and other
incompletions are flagged with "@@". Where applicable, the presentation style
is Guidelines-plus-Checkpoints, similar to WAI standards. Where appropriate,
some documents in this Framework family (including this one) have an
associated and copiously hyperlinked "Examples & Techniques" document, in
which details and especially discretionary choices are elaborated. In this
document version, guidelines and checkpoints are not yet numbered or
formatted [@@we need to do this before release].
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or obsoleted by other documents at any time. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress".
Please send comments to www-qa@w3.org.
1. Introduction
1.1 Terminology
2. Operational framework
2.1 W3C Process considerations
2.2 WG Charter considerations
2.2.1 Define commitment to QA
2.2.2 Where to produce test suite
2.2.3 Staffing commitments
2.3 Principal aims of TS development
2.3.1 Goals
2.3.2 Scope of W3C QA framework
2.3.2.1 Certification
2.3.2.2 Branding
2.4 Starting the test suite
2.4.1 Resources/staffing
2.4.2 Starting & Synchronizing with spec
progression
2.4.3 methodology
2.4.4 Transferring a TS from outside W3C(to
revise)
2.4.5 Test suite process document
2.4.5.1 Tests Contribution
2.4.5.2 IPR and copyright
2.4.5.3 Communication
2.4.5.4 Tests review process
2.4.5.5 Test materials publication
2.4.5.6 Usability of the test suite by
external parties
2.4.6 Results publication
2.5 Test suite maintenance
2.5.1 Resources/staffing
2.5.2 Contributions/review
2.5.3 Support for spec versioning/errata
2.5.4 Appeal process
2.5.5 Maintenance of the test suite after WG
disappears
2.6 WG relationship to QA Activity
2.7 Logos & branding.
2.8 other Ops&Proc topics
3. Conformance
4. Acknowledgments
5. References
(This is done by my rudimentary XSLT process--it numbers subsequent h2..h5
headers and pulls them forward, here, as a hyperlinked numbered TOC)
1. Introduction
This document defines a common Operational Framework for building
conformance test suites and tools for W3C specifications. It presents
operational and procedural guidelines for groups undertaking conformance
materials development. Therefore it ames to provide step by step instructions
and options on how to set up, aquire or develop and maintain the test suite.
It also represents relationship between W3C process and QA and charter
considerations for Working Groups that are about to start.
This document is one in a family of QA Framework documents described in
the QA Framework Introduction. It references other documents in the family,
such as:
- Introduction
- Specification guidelines
- Technical guidelines
It also refers to several examples of the succesful QA work within W3C and
beyond.
The first 2 sections give short description of how QA is reflected in the
current W3C Process document and what should be addressed by WG in the
charter. Then we describe in details the scope of TS development in W3C WG,
what are the goals of QA work within WG. The main 2 sections of the document
describe in details the procedure for starting and maintenance of the test
suite taking into account level of commitment that WG took in the charter.
The "Relationship to QA Activity" section ties QA process within Working
Group with resources available from QA Activity.
...we probably need to include at least the 2nd half of the following
terminology section into the framework...
1.1 Terminology
One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY].
Unusual terms in this framework specification are defined when first used,
and most generally useful QA-specific terms will eventually be in the
Glossary.
(@@Ed. note -- following is not uniformly implemented yet.)
Although the guidelines of this framework are nominally informative, the
key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD",
"SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in
RFC 2119 [RFC2119], and will be used as if the
guidelines are normative.
2. Operational framework
2.1 W3C Process considerations
In the current W3C Process Document two QA related
guidelines may be found:
- "[...]groups may produce technical reports, review the work of other
groups, develop sample code or test suites, etc." (sec 3.)
- "W3C should make every effort to maintain its Recommendations (e.g., by
tracking errata, providing test bed applications, helping to create test
suites, etc.)" (sec 4.)
Nevertheless, there is a body of
contemporary QA experience and activity amongst the working groups. The Matrix identifies more than a score of test suites and
validators, in various states of development.
Guideline. Working Groups should consider completion and
publication of significant test materials to be a criterion for CR-exit and
PR-entrance.
Explanation & Rationale. A de facto convention of "CR-exit" seems to
be a common process goal amongst a number the existing test suite efforts
(e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and
implementations to develop in parallel -- each is a help to the development
of the other. And, the Process Document does address
implementation requirement:
- "Advancement of a technical report to Candidate Recommendation is an
explicit call for implementation experience to those outside of the
related Working Groups or the W3C itself." (sec 5.)
- "Preferably, the Working Group should be able to demonstrate two
interoperable implementations of each feature." (sec 5.2.4, about
PR-entrance criteria)
2.2 WG Charter considerations
Guideline. Working Group charters should address goals and plans
for test suites and tools.
Explanation & Rationale. In approving and initiating the QA Activity,
W3C has endorsed the principle that, in order for W3C Web standards to
achieve full interoperability and access to all, the quality of
implementation must be given as much attention as the standards' development.
The principal factor for improving the quality of implementation is early
availability of conformance test materials. Therefore, conformance test
materials -- test suites and tools -- must be considered as WG deliverables
the same as the standards themselves. Accordingly, they must be addressed in
the WG charter.
2.2.1 Define commitment to QA
Checkpoints:
- Checkpoint. New WG charters SHALL address test suite plans.
- Checkpoint. Existing WG charters SHOULD (MAY?) be amended with
test suite plans.
- Checkpoint.Charters should express commitment to general intent
and plans.
Explanation. Detailed planning and specification of test materials is
inappropriate in the charter, as it binds the WG inflexibly to test materials
implementation specifics for which flexibility may be needed. There are
numerous possible levels of commitment that a WG could make, based on
anticipated WG resources, existence of outside efforts, and the WG's current
stage in the process [REC-TRACK]. A range of
possible commitments has been identified. See "Examples & Techniques"
[@@EX-TECH].
[...in EX-TECH, we would have:
The range of possibilities for WG
commitment is:
- WG promises to think about testability when writing their
documents.
- WG makes an effort to have QA expertise among their membership.
- Beyond a few normative examples, WG aims to have numerous normative
cases in the body of the Rec.
- WG commits that a test suite, not necessarily complete and thorough,
will exist somewhere before they go to Rec.
- WG commits that they review and fully approve a test suite, not
necessarily complete and thorough, before they go to Rec.
- WG commits to reviewing test cases on a continuing basis, including
after going to Rec.
- WG will attempt to generate a complete catalog of test cases before
going to Rec.
- WG insists on a complete catalog and partial implementation of a test
suite before going to Rec.
- WG insists on a test suite, not necessarily complete and thorough, and
provides an official test harness, before going to Rec.
- WG commits to writing test cases themselves and delivers a test harness
and suite at some point.
- WG insists on a complete test suite before going to Rec.
This 11-point enumeration is derived from proposal to the QA mail list, after the 4/2001 QA
Workshop.
The further down the scale (i.e., the higher the number) that the WG
commits, the better it is for achieving W3C's ultimate goals of
interoperability and accessibility.
/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]
2.2.2 Where to produce test suite
Checkpoint. The WG charter should address where and how conformance
test materials will be produced.
Explanation. There are several possibilities for producing conformance
test materials:
- build them entirely within W3C (normally, the WG). Example: SVG test
suite.
- build them externally and transfer to W3C. Example: XML test suite.
- build procedures for assembling contributions from multiple sources.
Example: XSLT test suite (a work in progress, this is actually being done
in OASIS XSLT/Xpath Conformance TC, but it could just as well work within
a W3C WG).
- hybrids of the above.
2.2.3 Staffing commitments
Checkpoint. The WG charter TS plans should address staffing
commitments.
Explanation. There will be at least some staffing required from the WG,
for any of the acceptable options presented so far. Depending upon the
general intent and plan, and how the test suite will be built, the commitment
can range from minimal to significant. See "Examples & Techniques"
[@@EX-TECH].
[...in EX-TECH, we would have:
-- quick tabular summary of size-of-effort data, from section 2.3 of
http://www.w3.org/Graphics/SVG/Test/svgTest-manual.htm.
-- quantify some examples, e.g., estimated XML uptake effort?
/EX-TECH, end of "Examples & Techniques" content, back to qaFrame.]
2.3 Principal aims of TS development
2.3.1 Goals
Common frameworks for W3C QA activities are obviously shaped by the
high-level goals for development and deployment of test materials. Before
further developing the framework guidelines, it is useful to state some of
these high-level goals.
The over-arching goal is stated in the QA Activity Statement:
- to enable W3C Web standards to achieve full interoperability and access
to all, by addressing the quality and correctness of implementions.
More specific goals of test materials development should include:
- provide self-assessment tools and materials for implementation
builders;
- help implementation builders to achieve interoperability of
implementations;
- help users to verify compliance of implementations with the stated
conformance requirements of their respective specifications;
2.3.2 Scope of W3C QA framework
The goals of a common W3C QA framework do not encompass:
- measurement of performance, scalability of implementations, or their
assesment against any other "goodness" criteria, except what is stated as
a conformance requirement of their respective specification.
- certification of implementations
2.3.2.1 Certification
While certification may have a positive quality assurance role in
standards-based Web environments, W3C does not presently intend to initiate
or offer any certification services. Nevertheless, test suites and materials
development should not arbitrarily preclude:
- usability by accredited third parties for testing purposes including
certification services.
Factors of rigor, objectivity, and defensibility are key determinants of
usability for certification, and these same properties are positive assets of
any test materials.
Note: although tests suites may be used for compliance
verification, Working Group MAY make test suites users aware of the fact
that
- passing all of the tests does not guarantee full compliance of
implementation to specification
- failing test suite means failing tests for specific feature they
target
2.3.2.2 Branding
Certification aside, some W3C activities support "branding" -- use of a
conformance icon to indicate some level of standard compliance. Examples:
HTML validation, CSS validation, WCAG compliance. While these content
branding schemes are relatively non-controversial, there are nevertheless
some serious issues which should be addressed -- particularly in the contexts
of user-agent and API conformance -- before any branding-related goals are
articulated.
2.4 Starting the test suite
This section will guide through setup of QA process in the Working Group.
For each checkpoint in this group WG should take one of the options depending
on decisions taken in the charter for test suite origin and level of
commitment.
2.4.1 Resources/staffing
Guideline: WG should address resources/staffing for the test suite
development/maintenance.
Checkpoint. Depending on the plan for origin of the test suite WG
should appoint test QA moderator from either of:
- - some WG staff (and other W3C members)
- - some W3C QA staff?
- - some 3rd party (e.g., for TR analysis)?
2.4.2 Starting & Synchronizing with spec progression
- start: the sooner, the better
- early benefits of, e.g., TA analysis
- when to start TS work (WD, CR, ...)
- coordination with spec (REC) progression
2.4.3 methodology
Before starting test suite development, test suite implementer (WG or 3rd
party) should look at Technical Guidelines and decide
what approach to take for test materials organization, tests criteria,
etc.
2.4.4 Transferring a TS from outside W3C(to revise)
In case if WG and some external party decided to move the test suite to
W3C from outside the following items should be resolved
- parties must to sort out IPR
- charter amendment? yes.
- need to sort out new home (WG), staffing, and other issues at "Charter"
level of detail.
(several guidelines and checkpoints can be distilled from XML transfer
example)
- after the transfer, test suite process MUST be established for future
contributions.
2.4.5 Test suite process document
Guideline: WG should publish Test process document.
Test suite process document is a central QA document describing how
testing process is set up within WG. Examples of the test process document
could be found at DOM TS process, XML Schema TS
process, XML Protocol TS process, etc. It is required part of QA process in
WG and should address the following questions:
2.4.5.1 Tests Contribution
Checkpoint. WG Test suite process document should explain
contribution process.
Process document must describe where to submit test materials and who
should be notified (moderator) of submission. Contribution process MAY
describe the format of contributed material, it may contain validation
harness, utilities that facilitates tests creation, templates.
2.4.5.2 IPR and copyright
Checkpoint. WG Test suite process document should provide licenses
for submitted and published test materials. WG MAY choose either W3C Document
or W3C Software license, but W3C Document license is recommended.
2.4.5.3 Communication
Checkpoint. WG Test suite process document should provide means for
communication regarding test suites.
WG MAY choose mailing list to send all comments and announcements to.
2.4.5.4 Tests review process
Checkpoint. WG Test suite process document MAY set/describe review
procedure for submitted test materials
[Example, need to formalize into checkpoints
- 1. Each contribution should be reviewed by at least one vendor other
then submitter for it's eligibility. Any vendor involved in creation of
WG test collection other then submitter may review tests being
contributed.
- 2. Eligibility of tests should be judged by:
- 2.1 The accuracy of the test. Accuracy of a test is determined by a
judgment of the reviewer whether the test case actually tests what
the submitter states the test case tests. The accuracy is the extent
to which the test matches the cited parts of the Specification. If it
does not match, or only partially matches, the test should be
considered inaccurate.
- 2.2 The scope of the test. Test should target MUST provisions of
the specification and not MAY/SHOULD provisions. The test should
target only conformance requirements. Data values limits, memory or
performance limits should not be targets to the test unless those are
specified in minimum conformance requirements.
- 2.3 The clarity of the test. Clarity of a test is a determination
of whether the aspect being tested is clearly described with the
anticipated results acceptably explained.
- 2.4 The clarity of aspect of the Specification being tested. The
Test Suite aims to test parts of the Specification and errata that
aren't vague.
- 3. After review results should be sent back to submitter for his
feedback.
- 4. In case of disagreement, request for clarification should go to WG,
tests in question should be left off the test suite till W3C WG clarifies
the issue.
- 5. Only those tests that are agreed to be correct by reviewers and
submitter or approved by W3C WG clarification should be published.
2.4.5.5 Test materials publication
Checkpoint. WG Test suite process document should point to the web
page with published test materials.
WG MAY describe the way it publishes test materials and it is recommended
to use one of practices described in Technical
Guidelines
2.4.5.6 Usability of the test suite by external parties
Checkpoint. WG Test suite process document should mention usability
of test materials by external parties.
2.4.6 Results publication
WG should encourage vendors to publish results for their implementations
including/describing test harness that would allow anyone to reproduce the
results.
2.5 Test suite maintenance
Once test suite development initiated, its maintenance requires almost the
same amount of resources
2.5.1 Resources/staffing
Checkpoint. Working Group SHOULD appoint task force or another
entity to maintain the test suite
It is recommended errata task force be involved in test suite maintenance,
since this will help to keep test suite and specification in sync.
Maintenance group should establish tracking system to track issues/changes to
the test suite.
2.5.2 Contributions/review
Contributions may arrive through out all test suite/standard's life cycle.
WG should be able to maintain contribution
and review procedures described earlier.
2.5.3 Support for spec versioning/errata
Test suite structure needs to support spec versioning/errata (to include
example from XSLT). The following process MAY apply:
- Initiating test suite change/update. As soon as errata is accepted or
new version/edition of the Rec is published, errata task force(or test
suite maintenance group) should identify affected test cases and initiate
updates to the test suite by adding entries to the tracking system with
status "ACTIVE"
- Providing change to the test suite for review. Test case submitter or
initiator of the change MAY provide updated version for the affected
tests. If no changes provided test maintenance group should find the way
to provided it. Once changes are ready, corresponded status in the
tracking system should be changed to "RESOLVED"
- Review/acceptance of the change. Tests change MAY go through the same
tests review process presented earlier. If
test changes are accepted, they should be added to the test suite with
indicator of errata/version. Old tests should stay in the test suite
marked with last errata/version level they comply with. Status should be
changed to "CLOSED". If tests are rejected, they should go back to
provider of the change with resolution. Status should be reverted back to
"ACTIVE"
2.5.4 Appeal process
Test Suite process identifies communication channel for appeals. Once
appeal for specific tests comes in the following process MAY apply:
- Initiating appeal
- Review of appeal
- Providing change to the test suite for review in case of appeal
acceptance
- Review/acceptance of the change
2.5.5 Maintenance of the test suite after WG charter expires
[Active issue]If there is still active work on the test suite even if the
WG is not re-chartered to handle the work, then it is up to the Director to
determine how this work should be done. It MAY be assigned to a Coordination
Group or a related WG or the Team could take care of minor items.
2.6 WG relationship to QA Activity
The QA Activity anticipates different relationships to the various Working
Groups. On the one hand, the QA-specific needs of the Working Groups will
vary depending upon their resources and experience, as well as their stage in
the Recommendation track. On the other hand, the W3C
processes may evolve so that QA-specific process requirements are tied to
milestones in the Recommendation track.
Potential relationships between Working Groups and the QA Activity
include:
- Liaison & consultation. In an assistive role, the QA Activity
provides consultation to Working Groups in planning and building their
test materials. This should be the baseline relationship, applicable to
all Working Groups.
- Active reviews. The QA Activity provides detailed review and assessment
of Working Group plans and test materials, on a requested basis (QA
resources permitting). It is still an open policy question, whether the
QA Activity will pro-actively initiate reviews, and whether it has
adequate resources to do so.
- Adjudicate entry and exit criteria. Although there are currently no W3C
process requirements for QA-specific entry and exit criteria at
milestones on the Recommendation track, Working Groups could define such
for themselves.
- Direct QA staff participation. QA Activity members and staff could, in
some cases, participate directly in the planning and building of test
materials for some Working Groups.
Key determinants of the WG/QA relationship will include level of available
QA Activity resources, and the evolution of W3C policy on QA requirements.
2.7 Logos & branding.
- WAI does this for content; CSS also and basic HTML validation.
- Controversial topic, as email thread and UAAG CR comments show.
- Taxonomy dependent: content (WCAG, CSS, etc) is less
difficult/controversial; UAs will be contentious.
2.8 other Ops&Proc topics
- ...any?...
3. Conformance
[This document should have a conformance clause. Uncertain what it should
say now, as it is unclear what is the normative force of this document. At
least it should probably address what is involved in (possibly voluntary)
conformance to this document. Precisely what it might say is an issue that
prehaps we can address later?]
4. Acknowledgments
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C),
- Karl DuBost (W3C),
- Kirill Gavrylyuk (Microsoft)
- Lofton Henderson (CGM Open),
- David Marston (Lotus Development Corp),
- Lynne Rosenthal (NIST).
- [...extend this list to include all 1st f2f participants?...]
In addition ideas expressed by [@@...there may be some others in the email
archive, e.g., Andrew Thackrah, whose ideas have been excerpted and who
should be credited...] in QA email and telephone discussions have contributed
to the contents of this document.
5. References
[@@Ed. note: This has not yet been cut down from original all-in-one
Framework document. It should be edited to only include those references
which actually pertain to this document.]
- EXTERN-TA
- QA activity email thread about third-party
participation in test materials production.
- MATRIX
- W3C-wide conformance activity survey covering all the WGs, "The Matrix".
- PROCESS
- W3C Process
Document, 19 July 2001.
- TAXONOMY
- QA Activity test
taxonomy, a classification scheme for conformance test
materials.
- QAIG
- Quality Assurance Interest
Group of the W3C QA Activity.
- QAWG
- Quality Assurance Working
Group of the W3C QA Activity.
- DOM WG TS
- Process document for DOM WG
Test suite.
- REC-TRACK
- Stages and milestones in the W3C
Recommendation Track, per the Process
Document (section 5.2).
- RFC2119
- Key words for use in
RFCs to Indicate Requirement Levels, March 1997.
- SVGTEST
- SVG Working Group's test suite resource
page.
- WCAG10
- Web Content Accessibility
Guidelines, version 1.0, W3C Recommendation, 5 May 1999.
- WG-QA-RANGE
- Email
proposal by David Marston, on the QA public mail list, for range of
WG commitment levels to conformance test materials production.
- XMLTEST
- OASIS XML Conformance TC's XML
test suite resource page.
- XSLT-TEST
- OASIS XML Conformance TC's XSLT/Xpath
test suite resource page.
- ...ETC...
- ...etc...etc...etc...