This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of QA Framework documents, which includes the other existing or in-progress specifications: Overview & Process; Specification Guidelines; and, Technical Guidelines.
This document is a Note, made available by the W3C Quality Assurance (QA) Activity for discussion on the QA email list.
This version incorporates the discussions at the first QA face-to-face, plus subsequent QA WG teleconferences, and supersedes all previous drafts. Open issues and other incompletions are flagged with "@@". The number of such are mostly minor editorial issues in this draft, except for the issue of the conformance clause. This version intended as the final WG review version before final editing for first public working draft (FPWD).
Where applicable, the presentation style is Guidelines-plus-Checkpoints, similar to WAI standards. Where appropriate, some documents in this Framework family (including this one) have an associated and copiously hyperlinked "Examples & Techniques" document, in which details and especially discretionary choices are elaborated. In this document version, guidelines and checkpoints are not yet numbered or formatted.
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
Please send comments to firstname.lastname@example.org.
2. Operational framework
2.1 W3C Process considerations
2.2 WG Charter considerations
2.2.1 Define commitment to QA
2.2.2 Where to do TS development
126.96.36.199 Where to produce test suite
188.8.131.52 Test materials home
2.2.3 Staffing commitments
2.3 Principal aims of TS development
2.3.2 Scope of W3C QA framework
2.4 Starting the QA work
2.4.2 Starting & Synchronizing with spec progression
2.4.3 Testing Methodology
2.4.4 Transferring a TS from outside W3C
2.4.5 QA process document
184.108.40.206 Tests Contribution
220.127.116.11 IPR and copyright
18.104.22.168 Tests review process
22.214.171.124 Test materials publication
126.96.36.199 Usability of the test suite by external parties
2.4.6 Results publication
2.5 Test materials maintenance
2.5.3 Support for spec versioning/errata
2.5.4 Appeals process
2.5.5 Maintenance of the test suite after WG charter expires
2.6 WG relationship to QA Activity
2.6.1 Liaison and consultation.
2.6.2 Active reviews.
2.6.3 Adjudicate entry and exit criteria.
2.6.4 QA resources supplement.
2.6.5 Resolution of external QA requests
This document defines a common Operational Framework for building conformance test suites and tools for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. Therefore it aims to provide step by step instructions and options on how to set up, acquire or develop and maintain the test suite. It also represents relationship between W3C process and QA and charter considerations for Working Groups that are about to start.
This document is one in a family of QA Framework documents described in the QA Framework Introduction. It references other documents in the family, such as:
It also refers to several examples of the successful QA work within W3C and beyond.
The first 2 sections give short description of how QA is reflected in the current W3C Process document and what should be addressed by a WG in its charter. Then we describe in detail the scope of TS development in W3C WG, what are the goals of QA work within WG. The main 2 sections of the document describe in detail the procedure for starting and maintenance of the test suite taking into account level of commitment that WG took in the charter. The "Relationship to QA Activity" section ties QA process within Working Group with resources available from QA Activity.
One of the deliverables of the QA Activity is a Glossary [@@GLOSSARY]. Unusual terms in this framework specification are defined when first used, and most generally useful QA-specific terms will eventually be in the Glossary.
Although the guidelines of this framework are nominally informative, the key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119], and will be used as if the guidelines are normative.
In the current W3C Process Document two QA related guidelines may be found:
Nevertheless, there is a body of contemporary QA experience and activity amongst the working groups. The Matrix identifies more than a score of test suites and validators, in various states of development.
Guideline. Working Groups should consider completion and publication of significant test materials to be a criterion for CR-exit and PR-entrance.
Explanation & Rationale. A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG, UAAG,...). This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:
Guideline. Working Group charters should address goals and plans for test suites and tools.
Explanation & Rationale. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials. Therefore, conformance test materials -- test suites and tools -- must be considered as WG deliverables the same as the standards themselves. Accordingly, they must be addressed in the WG charter.
Checkpoint. New WG charters SHALL address test suite plans.
Checkpoint. Existing WG charters SHOULD be amended with test suite plans.
Checkpoint. Charters SHOULD express commitment to general intent and plans.
Explanation. Detailed planning and specification of test materials is inappropriate in the charter, as it binds the WG inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a WG could make, based on anticipated WG resources, existence of outside efforts, and the WG's current stage in the process [REC-TRACK].
A range of possible commitments has been identified:
This 11-point enumeration is derived from a proposal to the QA mail list, after the 4/2001 QA Workshop.
The further down the scale (i.e., the higher the number) that the WG commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.
Guideline. Working Group commitment to QA SHOULD be not lower then level 4 - produce some test suite before specification goes to Recommendation.
Checkpoint. The WG charter should address where and how conformance test materials will be produced.
Explanation. There are several possibilities for producing conformance test materials:
Guideline. Test suites and tools should ultimately reside in W3C.
Checkpoint. W3C should provide secure and reliable repository location for the test materials.
Checkpoint. W3C should have ultimate responsibility -- though not necessarily sole responsibility -- for the ongoing evolution of test suites and tools, after some milestone of operational maturity of the materials.
Explanation & Rationale. This does not necessarily mean that the WG must solely build the test suites and tools -- in fact there are good reasons why external parties should have a strong participation during the building of the materials (see @@below). But once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), they should reside in and be the responsibility of W3C, and in particular the Working Group responsible for the subject standard.
If the WG responsible for a standard ceases to operate, then another home for the test materials would be found within W3C. Those responsible for ongoing interpretation and maintenance (errata) of the standard should be designated.
Guideline. The WG charter TS plans should address staffing commitments.
Explanation. There will be at least some staffing required from the WG, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques" [@@EX-TECH].
Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.
The over-arching goal is stated in the QA Activity Statement:
More specific goals of test materials development should include:
The goals of a common W3C QA framework do not encompass:
While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, test suites and materials development should not arbitrarily preclude:
Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.
Checkpoint. Working Group SHOULD ensure that test materials are documented, self contained and reusable by external parties
Guideline. Although tests suites may be used for compliance verification, Working Group SHALL make test suites users aware of the fact that
The last item requires some elaboration. Depending on the nature of the test, it may actually be sufficient to judge non-compliance of the subject. Based on taxonomy the following is valid:
Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.
This section will guide through setup of QA processes in the Working Group. For certain checkpoints in this group WG should take one of the options depending on decisions taken in the charter for test suite origin and level of commitment.
Guideline. WG should address resources/staffing for the test suite development/maintenance.
Checkpoint. Depending on the plan for origin of the test suite WG should appoint test QA moderator from either of:
Guideline. WG SHOULD start synchronizing specification writing and test materials development as early as possible at the Working Draft phase of the specification cycle
Explanation and Rationale. There are several strong benefits from starting test materials development on the early stages:
Checkpoint. WG SHOULD introduce intermediate milestones for test test materials completion, bounded to Working Drafts/Candidate Recommendation publications
Checkpoint. Latest version of the spec draft and latest version of the published test materials should be synchronized.After entering Recommendation phase every update to the specification through Errata or new versions MUST be reflected in the published test materials as outlined in Test Materials Maintenance.
Before starting test suite development, test suite implementer (WG or 3rd party) should look at Technical Guidelines and decide what approach to take for test materials organization, tests criteria, etc.
Guideline. Review Technical Guidelines before starting QA work.
In case when WG and some external party decide to move the test materials (TM) to W3C from external entity/group (EG) the following procedure is recommended:
Before transfer WG with the help of QA WG:
After transfer, initial process setup
First W3C public release of new TM
After first public release, TM enters maintenance phase which is described below.
Guideline. WG shall publish a QA process document.
A Working Group QA process document is a central QA document describing how testing process is set up within the WG. Examples of the test process document can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is a required part of QA process in WG and should address the following questions:
Checkpoint. The WG Test suite process document SHALL explain contribution process.
The process document must describe where to submit test materials and whom to notify (moderator) of submission. Contribution process SHOULD describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc.
Checkpoint. WG Test suite process document shall define the licenses applicable to submitted and published test materials. WG MAY choose either the W3C Document or the W3C Software license, but the W3C Document license is recommended.
Checkpoint. WG Test suite process document shall provide means for communication regarding test suites.
WG MAY choose mailing list to send all comments and announcements to.
Checkpoint. WG Test suite process document SHOULD set/describe review procedures for submitted test materials
The following example may serve as a guideline for tests review process
Checkpoint. WG Test suite process document should point to the web page with published test materials.
WG MAY describe the way it publishes test materials and it is recommended to use one of practices described in Technical Guidelines
Guideline. WG Test suite process document SHOULD address usability of test materials by external parties.
As stated before in Certification section WG should make sure that TM are self-contained and usable and
Checkpoint. Process document should contain disclaimers described in Certification section
WG should encourage vendors to publish results for their implementations including/describing test harness that would allow anyone to reproduce the results.
Once test suite development is initiated, its maintenance requires an equally significant amount of resources.
Guideline. Working Group SHOULD find/allocate resources for test suite maintenance
Checkpoint. Working Group SHOULD designate a task force or another entity to maintain the test suite, as well as maintenance procedures.
It is recommended that the errata task force be involved in test suite maintenance, since this will help to keep test suite and specification in sync. Maintenance group shall establish tracking system to track issues/changes to the test suite.
Contributions may arrive through out all test suite/standard's life cycle. WG should be able to maintain contribution and review procedures described earlier.
Test suite structure SHOULD support spec versioning/errata (to include: example from XSLT). The following process MAY apply:
Test Suite process MUST identify the communication channel for appeals of tests validity. Once appeals for specific tests come in, the following process MAY apply:
[Active issue]If there is still active work on the test suite even if the WG is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:
The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track.
Potential relationships between Working Groups and the QA Activity include:
Key determinants of the WG/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.
In an assistive role, the QA Activity provides consultation to Working Groups in planning and building or acquiring of test materials. This should be the baseline relationship, applicable to all Working Groups. Consultation may be provided in formal and informal ways. Informal questions MAY be sent to email@example.com. In order to make assistance quick and accurate the following process is introduced for formal requests:
[@@tbd...This document should have a conformance clause. Uncertain what it should say now, as it is unclear what is the normative force of this document. At least it should probably address what is involved in (possibly voluntary) conformance to this document. Precisely what it might say is an issue that must be addressed before FPWD.]
The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:
[@@Ed. note: This has not yet been cut down from original all-in-one Framework document. It should be edited to only include those references which actually pertain to this document.]