This document defines a common Operational Framework for building conformance test materials for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Materials Guidelines.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest status of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.
This version is the first public Working Draft, and incorporates the discussions at the first QA face-to-face, plus several subsequent QA Working Group [QAWG] teleconferences, and supersedes all previous drafts. It is expected that updated WD versions of this document will be produced regularly, along with other members of the Framework documents family. Future progression of this document beyond Working Draft is possible, but has not yet been determined.
This part of the Framework document family will eventually have an accompanying "Process & Operational Examples and Techniques" (in progress). Some of the lengthier examples and "how to" text of this current guidelines document version will be moved to the "Examples and Techniques" document.
Please send comments to firstname.lastname@example.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that any mail sent to this list will be publicly archived and available, do not send information you wouldn't want to see distributed, such as private data.
Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".
A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.
1.1 Motivation for this Guidelines document
1.2 Navigating through this document.
G 1. In charters, describe QA goals, criteria, deliverables and resources
G 2. In the Working Group charter, address where and how conformance test materials will be produced.
G 3. Ensure that the goals and scope of QA development are clear
G 4. Allocate staff for the QA process.
G 5. Synchronize specification and test materials development
G 6. Produce QA process document.
G 7. Plan the transfer of test materials to W3C if needed
G 8. Plan for test materials maintenance
3. Working Group relationship to QA Activity
3.1 Liaison and consultation.
3.2 Active reviews.
3.3 Adjudicate entry and exit criteria.
3.4 QA resources supplement.
3.5 Resolution of external QA requests
This document describes the process and operational activities for building conformance test suites and tools for W3C specifications. The primary goal is to assist the W3C Working Groups (WGs) by providing guidelines for the planning, development and deployment of conformance test materials. In this guideline, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance.
This document is part of a family of QA Framework documents designed to improve the quality of W3C specifications as well as their implementation by solidifying and extending current quality practices within the W3C. The QA Framework documents are:
The guidelines are intended for all Working Groups as well as developers of conformance materials for W3C specifications. Not only are the Working Groups the consumer of these guidelines, they are also key contributors. The guidelines capture the experiences, good practices, activities, and lessons learned of the Working Groups and present them in a comprehensive, cohesive set of documents for all to use and benefit from. The objective is to reuse what works rather than reinvent and to foster consistency across the various Working Group quality activities and deliverables.
As the complexity of W3C specifications and their interdependencies increases, quality assurance becomes even more important to ensuring their acceptance and deployment in the market. There has been a growing awareness and interest in conformance and quality. In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of the implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials.
Although not explicitly stated, the W3C Process Document supports the development of conformance test materials.
"[...] groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (see Process Document, section 3.)
"W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.)" (see Process Document, section 5.2.5, "Ongoing work".)
In an effort to meet these suggestions and address the implementation requirements of the Process Document, some Working Groups have included the development of conformance materials as part of their CR-exit and PR-entrance criteria. Examples include Scalable Vector Graphics (SVG), User Agent Accessibility Guidelines (UAAG), and Extensible Style Language - Formatting Objects (XSL-FO). This makes sense, since it is natural for test suites and implementations to develop in parallel - each is a help to the development of the other.
There is already a body of contemporary QA experience and activity amongst the Working Groups. The Matrix identifies more than a score of test suites and validators, in various states of development. Moreover, many Working Groups have already established procedures, techniques and tools for developing test materials (e.g., Document Object Model - DOM). It makes sense to capitalize on what has already been done and share that with those who are starting out and those who are already in the process of developing conformance materials.
The Guidelines in the document are organized chronologically starting with those that apply at the formation of a Working Group (e.g., charter considerations) and through a progression of guidelines leads the reader through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials. This document is applicable to all Working Groups, including those that are being rechartered or already exist. Working Groups may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work.
This document employs the WAI (Web Accessibility Initiative) model for representing guidelines or general principles for the development of conformance materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:
The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:
Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied.
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as defined in RFC 2119 [RFC2119].
Unusual terms in these framework documents are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].
Guideline 1. In charters, describe QA goals, criteria, deliverables and resources
In approving and initiating the QA Activity, W3C has endorsed the principle that, in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials (TM). Therefore, conformance test materials -- test suites and tools -- must be considered as Working Group deliverables the same as the standards themselves. Accordingly, they must be addressed in the Working Group charter.
Checkpoint 1.1. In the Working Group charter include conformance test materials plans. Plan to have at least a commitment to "QA level three". [Priority 1]
Detailed planning and specification of test materials is inappropriate in the charter, as it binds the Working Group inflexibly to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a Working Group could make, based on anticipated Working Group resources, existence of outside efforts, and the Working Group's current stage in the process [REC-TRACK].
Based on level of testability of the specification and extent of test materials, a range of possible commitments has been identified.
|Level||testability of the specification||extent of test materials|
|1.||Working Group plans regular specification reviews for testability, following Specification Guidelines|
|2.||Working Group plans regular specification reviews for testability, following Specification Guidelines. Beyond a few normative examples, Working Group aims to have numerous normative use cases in the body of the Recommendation.|
|3.||Working Group plans regular specification reviews for testability, following Specification Guidelines, and has numerous normative use cases in the body of the Recommendation.||Working Group commits that a test suite for the specification, not necessarily complete and thorough, will exist before specification becomes Recommendation.|
|4.||Working Group plans regular specification reviews for testability, following Specification Guidelines, and has numerous normative use cases in the body of the Recommendation.||Working Group commits that they review and fully approve a test suite, not necessarily complete and thorough, before specification becomes Recommendation.|
|5.||Working Group plans regular specification reviews for testability, following Specification Guidelines, and has numerous normative use cases in the body of the Recommendation.||Working Group commits to reviewing incoming test cases on a continuing basis if needed, including after corresponding specification becomes Recommendation.|
|6.||Working Group commits to reviewing incoming test cases. Working Group will attempt to generate a complete catalog of test cases before corresponding specification becomes Recommendation.|
|7.||Working Group commits to reviewing incoming test cases. Working Group insists on a complete catalog and partial implementation of a test suite before corresponding specification becomes Recommendation.|
|8.||Working Group insists on a test suite, not necessarily complete and thorough, and provides an test suite and/or tools, before specification in question goes to Recommendation.|
|9.||Working Group commits to writing test cases themselves and delivers a test harness and suite at some point.|
|10.||Working Group insists on a complete test suite before corresponding standard becomes Recommendation.|
This 10-point enumeration is derived from a proposal to the QA mail list, after the 4/2001 QA Workshop.
The further down the scale (i.e., the higher the number) that the Working Group commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.
Checkpoint 1.2. In the Working Group charter, specify completion and publication of test materials to be a criterion for CR-exit and PR-entrance. [Priority 2]
A de facto convention of "CR-exit" seems to be a common process goal amongst a number the existing test suite efforts (e.g., SVG), UAAG), ...) This makes sense, as it is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:
"Advancement of a technical report to Candidate Recommendation is an explicit call for implementation experience to those outside of the related Working Groups or the W3C itself." (see Process Document, section 5.2, about "Candidate Recommendation".)
"Preferably, the Working Group should be able to demonstrate two interoperable implementations of each feature." (see Process Document, section 5.2.4, about "Entrance Criteria".)
Checkpoint 1.3. In the Working Group charter, address QA staffing commitments. [Priority 2]
There will be at least some staffing required from the Working Group, for any of the acceptable options presented so far. Depending upon the general intent and plan, and how the test suite will be built, the commitment can range from minimal to significant. See "Examples & Techniques".
Guideline 2. In the Working Group charter, address where and how conformance test materials will be produced.
There are several possibilities for producing conformance test materials:
Checkpoint 2.1. In the Working Group charter, specify the WG's level of involvement in the ongoing evolution of test materials. [Priority 1]
There are good reasons why external parties should have a strong participation during the building of the materials. But even if the test suite is being developed outside of W3C, the Working Group needs to ensure it is completed according to the level to which the Working Group committed in its charter.
Checkpoint 2.2. Ensure a secure and reliable repository location for future test materials. [Priority 2]
This doesn't necessarily mean that test suite/tools should physically reside in the W3C, although that is a recommended way of meeting the checkpoint, because:
If the Working Group ceases to operate, then other responsible for ongoing interpretation and maintenance (errata) of the standard should be designated to ensure continued availability of the test suite.
Guideline 3. Ensure that the goals and scope of QA development are clear
Common frameworks for W3C QA activities are obviously shaped by the high-level goals for development and deployment of test materials. Before further developing the framework guidelines, it is useful to state some of these high-level goals.
The over-arching goal is stated in the QA Activity Statement:
Specific goals of test materials development include:
The goals of QA work within Working Group do not encompass:
Checkpoint 3.1. Ensure that test materials are documented, self contained and reusable by accredited third parties for testing purposes. [Priority 2]
While certification may have a positive quality assurance role in standards-based Web environments, W3C does not presently intend to initiate or offer any certification services. Nevertheless, developed test materials can be used by external parties including certification services. Factors of rigor, objectivity, and defensibility are key determinants of usability for certification, and these same properties are positive assets of any test materials.
Checkpoint 3.2. Specify a policy for branding materials if applicable. [Priority 3]
Certification aside, some W3C activities support "branding" -- use of a conformance icon to indicate some level of standard compliance. Examples: HTML validation, CSS validation, WCAG compliance. While these content branding schemes are relatively non-controversial, there are nevertheless some serious issues which should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.
Checkpoint 3.3. Provide disclaimer regarding use of the test materials for compliance verification. [Priority 2]
Although tests suites may be used for compliance verification, users must be aware that:
The last item requires some elaboration. Depending on the nature of the test, it may actually be sufficient to judge non-compliance of the subject. Based on taxonomy the following is valid:
Guideline 4. Allocate staff for the QA process.
Before starting test materials development, a Working Group has to appoint several key participants for its QA process.
Checkpoint 4.1. Appoint a test QA moderator. [Priority 2]Depending on the plan for the origin of the test suite a Working Group can appoint a test moderator from
When a test suite is developed outside of the Working Group, a moderator often already exists in the external group that is the source of the test suite. It is recommended to invite the moderator to become a Working Group member, since he/she needs access to all Working Group materials and resources. See also section on transferring test suite from outside W3C.
Checkpoint 4.2. Designate a task force or another entity to maintain the test suite, as well as maintenance procedures. [Priority 2]
It is recommended that the Working Group's errata task force be involved in test suite maintenance, since this will help to keep test suite and specification synchronized. The maintenance group shall establish a tracking system to track issues and changes to the test suite.
Guideline 5. Synchronize specification and test materials development
There are several strong benefits from starting synchronizing of specification and test materials developments as early as possible:
Checkpoint 5.1. Introduce intermediate milestones for test materials completion. [Priority 3]
These milestones can be naturally bound to Working Draft and/or Candidate Recommendation publications.
Checkpoint 5.2. Synchronize the latest published version of the specification and the latest published version of the test materials. [Priority 3]
Checkpoint 5.3. Support specification versioning/errata in the test materials. [Priority 2]
The structure of the test materials should allow versioning/errata to enable verification of compliance with different versions of the specification.
Guideline 6. Produce QA process document.
A Working Group QA process document describes how the QA-related processes are set up within the Working Group. Examples of such a document can be found at DOM TS process, XML Schema TS process, XML Protocol TS process, etc. It is a required part of the QA process in a Working Group, and the following requirements apply to it.:
Checkpoint 6.1. Review Test Materials Guidelines before writing the QA Process document and starting test materials development. [Priority 2]
Taking into account the nature of the future standard (Recommendation), before starting test suite development, the test suite implementer (Working Group or 3rd party) should look at Test Materials Guidelines and decide what approach to take for test materials organization, test criteria, etc.
Checkpoint 6.2. In the QA Process document, define a contribution process. [Priority 2]
The process document must describe where to submit test materials and whom to notify (e.g., moderator) of a submission. The contribution process SHOULD describe the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc.
Checkpoint 6.3. In the QA Process document, define the licenses applicable to submitted and published test materials. [Priority 2]
If test materials are produced within the W3C, the Working Group can choose either the W3C Document or the W3C Software license. If applicable, W3C Document license is recommended. For publishing test materials acquired from an external group, other licenses may be applicable (see transferring test suite).
Checkpoint 6.4. In the QA Process document, define a means for communication regarding test suites. [Priority 2]
Working Group needs to choose mailing list to send all comments and announcements to.
Checkpoint 6.5. In the QA Process document, define review procedures for submitted test materials. [Priority 2]
The following is an example that meets the requirements:
Checkpoint 6.6. In the QA Process document, describe the way test materials will be published and point to the corresponding web page. [Priority 2]
If the test materials are to be published on the W3C site, it is recommended to locate them within the corresponding activity domain.
Checkpoint 6.7. In the QA Process document, describe how vendors can publish test results for their products, if applicable. [Priority 3]
It is recommended to use one of the practices for publishing test materials, described in Test Materials Guidelines. The Working Group can encourage vendors to publish results for their implementations, such publication ideally including or describing a test harness that would allow anyone to reproduce the results.
Guideline 7. Plan the transfer of test materials to W3C if needed
As stated before in the notes to the checkpoint for test materials repository, it is recommended that test materials reside in the W3C. If test development was done outside by some external entity or group (EG), and the Working Group (WG) together with EG decided to move test materials (TM) to W3C, the following checkpoints define the requirements for the Working Group.:
Checkpoint 7.1. If transfer of the test materials to W3C is planned, perform assessment of their quality. [Priority 2]
Checkpoint 7.2. If transfer of the test materials to W3C is planned, identify staff resources to meet requirements. [Priority 2]
Checkpoint 7.3. If transfer of the test materials to W3C is planned, resolve IPR questions and reach agreement with external party that produced test materials. [Priority 1]
Checkpoint 7.4. If transfer of the test materials to W3C is planned, update QA commitments in the Working Group charter if necessary. [Priority 2]
The following procedures meets these requirements.
Legend for the procedure::
Before the transfer, WG with the help of QAWG:
After transfer, initial process setup:
First W3C public release of new TM:
After the first public release, the TM enter maintenance phase which is described below.
Guideline 8. Plan for test materials maintenance
Once test materials development is initiated, its maintenance requires an equally significant amount of resources. Resources allocation is needed as prescribed by the maintenance checkpoint above.
Checkpoint 8.1. Maintain contribution and review procedures throughout test materials' and standard's entire life cycles. [Priority 3]
Checkpoint 8.2. In the Working Group's QA process document, specify a procedure for updates of the test materials to track new errata/specification versions. [Priority 2]
The following procedure is an example that meets the requirements:
Checkpoint 8.3. In the Working Group's QA process document, identify the communication channel and procedure for appeals of tests validity. [Priority 2]
The following procedure is an example that meets the requirements:
[Active issue] If there is still active work on the test suite even if the Working Group is not re-chartered to handle the work, then it is up to the Director to determine how this work should be done. Some options include:
The QA Activity anticipates different relationships to the various Working Groups. On the one hand, the QA-specific needs of the Working Groups will vary depending upon their resources and experience, as well as their stage in the Recommendation track. On the other hand, the W3C processes may evolve so that QA-specific process requirements are tied to milestones in the Recommendation track. This section describes relationships that may exist between a Working Groups and the QA Working Group, as well as procedures that the Working Group and QA Working Group should follow. It is more informative than normative, and does not contain any guidelines and checkpoints.
Potential relationships between Working Groups and the QA Activity include:
Key determinants of the Working Group/QA relationship will include level of available QA Activity resources, and the evolution of W3C policy on QA requirements.
In an assistive role, the QA Activity provides consultation to Working Groups in the planning and building or acquiring of test materials. This should be the baseline relationship, applicable to all Working Groups. Consultation may be provided in formal and informal ways. Informal questions MAY be sent to email@example.com. In order to make assistance quick and accurate the following process is defined for formal requests:
This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.
This section defines three levels of conformance to this specification:
A Working Group conforms to the "QA Framework: Process & Operational Guidelines" at Level X (A, AA, or AAA) if the Working Group meets at least all Conformance Level X requirements.
To make an assertion about conformance to this document, specify:
"This QA processes and operations of this Working Group conform to W3C's 'QA Framework: Process & Operational Guidelines', available at http://www.w3.org/TR/2002/qaframe-ops/, Level AA."
The following QA Working Group and Interest Group participants have contributed significantly to the content of this document: