W3C

QA Framework: Operational Guidelines

W3C Working Draft 10 February 2003

This version:
http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/
Latest version:
http://www.w3.org/TR/qaframe-ops/
Previous version:
http://www.w3.org/TR/2002/WD-qaframe-ops-20021108/
Editors:
Lofton Henderson (lofton@rockynet.com)
Dominique Hazaël-Massieux (dom@w3.org)
Kirill Gavrylyuk (kirillg@microsoft.com)
Dimitris Dimitriadis (dimitris@ontologicon.com)
Lynne Rosenthal (lsr@nist.gov)
Contributors:
See Acknowledgments.

Abstract

This document defines a common Operational Framework for building conformance test materials for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Guidelines.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This Working Draft is published for Last Call review. This version incorporates the closure of all open QAWG (QA Working Group) issues about this specification, as well as editorial improvements to more clearly explain the rationale behind a number of the checkpoints. The Previous Version (3rd published WD) represented convergence on a stable set of the individual checkpoints and their priorities, and was structured so that its conformance requirements (fulfillment criteria) were clearly isolated and identified. For details, please see Change history.

This version supersedes all previous drafts. Future progression of this document beyond Working Draft is planned, but the final status has not been determined at this time. See QAWG issue #18 and issue #71. It is anticipated that this specification will eventually advance to Candidate Recommendation (CR), after successful discussion and resolution of any and all issues that arise during Last Call review. The QAWG has discussed criteria for finishing CR phase and entrance to Proposed Recommendation ( PR). The agreed criteria are: for each Priority 1 and Priority 2 checkpoint, two example implementations that successfully fulfill the checkpoint.

This part of the Framework document family has a companion QA Framework: Operational Examples & Techniques. That informative companion document is currently being progressed as a W3C Note. At least until this guidelines document stabilizes, that Examples & Techniques companion will be maintained and frequently updated in QA Working Group Web space (as opposed to /TR/). Although that document is not the principal subject of this review, the QAWG welcomes feedback on it as well.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

Please use the provided form to make your comments. If for some reason you are unable to use the form, you may email comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that comments that you make will be publicly archived and available, do not send information you would not want to see distributed, such as private data. Please submit your comments by 14 March 2003, when Last Call review ends.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.

Table of contents

  1. Introduction
    1. Scope and goals
    2. Class of product and audience
    3. Motivation and expected benefits
    4. Relationship to other specifications
    5. Understanding and using this document
    6. Checkpoint priorities
    7. Terminology
  2. Guidelines
    1. Integrate Quality Assurance into Working Group activities.
    2. Define resources for Working Group QA activities.
    3. Synchronize QA activities with the specification milestones.
    4. Define the QA process.
    5. Plan test materials development.
    6. Plan test materials publication.
    7. Plan the transfer of test materials to W3C if needed.
    8. Plan for test materials maintenance.
  3. WG relationship to QA Activity
  4. Conformance
    1. Normative sections
    2. Extensibility
    3. Conformance requirements & test assertions
    4. Conformance definition
    5. Conformance disclaimer
  5. Acknowledgments
  6. References
    1. Normative references
    2. Informative references
  7. Change history

Two separate appendices to this document [OPS-CHECKLIST] and [OPS-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by their priorities, for convenient reference. The latter is an Implementation Conformance Statement (ICS) pro-forma for this specification.


1. Introduction

1.1. Scope and goals

The scope of this specification is a set of verifiable requirements for the process and operational aspects of the quality practices of W3C Working Groups. The primary goal is to help the W3C Working Groups (WGs) with the planning, development, deployment, and maintenance of conformance test materials (TM).

In this guidelines document, the term "conformance test materials" refers conformance test suites, validation tools, conformance checklists, and any other materials that are used to check or indicate conformance.

1.2. Class of product and audience

The class of product that is the target of the requirements in this specification is: all process and operational aspects of Working Group quality practices. This includes the WG's planning and commitment for QA, staffing to meet commitments, organization and logistics of the quality practices, publication and maintenance, and synchronization of quality processes with other WG activities.

The intended audience for these guidelines is all W3C Working Group members, as well as the actual developers of conformance materials for W3C specifications.

While it is optimal that sound quality practices are integrated into WG activities from the very beginning, these guidelines are intended for newly chartered and existing Working Groups alike. Working Groups who may already be doing some of these activities should review the document and incorporate the principles and guidelines into their quality practices as much as possible.

1.3. Motivation and expected benefits

As the complexity of W3C specifications and their interdependencies increase, quality assurance becomes even more important to ensuring their acceptance and deployment in the market. There has been a growing awareness and interest in conformance and quality. In approving and initiating the QA Activity, W3C has endorsed the principle that in order for W3C Web standards to achieve full interoperability and access to all, the quality of the implementation must be given as much attention as the standards' development. The principal factor for improving the quality of implementation is early availability of conformance test materials.

Although not explicitly stated, the W3C Process Document supports the development of conformance test materials.

[...] groups may produce technical reports, review the work of other groups, develop sample code or test suites, etc." (see Process Document, section 3.)

W3C should make every effort to maintain its Recommendations (e.g., by tracking errata, providing test bed applications, helping to create test suites, etc.) (see Process Document, section 5.2.5, "Ongoing work".)

To meet these suggestions and address the implementation requirements of the Process Document, some Working Groups have included the development of conformance materials as part of their PR-entrance criteria. Examples include Scalable Vector Graphics (SVG), User Agent Accessibility Guidelines (UAAG), and Extensible Style Language - Formatting Objects (XSL-FO). This makes sense, since it is natural for test suites and implementations to develop in parallel - each is a help to the development of the other.

There is already a body of contemporary QA experience and activity amongst the Working Groups. The Matrix [MATRIX] identifies more than a score of test suites and validators, in various states of development. Moreover, many Working Groups have already established procedures, techniques and tools for developing test materials (e.g., Document Object Model - DOM). It makes sense to capitalize on what has already been done and share that with those who are starting out and those who are already in the process of developing conformance materials.

Accordingly, these guidelines aim to capture the experiences, good practices, activities, and lessons learned of the Working Groups, and to present them in a comprehensive, cohesive set of documents for all to use and benefit from. By standardizing the best of current practice, these guidelines should allow the WGs to reuse what works rather than having to reinvent, which in turn should facilitate and expedite the work of the WGs. Conformance with these guidelines should also promote consistency across the various Working Group quality activities and deliverables

1.4. Relationship to other specifications

This document is part of a family of QA Framework documents designed to help the WGs improve all aspects of their quality practices by solidifying and extending current quality practices found within the W3C. The QA Framework documents are:

The QA Framework documents are interrelated and complement each other. For example, there is a close relationship between the processes for dealing with versions and errata, and the test framework architecture for handling multiple versions and errata levels. Hence there is interrelationship between this document and the Test Guidelines. The reader is strongly encouraged to be familiar with the other documents in the family.

1.5. Understanding and using this document

The guidelines in this document start with those that are applicable as early as the formation of a Working Group (e.g., charter considerations), and continue through the various process and operational activities necessary in planning, developing, deploying and maintaining conformance materials.

This specification applies the WAI (Web Accessibility Initiative) guidelines model to Working Group quality practices and the development of conformance test materials. See, for example, Web Content Accessibility Guidelines. Each guideline includes:

The checkpoint definitions in each guideline define the processes and operations that need to be implemented in order to accomplish the guideline. Each checkpoint definition includes:

Each checkpoint is intended to be specific enough so that someone can implement the checkpoint as well as verify that the checkpoint has been satisfied. A checkpoint will contain at least one, and may contain multiple individual requirements, that use RFC2119 normative keywords. See the Conformance section for further discussion of requirements and test assertions.

Two separate appendices to this document [OPS-CHECKLIST] and [OPS-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by their priorities, for convenient reference. The latter is an Implementation Conformance Statement ( ICS) pro-forma for this specification.

1.6. Checkpoint priorities

Some checkpoints are more critical than others for the timely production of high-quality, highly usable test materials. Therefore each checkpoint has a priority level based on the checkpoint's impact on the quality and timing of the test materials produced by a Working Group.

[Priority 1]
Critical/essential. These checkpoints are considered to be basic requirements for ensuring that test materials are usable, and are produced in time to ensure the quality of the standard and its implementations. Satisfying these checkpoints is a basic requirement to ensure quality and interoperability of the standard.
[Priority 2]
Important/desirable. Satisfying these checkpoints, in addition to the priority 1 checkpoints, should significantly improve the usability and timeliness of the test materials, as well as the quality of the standard and its implementations.
[Priority 3]
Useful/beneficial. Satisfying these checkpoints, on top of all the others, will further improve the quality, usability, and timeliness of the test materials and the standard itself.

1.7. Terminology

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they are all upper case. Occurrences of these words in lower case comprise normal prose usage, with no normative implications.

Unusual terms in this framework document are defined when first used, and most generally useful QA-specific terms will eventually be in the QA Glossary [QA-GLOSSARY].

2. Guidelines

Guideline 1. Integrate Quality Assurance into Working Group activities.

In approving and initiating the QA Activity, W3C has endorsed the principle that in order for W3C Web standards to achieve full interoperability and access to all, the quality of implementations must be given as much attention as specifications' development.

The quality of specifications and the early availability of conformance test materials (TM) are significant contributors to the quality of implementations. Early adoption of a suitable commitment level for these two components of a WG's quality practices -- followed by identification of specific QA deliverables, milestones, and dependencies with other WG deliverables -- is prerequisite to detailed planning, execution, and timely achievement of at least a minimal quality level across all of a Working Group's deliverables.

New Working Groups -- i.e., those that are writing and submitting their Charters for approval -- are required in this guideline's checkpoints to make the specified commitments in their charters. Working Groups that are renewing their charters are considered the same as new WGs. Existing Working Groups could satisfy the checkpoints by amendment to their charter, or in other ways, the latter to be specified in QA Framework: Operational Examples & Techniques [OPS-EXTECH].

Checkpoints:

Checkpoint 1.1. Commit to at least "QA level three". [Priority 1]

Conformance requirements: a new or rechartering Working Group MUST commit, in its Charter, to at least QA level three. An existing Working Group MUST document a consensus commitment to at least QA level three in one of various other ways.

Discussion. Detailed planning and specification of test materials is inappropriate in the charter because it inflexibly binds the Working Group to test materials implementation specifics for which flexibility may be needed. There are numerous possible levels of commitment that a Working Group could make based on anticipated Working Group resources, existence of outside efforts, and the Working Group's current stage in the process [REC-TRACK].

Based on the level of testability of the specification and the extent of test materials, a range of possible commitments is enumerated.

Level testability of specification extent of test materials
1. Summary. Working Group (WG) plans regular specification reviews for testability, following Specification Guidelines (SpecGL)

Conformance requirements: the WG MUST commit to documenting a SpecGL conformance review at each of the milestones: (first) Last Call ( LC), (first) Candidate Recommendation ( CR), Proposed Recommendation ( PR), and Recommendation. (Note. Completing a SpecGL Implemetation Conformance Statement ( ICS) is one way to do this.)

2. Summary. In addition to the previous level, Working Group (WG) provides a set of test assertions, not necessarily complete, before beginning development of a test materials.

Conformance requirements: the WG MUST commit that a set of test assertions for each of the WG's specifications will be publicly available at the time that test materials development commences, and it SHOULD be located on the WG "/Test/" page (see Checkpoint 4.4).

Summary. Working Group ( WG) ensures that test materials for the specification, not necessarily complete and thorough, will exist by the time the specification becomes Recommendation.

Conformance requirements: the WG MUST commit that each of the WG's specifications will have associated test materials before it becomes Recommendation.

3. Summary. In addition to the previous level, a Working Group ( WG) will provide numerous use cases and examples in the body of the specification.

Conformance requirements: the WG MUST commit that specification(s) will at least satisfy the relevant checkpoints of SpecGL Guideline 1.

Summary. In addition to the previous level, Working Group ( WG) ensures that any test materials provided by the WG are reviewed, and the review process meets the criteria established in this specification.

Conformance requirements: the WG MUST commit that test materials will meet the review requirements of these guidelines. (Note: This requires at least checkpoint 7.1 for externally written test materials, and the application of procedures equivalent to 5.4 and 8.1 for internal.)

4. Same as previous. Summary. In addition to the previous level, Working Group ( WG) will establish and maintain a test cases contribution and review process before the specification becomes Recommendation.

Conformance requirements: the WG MUST commit to establish and maintain a contribution and review process (see checkpoints 5.4 and 8.1) before Recommendation.

5. Summary. In addition to the previous level, Working Group ( WG) will derive the list of test assertions as an addendum to specification by the time it becomes Recommendation.

Conformance requirements: the WG MUST commit that specifications will satisfy the test-assertion requirements of SpecGL (see checkpoint 14.1) before the specifications become Recommendation.

Same as previous.
6. Same as previous. Summary. In addition to the previous level, a Working Group ( WG) insists on a complete test suite before a specification becomes Recommendation.

Conformance requirements: the WG MUST commit to a complete test materials before Recommendation, where complete is defined as: at least one test case for every identifiable conformance requirement of the specification.

7. Same as previous. Summary. In addition to the previous level, Working Group ( WG) plans to maintain the test suite even after specification becomes Recommendation. Working Group plans to support versions and errata in the test suite and review appeals.

Conformance requirements: the WG MUST commit to both maintenance and to version/errata support, for test materials of each specification after it becomes Recommedation.

This seven-point enumeration is derived from the proposal to the QA mail list, after the 4/2001 QA Workshop.

The further down the scale (i.e., the higher the number) that the Working Group commits, the better it is for achieving W3C's ultimate goals of interoperability and accessibility.

For existing working groups that make the QA commitment after chartering, various ways to do it are discussed in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 1.2. Commit to at least "QA level five". [Priority 2]

Conformance requirements: a new or rechartering Working Group MUST commit, in its Charter, to at least QA level five. An existing Working Group MUST document a consensus commitment to at least QA level five in one of various other ways.

Discussion. This checkpoint advances the level of QA commitment required by the previous checkpoint and therefore supersedes it. Satisfying this level of requirements for the specification and the test materials significantly increases confidence in the interoperability of the standard.

For existing working groups that make the QA commitment after chartering, various ways to do it are discussed in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 1.3. Commit to at least "QA level seven". [Priority 3]

Conformance requirements: a new or rechartering Working Group MUST commit, in its Charter, to at least QA level seven. An existing Working Group MUST document a consensus commitment to at least QA level seven in one of various other ways.

Discussion. This checkpoint advances the level of QA commitment required by the previous checkpoint and therefore supersedes it. Satisfying this level of requirements for the specification and the test materials further increases confidence in the interoperability of the standard.

For existing working groups that make the QA commitment after chartering, various ways to do it are discussed in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 1.4. Enumerate QA deliverables and expected milestones. [Priority 1]

Conformance requirements: a new or rechartering Working Group MUST document the QA deliverables and milestones its Charter. An existing Working Group MUST document QA deliverables and milestones in some consensus record, in one of various other ways.

Discussion. The W3C Process Document requires for Charters, that deliverables be identified with milestones. It is vital that the milestones for QA deliverables are synchronized and even serve as criteria for WG technical deliverables (specifications).

Examples of QA deliverables include sets of use cases, test suites (produced or acquired), validators, test harnesses.

Examples & Techniques for checkpoint.

Checkpoint 1.5. Associate QA criteria with WG milestones. [Priority 2]

Conformance requirements: a new or rechartering Working Group MUST, in its charter, specify the QA criteria required for the attainment of major WG milestones. An existing Working Group MUST define the association in some consensus record, in one of various other ways.

Discussion. A de facto convention of test suite completion before entrance into PR phase (and by implication, completion of CR phase, if there was a CR phase) seems to be a common process goal amongst a number of the existing test suite efforts. For example, SVG, UAAG, and others have defined this criterion. It is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. And, the Process Document does address implementation requirement:

Advancement of a technical report to Candidate Recommendation is an explicit call for implementation experience to those outside of the related Working Groups or the W3C itself." (see Process Document, section 5.2, about "Candidate Recommendation.)

Preferably, the Working Group should be able to demonstrate two interoperable implementations of each feature. (see Process Document, section 5.2.4, about "Entrance Criteria".)

Examples & Techniques for checkpoint.

Guideline 2. Define resources for Working Group QA activities.

It is highly beneficial to the success of the WG's QA work that it address QA resource requirements from the beginning of the Working Group formation. Starting from the Working Group Charter and later in the Call For Participation special attention is required to the staffing and other resource requirements for successful QA work.

The sooner the resources are defined and committed, the better. Therefore new Working Groups -- i.e., those that are writing and submitting their Charters for approval -- are required to have the resource commitments in their charters. Working Groups whose charters are being renewed are considered the same as new WGs. While "sooner is better", existing Working Groups can still satisfy the checkpoints, in other ways to be specified in QA Framework: Examples & Techniques [OPS-EXTECH].

Checkpoints:

Checkpoint 2.1. Address where and how conformance test materials will be produced. [Priority 1]

Conformance requirements: a new or rechartering Working Group MUST, in its charter, address and discuss who will produce the test materials and how. An existing Working Group MUST define the who & how in some consensus record, in one of various other ways.

Rationale. Determining whether the WG plans to produce the test materials themselves or get them from other sources is a first step in the overall planning for QA deliverables. Identifying the scenario for producing these deliverables is a prerequisite to estimating the resources and effort needed for the WG's level of QA commitment.

Discussion. TM-production scenarios range from all intra- WG effort concurrent with other WG deliverables, to importing completed materials from an external group, and hybrid internal-external efforts. Further discussion and example scenarios are found in Operational Examples & Techniques.

Examples & Techniques for checkpoint.

Checkpoint 2.2. Address QA staffing commitments. [Priority 1]

Conformance requirements: a new or rechartering Working Group MUST, in its charter, allocate WG staffing resources commensurate with the QA level and where-and-how plan. An existing Working Group MUST define the allocation in some consensus record, in one of various other ways.

Rationale. For any acceptable level of QA commitment, there will be some staffing requirement. If the WG addresses the staffing level at the same time that it defines the commitment level and general scenario, then that both helps to confirm that the overall plans are realistic, as well as facilitates that the commitments will be successfully realized.

Discussion. Depending upon the general intent and plan, and how the test materials will be built, the commitment can range from minimal to significant. At the least, a QA point-of-contact will be needed by the WG.

Examples & Techniques for checkpoint.

Checkpoint 2.3. Request allocation of QA resources to the Working Group. [Priority 1]

Conformance requirements: a new or rechartering Working Group MUST, in its Call for Participation, request that participating members allocate some staffing resources specifically for QA work. An existing Working Group MAY make external appeal for QA-specific resources in one of various other ways.

Rationale. The WG needs explicit indication of resources dedicated for QA, in order to assess Working Group capabilities against deliverables and milestones. Additionally, if the WG emphasizes in its participation calls that QA specialists are needed and welcome, that should facilitate getting competent and interested people to work on the WG's QA commitments.

Discussion. Once the Charter is prepared, the Director sends a Call for Participation to the Advisory Committee. At this point AC Representatives are asked to provide information about amount and type of resources their organization plans to allocate for the particular Working Group.

Examples & Techniques for checkpoint.

Guideline 3. Synchronize QA activities with the specification milestones.

The benefits of starting synchronization of the specification and test materials development as early as possible include:

Checkpoints:

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]

Conformance requirements: the Working Group MUST publish QA deliverables concurrently with each Working Group specification publication milestone.

Rationale. Because each version of the specification -- WDs, CR(s), PR, etc -- is a changed document, therefore all important dependencies such as test materials need to be updated concurrently.

Discussion. This is a specialization of the checkpoint requirement above -- associate QA criteria with each Working Group milestone -- to the specific WG milestones of specification stages on the Recommendation track. The natural QA criterion for such a specification publication milestone is the publication of the updated test materials or related QA deliverables.

Examples of QA deliverables might range from a TS (test suite) production schedule in early WDs, to TS design document in later WDs, to a first public TS release at CR.

Examples & Techniques for checkpoint.

Checkpoint 3.2. Support specification versioning/errata in the QA deliverables. [Priority 3]

Conformance requirements: the Working Group MUST ensure that the final published test materials support specification versioning/errata, and SHOULD include specification versioning/errata considerations in any other QA deliverables such as intermediate planning documents.

Rationale. There will be products that support different versions and/or different errata levels of the specification after it becomes Recommendation. Building specification versioning/errata support into the test materials' infrastructure will allow users to select and apply the appropriate test materials for a given product.

Discussion. This checkpoint specifically extends the previous one to cover specification milestones after Recommendation status. Whether or not a Working Group plans to maintain the test suite after the specification becomes Recommendation (see the Table of WG Commitment Levels), it has to ensure support of versioning/errata in the structure of the test materials.

Note. This checkpoint concerns support for versioning/errata in the infrastructure or framework of test materials. The related checkpoint 8.2 is about defining maintenance procedures for test materials to track specification versions or errata levels.

Examples & Techniques for checkpoint.

Guideline 4. Define the QA process.

A Working Group QA process encompasses all aspects of QA life within the Working Group, including:

Documented examples of the QA process can be found at DOM TS process, at XML Schema TS process, and at XML Protocol TS process.

Checkpoints:

Checkpoint 4.1. Appoint a QA moderator. [Priority 1]

Conformance requirements: the Working Group MUST identify a person to manage the WG's quality practices.

Rationale. Having a single person assume primary responsibility for the WG's quality practices will help ensure that its QA commitments and milestones are met efficiently and on schedule, and a single identified QA point-of-contact will optimize external QA-related communications.

Discussion. The QA moderator is the overall manager of all of the QA activities in the Working Group, and (by default) principal point-of-contact. At even the lowest acceptable levels of QA commitment, there are sufficient requirements that at least part-time of one person will be needed. Depending on the anticipated origin of the test materials, there are several possibilities for the QA moderator, ranging from appointing a QA member to inviting someone from an external organization.

Examples & Techniques for checkpoint.

Checkpoint 4.2. Appoint a QA task force. [Priority 2]

Conformance requirements: the Working Group MUST identify a QA task force of size commensurate with the QA commitment level and the agreed QA deliverables.

Rationale. At higher levels of QA commitment and deliverables, the resource requirements may exceed (part-time of) one person. Identifying and assigning any additional needed QA staff resources is a prerequisite to realistic schedules and QA delivery milestones.

Discussion. Depending on the level of involvement of a Working Group in the test materials development, a task force assisting the QA moderator can take responsibility for a QA framework development, test materials development, review of contributions, and maintenance. In practice, the task force can range from a subset of dedicated individuals, to some fraction of the time of each WG member.

Examples & Techniques for checkpoint.

Checkpoint 4.3. Produce the QA Process Document. [Priority 1]

Conformance requirements: the Working Group MUST produce a QA Process Document that:

Rationale. A QA Process Document records, in one place, key information about the WG's quality-related logistical and communications setup, contribution and licensing policies, publication and maintenance plans, and other critical process and operational details. The production of this document ensures that these key aspects of the WG's quality practices are explicitly addressed and publicly recorded.

Discussion. This checkpoint depends on other checkpoints that contain specific requirements for topics and items in the QA Process Document (QAPD). To summarize, the QAPD must address:

Satisfaction of this checkpoint requires satisfaction of these related checkpoints: 4.4, 4.5, 4.6, 5.2, 5.4, 5.3, 6.2, 6.3, 6.4, 6.5, 8.2, 8.3.

Examples & Techniques for checkpoint.

Checkpoint 4.4. Specify means for QA-related communication. [Priority 2]

Conformance requirements: the WG's QA Process Document MUST specify at least one public archived mailing list for QA announcements and submission of public QA comments, and MUST establish a publicly readable "Test" Web page.

Discussion. In practice, the mailing list may be a dedicated "test" mailing list, or something like an existing Interest Group mail list could be used for the purpose. A common convention for the QA-related Web page is recommended: name it /Test/ directly under the WG's home page, e.g., http://www.w3.org/Graphics/SVG/Test/.

Examples & Techniques for checkpoint.

Checkpoint 4.5. Define the QA framework for test materials development. [Priority 1]

Conformance requirements: the WG's QA Process Document MUST define the framework that will be used for test materials development.

Rationale. The framework is effectively the top-level design for test materials production. As in software development projects, a sound top-level design enables the efficient production of quality deliverables that work as intended.

Discussion. The QA framework describes how to develop, document and use the tests.

Examples & Techniques for checkpoint.

Checkpoint 4.6. Define branding policy details. [Priority 3]

Conformance requirements: the WG MUST document in its QA Process Document the branding policy details, if branding is to be supported.

Rationale. Branding policy (like certification) is potentially one of the most contentious topics that a WG might face. If the WG opts to support branding, then defining the branding policy up front ensures that the WG undertakes test materials development (or acquisition) with a common understanding of their ultimate deployment and use. For end users, early definition of branding policy clarifies the WG intentions for a highly visible aspect of test materials' use.

Discussion. Some W3C activities support "branding". Branding refers to the use of a conformance icon to indicate some level of standard conformance. Examples: HTML validation, CSS validation, WCAG conformance. While these content branding schemes are relatively non-controversial, there are nevertheless issues that should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

This checkpoint is not applicable if the WG does not support branding.

Examples & Techniques for checkpoint.

Guideline 5. Plan test materials development.

As described earlier, a Working Group may have different levels of involvement in test materials development. Nevertheless at any level a Working Group needs to have clear understanding of what QA framework it will use and how to insure the quality and usability of the test materials themselves. As a part of the WG's QA process, before starting a test materials development, it is recommended the test materials implementer (Working Group or 3rd party) looks at Test Guidelines and decide what approach to take for test materials organization, test criteria, etc.

Checkpoints:

Checkpoint 5.1. Ensure test materials are documented and usable for their intended purposes. [Priority 1]

Conformance requirements: the WG MUST ensure that the test framework contains user documentation that instructs the use of the test materials for the full range of their intended purposes.

Discussion. Documentation should be thorough enough that external entities unknown to the developers of the test materials -- such as external certification services -- can effectively and fully use the test materials.

Examples & Techniques for checkpoint.

Checkpoint 5.2. Define a contribution process. [Priority 2]

Conformance requirements: the WG MUST describe in its QA Process Document where, how, by whom, and to whom test materials submissions are to be made.

Rationale. Clear and simple directions for where and to whom to submit materials will encourage outside contributions. Explicit instructions on formats and other technical details will enable the WG to build processes and procedures for the efficient processing of contributions, whether the contributions are internal or external.

Discussion. The process document must describe where to submit test materials and whom to notify (e.g., moderator) of a submission. The contribution process describes the format of contributed material, it may contain validation harness, utilities that facilitates tests creation, templates, etc. A well defined submission process is applicable to both internal and external contributors of test materials.

Examples & Techniques for checkpoint.

Checkpoint 5.3. Address license terms for submitted test materials. [Priority 1]

Conformance requirements: in its QA Process Document the WG MUST define a submission license policy applicable to test materials submitted to the WG by external parties, and the submission license policy MUST include at least an outline of terms, conditions, constraints, and principles that will govern acceptable submissions to the WG.

Rationale. Defining submission license policies in advance will help clarify the WG's expectations to prospective TM submitters, and will facilitate the efficient negotiation of any needed custom submission licenses with submitters.

Discussion. Unless exempted by custom submission terms with W3C Legal staff approval, a WG's submission license policies will necessarily conform to standard W3C policies for submitted materials, and specifically those procedures and terms defined in Contribution of Software or Test Materials to W3C [CONTRIB].

Note also that any submission policy will inherit some constraints derived from other checkpoints in this guidelines specification, such as the requirements of publication licenses, and the requirements for free access to test materials. A Working Group may in fact decide to publish a prototype submission license agreement that embodies terms and conditions acceptable to the WG. In cases where a standard submission license is not acceptable, the WG will have to negotiate licenses with prospective contributors for their specific needs, under the principles defined.

Documented examples of TM submission licenses can be seen in the XML Schema submission license, and in the XML Protocol submission license.

Examples & Techniques for checkpoint.

Checkpoint 5.4. Define review procedures for submitted test materials. [Priority 2]

Conformance requirements: the WG MUST define in its QA Process Document a procedure for reviewing test materials contributions, and the procedure MUST minimally address criteria for accuracy, scope, and clarity of the tests.

Rationale. A well-defined and objective review procedure expedites the processing of contributions, and facilitates quick and impartial decisions about their disposition.

Discussion. These procedures are a follow-on and complement to the contribution process that WGs must define. They should at least include criteria for acceptance or rejection of reviewed tests.

Examples & Techniques for checkpoint.

Guideline 6. Plan test materials publication.

Once the test suite development is in progress, a Working Group needs to publish the test suite drafts and releases, as part of its QA processes.

Checkpoints

Checkpoint 6.1. Ensure a suitable repository location for test materials. [Priority 1]

Conformance requirements: the Working Group MUST ensure that test materials have repository locations that are secure, reliable, and freely accessible.

Discussion. Once the materials have reached some advanced milestone of maturity and development (e.g., operationally usable), W3C needs to assure that:

This checkpoint does not necessarily require that test suite/tools should physically reside in the W3C. Nevertheless, having the repository at W3C is a recommended way of meeting the checkpoint, because:

  1. At the point that test materials become operationally deployed, then challenges to the correctness of both the test materials and specifications normally increase, and the Working Group is the best venue for initial processing and adjudication of such queries against both.
  2. Further to #1, it is more likely that technical report (Recommendation) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, the Working Group.

It is implicit in the "secure" criterion of this checkpoint that, if the Working Group ceases to operate, then the repository arrangements must provide for the continued availability of the test materials.

Examples & Techniques for checkpoint.

Checkpoint 6.2. Define the licenses applicable to published test materials. [Priority 1]

Conformance requirements: the Working Group MUST define in its QA Process Document the licenses that are applicable to published test materials.

Rationale. Any W3C-hosted materials must have approved license and use terms associated. Because there is no single license that is appropriate for all test materials, the WG needs consider the options, select or define a best license for its particular test materials, and clearly inform potential users.

Discussion. If test materials are produced within the W3C, the Working Group is recommended to use the W3C Document License. The Document License is preferred because:

If the Document License is not applicable or not acceptable, then the Software License or some W3C-approved custom license may be used instead. The Document License may be inapplicable, for example, in cases where the user needs to modify the test materials in order to use them. The Document License may be unacceptable, for example, in cases where the WG publishes test materials acquired from a member or an external group (see also about transferring a test suite). An example of a candidate for a W3C-approved custom Test Materials License -- which incorporates scope-of-use restrictions -- may be found in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 6.3. Describe how and where the test materials will be published. [Priority 2]

Conformance requirements: in its QA Process Document the Working Group MUST describe how test materials will be published, and point to the Web site at which they will be published.

Discussion. If the test materials are to be published on the W3C site, it is recommended to locate them within the corresponding activity domain. It is strongly recommended not to publish test materials in the TR space for the following reasons:

It is recommended to use one of the practices for publishing test materials, described in Test Guidelines.

Examples & Techniques for checkpoint.

Checkpoint 6.4. Provide a conformance verification disclaimer with the test materials. [Priority 1]

Conformance requirements: the Working Group MUST provide a prominent disclaimer about the use of the test materials for conformance verification of implementations.

Rationale. It is common to draw unwarranted conclusions about conformance to the specification from test suite results. A conformance disclaimer clarifies the relationship between test suite results and conformance.

Discussion. The central principle that a conformance disclaimer for test materials needs to clarify for users of the test materials and consumers of test results is this: passing all tests of the test materials does not guarantee or imply full conformance of an implementation to the specification.

Examples & Techniques for checkpoint.

Checkpoint 6.5. Address the publication of test results for products. [Priority 2]

Conformance requirements: in its QA Process Document the Working Group MUST encourage the publication of test results, MUST address sample scenarios of where and how such publication can be done, and MAY identify a WG-sponsored Web site for publishing collected results or a directory of results.

Discussion. Publishing test results for implementations of the WG's specifications promotes interoperability and is helpful to users and vendors alike.

Discussion. Taking responsibility to publish test results for vendors' products could be a problem for the Working Group. One way to address the problem is for the Working Group to encourage vendors to publish results for their implementations themselves. Such publication should include or describe a test harness that would allow anyone to reproduce the results. The Working Group may want to provide the web space to publish collected results.

Examples & Techniques for checkpoint.

Guideline 7. Plan the transfer of test materials to W3C if needed.

As stated before in the notes to the checkpoint for test materials repository, it is recommended that test materials reside in the W3C. If test development was done outside by some external entity or group (EG), and the Working Group ( WG) together with EG decided to move test materials (TM) to W3C, the following checkpoints define the requirements for the Working Group.

All of the checkpoints of this guideline are not applicable if the Working Group does not transfer test materials from an external entity.

Checkpoints:

Checkpoint 7.1. Perform a quality assessment of any test materials that are candidates for transfer. [Priority 2]

Conformance requirements: as a part of any test materials transfer process, the Working Group MUST perform and record an assessment of the quality of the test materials.

Discussion. This checkpoint parallels the checkpoint for the test review process. During test materials review, the Working Group must follow the criteria defined in the document that is required by that checkpoint.

This checkpoint is not applicable if the Working Group does not transfer test materials from an external entity.

Examples & Techniques for checkpoint.

Checkpoint 7.2. Identify sufficient staff resources to meet the needs of any transferred test materials. [Priority 1]

Conformance requirements: as a part of any test materials transfer process, the Working Group MUST commit staff resources commensurate with planned ongoing WG test materials' deliverables, and MUST record that commitment in some consensus WG document.

Discussion. This checkpoint parallels checkpoint 2.2 for the specific circumstance of transferring the test materials from the outside to the W3C.

This checkpoint is not applicable if the Working Group does not transfer test materials from an external entity.

Examples & Techniques for checkpoint.

Checkpoint 7.3. For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [Priority 1]

Conformance requirements: as a part of any test materials transfer process, the Working Group MUST have a documented agreement with the external entity on all IPR aspects that are applicable to the transferred materials.

Rationale. Completion of suitable IPR agreements for any transferred test materials, before the transfer is actually completed, will ensure that the WG can publish and the public can use the test test materials according terms and conditions planned and envisioned by the WG.

Discussion. This checkpoint is not applicable if the Working Group does not transfer test materials from an external entity.

Examples & Techniques for checkpoint.

Guideline 8. Plan for test materials maintenance.

The ongoing maintenance of test materials is critical to their long term integrity, and may require a significant commitment of resources. The checkpoints of this guideline address planning and resource commitment requirements to ensure that proper specification tracking happens, and that maintenance and update procedures are provided for.

If the Working Group is not re-chartered, then it is problematic where the resources will come from. In this case, it is up to the Director to determine how the work should be done. Some options include:

Checkpoints:

Checkpoint 8.1. Provide for the long-term maintenance of the contribution and review procedures. [Priority 3]

Conformance requirements: in some WG consensus document, the Working Group MUST define a plan and identify resources for the maintenance of the test materials' contribution and review procedures throughout the entire life cycle of the test materials and the Recommendation itself.

Discussion. It is implicit in the criteria of this checkpoint that, if the Working Group ceases to operate, then the WG's maintenance plan must survive and address the ongoing maintenance of the contribution and review procedures.

Examples & Techniques for checkpoint.

Checkpoint 8.2. Specify a test materials update procedure to track new specification versions/errata. [Priority 2]

Conformance requirements: in its QA Process Document the Working Group MUST specify procedures to update the test materials to track new specification versions and errata levels.

Rationale. Defining tracking procedures for specification versions/errata will help to ensure that all needed versions of test materials are available. This in turn facilitates product version tracking of specification versions, as well as enables products' users to better assess the quality and conformance of the individual product versions.

Discussion. This checkpoint is about defining maintenance procedures for test materials to track specification versions or errata levels. The related checkpoint 3.2 concerns support for versioning/errata in the infrastructure or framework of the test materials.

Examples & Techniques for checkpoint.

Checkpoint 8.3. Identify a procedure for test validity appeals. [Priority 2]

Conformance requirements: in its QA Process Document the Working Group MUST identify the communication channel for appeals of test validity and a procedure for resolving such appeals.

Rationale. A well-defined and deterministic procedure for handling challenges will expedite the resolution of challenges, and improve their defensibility and acceptance.

Discussion. Such a procedure is applicable both to external challenges once test materials have been published, as well as to internal challenges during the building of test materials.

Examples & Techniques for checkpoint.

3. WG relationship to QA Activity

The QA Activity works closely with other W3C WGs, providing assistance and expertise in helping them achieve their QA goals and deliverables. The QA Activity anticipates different relationships with the various WGs depending on the QA-specific needs of the particular Working Group. For example, the resources and experience of Working Group members as well as their stage in the Recommendation track may influence the type and level of collaboration between the Working Group and QA Activity. Potential relationships between Working Groups and the QA Activity include:

More details about the ways in which the QA Activity may interact with and assist the W3C WGs may be found in the QA Working Group Process Document [QAWGPD].

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

4.1. Normative sections

The following sections are normative in this document:

Any other section not explicitly marked as normative is assumed to be informative.

4.2. Extensibility

This specification is extensible. That is, the Working Groups MAY set up quality-related processes and operations in addition to those required for conformance to this specification. Extensions to this specification MUST NOT contradict or negate the requirements of this specification.

For each degree of conformance claimed (I.e., A, AA or AAA), it is allowable to implement more than the checkpoints required to satisfy that degree of conformance. This may be achieved by either satisfying some, but not all of the checkpoints of the next degree of conformance or by implementing additional conformance related features beyond what is specified in this document. For example, claiming to be A-conforming but also satisfying some of the Priority 2 checkpoints and some Priority 3 checkpoints.

The rationale for allowing Working Groups to define extensions to these operational guidelines is that these requirements are considered to be the minimal requirements for successful quality practices within the WGs. Doing more than the minimum is not only acceptable, but beneficial. Extensions also allow Working Groups tailor their quality practices more closely to their specific needs. The guidelines of this specification may not be sufficient to meet the needs of all WGs.

4.3. Conformance requirements & test assertions

Within each prioritized checkpoint there is at least one conformance requirement, and there may be more than one. These are prefaced by "Conformance requirements:" and highlighted in a different style. This Operational Guidelines document does not enumerate a list of test assertions. A test assertion can be derived from each "MUST" requirement in a straightforward manner.

4.4. Conformance definition

This section defines three degrees of conformance to this guidelines specification:

A checkpoint is satisfied by satisfying all of the individual conformance requirements. Failing one individual mandatory requirement means that the checkpoint is not satisfied. Mandatory requirements are those that use the conformance keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", or "SHALL NOT".

A Working Group conforms to the "QA Framework: Operational Guidelines" at degree X (A, AA, or AAA) if the Working Group meets at least all degree X conformance requirements.

An assertion of conformance to this specification -- i.e., a conformance claim -- MUST minimally specify:

The checklist for this specification ([OPS-CHECKLIST]) is the Implementation Conformance Statement ( ICS) pro-forma for this specification. Any assertion of conformance to this specification MUST link to a completed ICS.

Example of a valid conformance claim:

The QA processes and operations of this FooBar Working Group, http://www.w3.org/FooBar/, conform to W3C's 'QA Framework: Operational Guidelines', available at http://www.w3.org/TR/2002/WD-qaframe-ops-20021108/, on 23-December-2002, AA-Conforming. The Implementation Conformance Statement for this claim is available at http://www.w3.org/FooBar/Test/OpsGL-ICS.

4.5. Conformance disclaimer

The checkpoints of this specification present verifiable conformance requirements about the operational aspects of Working Group quality processes. As with any verifiable conformance requirements, users should be aware of this disclaimer:

Passing all of the requirements to achieve a given conformance degree -- A, AA, or AAA -- does not guarantee that the subject operations and processes are well-suited to or will achieve their intended purposes, nor does it guarantee the quality or suitability of test materials produced under the processes.


5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References

6.1. Normative references

PROCESS
World Wide Web Consortium Process Document, I. Jacobs, Ed., 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/.
RFC2119
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt.

6.2. Informative references

CONTRIB
Contribution of Software or Test Materials to W3C, which defines W3C-approved procedures and terms for submission of Software and Test Materials, available at http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231.
EXTERN-TA
QA activity email thread about third-party participation in test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html.
MATRIX
W3C-wide conformance activity survey covering all the Working Groups, "The Matrix", available at http://www.w3.org/QA/TheMatrix.
TAXONOMY
QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy.
OPS-CHECKLIST
Appendix to this operational guidelines document that presents all checkpoints in a table sorted by guideline. Available at http://www.w3.org/TR/2002/WD-qaframe-ops-20030210/qaframe-ops-checklist
OPS-ICS
Appendix to this operational guidelines document that presents all checkpoints in a table sorted by Priority. Available at http://www.w3.org/TR/2002/WD-qaframe-ops-20030210/qaframe-ops-ics
OPS-EXTECH
QA Framework: Operational Examples & Techniques, L. Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Note, (initially) 10 February 2003, companion version to this document, latest companion version available at http://www.w3.org/QA/WG/2003/02/qaframe-ops-extech.
PROTOCOL-WG-TS
The XML Protocol WG has a TS process document, available at http://www.w3.org/2000/xp/Group/1/10/ts-contribution, and a contribution/submission license (example of a submission legal notice), available at http://www.w3.org/2001/10/test-materials-license.html
QA-GLOSSARY
A comprehensive glossary of QA terms, maintained by the QA Working Group. (Initial version under construction.)
QAF-INTRO
QA Framework: Introduction, L. Henderson, K. Gavrylyuk, D. Dimitriadis, L. Rosenthal, Eds., W3C Working Draft, 10 February 2003, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/.
QAF-SPEC
QA Framework: Specification Guidelines, D. Hazaël-Massieux, L. Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 08 November 2002, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/.
QAF-TEST
QA Framework: Test Guidelines, K. Gavrylyuk, D. Dimitriadis, L. Henderson, M. Skall, P. Fawcett, Eds, W3C Working Draft, 20 December 2002, companion version to this document, available at http://www.w3.org/TR/2002/WD-qaframe-test-20021220/.
QAIG
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/.
QAWG
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/.
QAWGPD
QA Working Group Process Document, defining both the general and QA-specific processes of the QA Working Group, available at (under construction).
DOM Working Group TS
DOM TS Process Document, D. Dimitriadis, Ed., 15 January 2002, available at http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115.
REC-TRACK
Stages and milestones in the W3C Recommendation Track, per the Process Document (Process Document is available at http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
SCHEMA-WG-TS
The XML Schema WG has a TS process document, available at http://www.w3.org/2001/05/xmlschema-test-collection.html, and a contribution/submission license (example of a submission legal notice), available at http://www.w3.org/2001/05/test-materials-license.html
WCAG10
Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/.
WG-QA-RANGE
Email proposal by David Marston, on the QA public mail list, for range of Working Group commitment levels to conformance test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html.

7. Change history

2003-02-10, Last Call Working Draft

Revised Introduction, Conformance chapters for better conformance to SpecGL. Restructure all checkpoints for better verifiability. Add rationale to many checkpoints. Implement resolution of all closed OpsGL issues. Editorial improvements.

2002-11-08, third published WD

Revised Introduction, Conformance for better conformance to SpecGL. Implemented other minor editorial and resolved-issue changes. Make over all checkpoints for better verifiability.

2002-05-15, second published WD

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

[1] http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops

03-11-2002

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

[1] http://www.w3.org/QA/WG/2002/02/qaframe-ops-0225.html

02-28-2002
Major rewrite of the previous draft. Incorporated Lynne's comments about Gd1 and Gd4, split Gd regarding QA process, added bunch of new checkpoints.