QA Framework: Operational Guidelines

W3C Candidate Recommendation @@02 September 2003@@

This version:

http://www.w3.org/QA/WG/2003/09/qaframe-ops-20030905

Latest (published) version:

http://www.w3.org/TR/qaframe-ops/

Previous version(s):

Published: http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/

WD-discussion: http://www.w3.org/QA/WG/2003/08/qaframe-ops-20030804

Editors:

Lofton Henderson (lofton@rockynet.com)

Dominique Hazaël-Massieux (dom@w3.org)

Lynne Rosenthal (lsr@nist.gov)

Kirill Gavrylyuk (kirillg@microsoft.com)

Contributors:

See Acknowledgments.

Copyright © 2003 W3C ^® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply.

---

Abstract

This document defines a common Operational Framework for building conformance test materials for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Guidelines.

Status of this document

This version is a QA Working Group (QAWG) editor's draft, but very close to the final text for the Candidate Recommendation version. CAVEAT: some of the links to companion documents that will be published simultaneously are not yet properly configured, and won't be until the document are moved to /TR/ space.

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This document is a W3C Candidate Recommendation (CR), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

The QA Working Group has made changes and improvements in response to Last Call comments, as documented in Change History. At a high level, the normative requirements of the guidelines and checkpoints are substantially the same. However, much of the text has been edited for clarity and simplicity. New structuring and CSS styling have been applied to all of the guidelines and checkpoints.

The QA Working Group resolved to seek transition of Operational Guidelines to Candidate Recommendation (CR) in order to gather trial application experience from the W3C Working Groups. The CR period will last approximately six months, during which the Working Groups will be invited to assess their practices against the Operational Guidelines. At the end of the CR period, feedback from the Working Groups will be incorporated, and the further processing of Operational Guidelines will be decided.

QAWG has agreed that the implementation criteria to advance beyond CR should be: for each Priority 1 and each Priority 2 checkpoint, two example implementations that successfully fulfill the checkpoint.

Publication as a Candidate Recommendation does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

Please use the provided form to make your comments. If for some reason you are unable to use the form, you may email comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that comments that you make will be publicly archived and available, do not send information you would not want to see distributed, such as private data.

Table of contents

1. Introduction
   • 1.1. Document overview
   • 1.2. Scope and goals
   • 1.3. Class of product and audience
   • 1.4. Motivation and expected benefits
   • 1.5. Relationship to other specifications
   • 1.6. Understanding this document
   • 1.7. Checklists to assess conformance
   • 1.8. Checkpoint priorities
   • 1.9. A chronological view of the guidelines
   • 1.10. Usage of terminology in this document
   • 1.11. Use cases
2. Guidelines
   • Guideline 1. Commit to Quality Assurance in Working Group activities.
   • Guideline 2. Commit to resource level for Working Group QA activities.
   • Guideline 3. Synchronize QA activities with the specification milestones.
   • Guideline 4. Define the QA process.
   • Guideline 5. Plan test materials development.
   • Guideline 6. Plan test materials publication.
   • Guideline 7. Plan the transfer of test materials to W3C if needed.
   • Guideline 8. Plan for test materials maintenance.
3. WG relationship to QA Activity
4. Conformance
   • 4.1. Normative parts
   • 4.2. Extensibility
   • 4.3. Conformance requirements & test assertions
   • 4.4. Checkpoint priority definitions
   • 4.5. Conformance definition
   • 4.6. Conformance claims
   • 4.7. Conformance disclaimer
5. Acknowledgments
6. References
   • 2.1. Normative references
   • 2.2. Informative references
7. Change history

Two separate appendices to this document [OPS-CHECKLIST] and [OPS-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by their priorities, for convenient reference. The latter is an Implementation Conformance Statement (ICS) pro forma for this specification.

1. Introduction

1.1. Document overview

The principal parts of this specification are:

• This introduction provides the context for understanding the requirements in section 2, including some use cases.
• Section 2 presents eight guidelines, each consisting of a number of checkpoints that contain lists of conformance requirements.
• Section 3 briefly summarizes the relationship of W3C's Working Groups to the QA Activity
• Section 4 defines conformance to this specification and explains how to make conformance claims.
• Two checklists in external appendices present: all checkpoints in document order; and, all checkpoints sorted by priority (the ICS for this specification).

Integrated with this specification on a checkpoint-by-checkpoint basis, although published separately, is:

• Operational Examples & Techniques, which provides suggestions and examples of how each checkpoint might be satisfied, as well as templates to aid the implemention of the checkpoints' requirements.

1.2. Scope and goals

The scope of this specification is a set of verifiable requirements for the process and operational aspects of the quality practices of W3C Working Groups. The primary goal is to help the W3C Working Groups (WGs) with the planning, development, deployment, and maintenance of conformance test materials (TM).

For this guidelines document, the term conformance test materials includes:

• conformance test suites,
• validation tools,
• conformance checklists,
• any other materials that are used to check or indicate conformance of an implementation to a specification.

1.3. Class of product and audience

The class of product that is the target of the requirements in this specification is: all process and operational aspects of Working Group quality practices. This includes

• the WG's planning and commitment for QA,
• staffing to meet commitments,
• organization and logistics of the quality practices,
• publication and maintenance,
• synchronization of quality processes with other WG activities.

The intended audience for these guidelines is all W3C Working Group members, as well as the actual developers of conformance materials for W3C specifications.

Early adoption of sound quality practices by the Working Groups (WGs) is optimal. Nevertheless, these guidelines are written for newly chartered WGs and existing WGs alike. Working Groups who may already be doing some of these activities should review the document and incorporate the principles and guidelines into their quality practices as much as possible.

1.4. Motivation and expected benefits

As the complexity of W3C specifications and their interdependencies increase, quality assurance becomes even more important to ensuring acceptance and deployment in the market.

These guidelines aim to capture the experiences, good practices, activities, and lessons-learned of the Working Groups, and to present them in a comprehensive, cohesive set of documents for all to use and benefit from. They thereby aim to:

• standardize the best of current practice,
• allow the WG's to reuse what works rather than having to reinvent,
• which should facilitate and expedite the work of the WGs,
• and should also promote consistency across the various WG quality activities and deliverables.

Read more in the "Motivation" and other sections of the "Overview" chapter of QA Framework: Introduction.

1.5. Relationship to other specifications

This document is part of a family of seven QA Framework documents, whose aim is to help the WGs improve all aspects of their quality practices by solidifying and extending current quality practices found within the W3C.

The QA Framework family includes:

• an introduction, QA Framework: Introduction [QAF-INTRO],
• a set of three guidelines documents,
• a companion set of three examples & techniques documents.

The Introduction should be read. It is the key to understanding the whole QA Framework family, and especially the pairwise relationship between the guidelines documents and their examples & techniques counterparts.

The QA Framework documents are independently implementable, but still are interrelated and complementary. For example, there is a close relationship between the WG processes for dealing with versions and errata (in this Operational Guidelines), and the test framework architecture for handling multiple versions and errata levels (in Test Guidelines).

1.6. Understanding this document

This specification uses a model of organizing guidelines and implementable, verifiable checkpoints. Each guideline and its checkpoints are organized and structured as follows:

Guideline [Number]. [Statement of the guideline]

[some discussion of the guideline and its rationale]

Checkpoints:

Checkpoint [Number]. [the statement of the checkpoint (its "title")] [Priority [Level]]

Conformance requirements:

[a statement of the conformance requirements of the checkpoint, i.e., how to fulfill the checkpoint]

Discussion [, Notes, Rationale]

[optional sections of informative notes, rationale, clarifying examples, and cross references to related guidelines or checkpoints]

Examples & Techniques for checkpoint.

Notes:

• the checkpoint priority has value 1, 2 or 3;
• "Examples & Techniques for checkpoint" is a link to the corresponding section in QA Framework: Operational Examples & Techniques; [OPS-EXTECH]
• the "Conformance requirements" section contains one or more individual conformance requirements that use the RFC2119 keywords (the Conformance clause discusses this more).

1.7. Checklists to assess conformance

Two checklists facilitate the use of this guidelines specification. The checklists are in two separate appendices:

• a single table containing all guidelines and checkpoints sorted in their original order [OPS-CHECKLIST];
• all checkpoints sorted by their priorities into three tables, respectively containing the checkpoints of prioritiy 1, 2, 3 (see next section) [OPS-ICS].

The first checklist provides a topical outline of the "Guidelines" chapter of this specification. The second checklist is an Implementation Conformance Statement (ICS) pro forma for this specification.

Use the ICS checklist to record a conformance assessment of a Working Group to this specification. For each checkpoint, consider the individual conformance requirements of the checkpoint and record in the ICS checklist whether the checkpoint is satisfied, or not satisfied, or not applicable.

1.8. Checkpoint priorities

Some checkpoints are more critical than others for the timely production of high-quality, highly usable test materials. Therefore each checkpoint has a priority level -- Priority 1 (highest), Priority 2, or Priority 3 -- based on the checkpoint's importance. See the priority definitions in the Conformance clause.

1.9. A chronological view of the guidelines

"Earlier is better" is a general truism for QA planning and execution. It is also true that later (than ideal timing) is better than nothing. Recognizing the diverse realities of W3C's many existing working groups, the Operational Guidelines specifically avoided binding the guidelines to any ideal timeline.

This table presents an example of an ideal chronological correlation, between the individual guidelines and the stages of a WG's life and work. The ideal example assumes test materials (TM) development within the WG, starting at early working drafts, proceeding in parallel with specification development, and culminating with TM publication at the start of CR.

Notes:

• Abbreviations used for Recommendation-track phases: WD -- (published) Working Draft; LC -- Last Call WD; CR -- Candidate Recommendation; PR -- Proposed Recommendation; Rec -- Recommendation
• The "applies" sub-row is about the stage(s) of WG life and work to which the topic of the guideline applies
• The "address" sub-row is about the stage(s) at which the WG ideally addresses the requirements of the guideline.

1.10. Usage of terminology in this document

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they are all upper case. Occurrences of these words in lower case comprise normal prose usage, with no normative implications.

Unusual terms in this framework document are defined when first used. Generally useful QA-specific terms will usually be put into the QA Glossary [QA-GLOSSARY] eventually. When used in this specification, terms have the meanings assigned herein, or in the QA Glossary if they are not defined herein.

1.11. Use cases

The following scenarios illustrate typical uses of Operational Guidelines:

• A new Working Group is just drafting a Charter. The Charter-drafting team, including prospective WG members, consults the QA Framework: Introduction. Appreciating the payback of early quality commitment and integration, the team refers to Operational Guidelines. Guided by the checkpoints of the charter-phase guidelines, and using the provided Charter Template, the WG writes its quality level and test materials commitments into its draft Charter.
• An existing Working Group has just finished writing its Requirements and Use Cases documents, and is beginning to draft its specification. Concurrently, it is starting to discuss test suite (TS) plans. Consulting Operational Guidelines, it jump starts the TS project by appointing a QA Moderator and a part-time TS team, and quickly produces a first version of its QA Process Document (QAPD) using the provided QAPD template.
• A Working Group has been re-chartered to finish a Second Edition (maintenance release) of its specification, and to develop the next functional version, 2.0. The group did not develop a test suite (TS) during its first charter, but a collaboration of outside organizations and an industry consortium has developed one. The WG and developers have agreed in concept to transfer it to a stable home in W3C. Operational Guidelines is consulted for a detailing of preliminary steps, requirements, and specific activities involved in making a smooth and trouble-free transfer.
• A Working Group is almost ready to request Candidate Recommendation (CR), and has met its goal of having a comprehensive test suite (TS) ready for the trial implementation period of CR. Looking at making the TS publicly available, it uncovers numerous unanticipated procedural issues about TS publication. The issue of the right TS distribution licenses is proving particularly hard. Consulting Operational Guidelines, the WG finds discussion of the pros and cons of the W3C Software License and the Document License, as well as guidance on resolving an optimal licensing strategy .
• A Working Group has finished building a basic test suite (TS) for its specification, and is nearly finished with plans for the first release of its basic TS. After the release, it wants to take advantage of the large test collections of several early implementors, and start accepting contributions. Rather than figure out the issues and write a Test Contribution & Review process from scratch, it takes guidance from Operational Guidelines, and uses the examples and templates of Operational Examples &Techniques to significantly shorten the development of its contribution process.

2. Guidelines

Guideline 1. Commit to Quality Assurance in Working Group activities.

In order for W3C Web standards to achieve full interoperability and access to all, the quality of implementations needs as much attention as specifications' development. The quality of specifications and the early availability of test materials (TM) are significant contributors to the quality of implementations.

Early commitment to quality practices at a suitable level, followed by identification of specific QA deliverables, milestones, and advancement criteria for the WG's specifications, is a prerequisite for detailed planning, execution, and timely achievement of a suitable quality level across all of a WG's activities.

This guideline and Guideline 2 concern the early statement by the Working Group of degree(s) of QA commitment, commitment to test materials and other QA deliverables, and commitment to needed QA staffing resources. See also the chronological view of the guidelines in Chapter 1.

New Working Groups -- i.e., those that are writing and submitting their charters for approval -- are required in this guideline's checkpoints to make the specified commitments in their charters. Working Groups that are renewing their charters are considered the same as new WGs. Existing Working Groups could satisfy the checkpoints by amendment to their charter, or in other ways, the latter to be specified in QA Framework: Operational Examples & Techniques [OPS-EXTECH].

Checkpoints:

Guideline 2. Commit to resource level for Working Group QA activities.

It is highly beneficial to the success of the Working Group's QA work that it address QA resource requirements from the beginning of the Working Group formation, and reexamine the requirements throughout the WG's life cycle.

This guideline and Guideline 1 concern the early statement by the Working group of levels of overall QA commitment, commitment to test materials and other QA deliverables, and commitment to needed QA staffing resources. See further discussion of the chronology of the guidelines in "Introduction".

The sooner the resource needs are defined and a resource level is committed, the better. Therefore new Working Groups -- i.e., those that are writing and submitting their Charters for approval -- are required to have the resource commitments in their charters. Working Groups whose charters are being renewed are considered the same as new WGs. While "sooner is better", existing Working Groups can still satisfy the checkpoints, in other ways to be specified in QA Framework: Operational Examples & Techniques [OPS-EXTECH].

Checkpoints:

Guideline 3. Synchronize QA activities with the specification milestones.

The benefits of starting synchronization of the specification and test materials development as early as possible include:

• it gives an extensive list of use cases;
• it identifies ambiguities in the specification at the early stages, which saves cycles in the later phases;
• it provides clear set of testable statements, which in turn facilitates development of conformant and interoperable implementations. The latter is a W3C process criterion for entering the Candidate Recommendation phase.

Checkpoints:

Guideline 4. Define the QA process.

A Working Group QA process encompasses all aspects of QA life within the Working Group, including:

• a QA leader -- the moderator;
• a task force commensurate with QA commitment;
• producing the QA Process Document (QAPD);
• communications;
• and policies such as branding.

Documented examples of the QA process can be found at DOM TS process, at XML Schema TS process, and at XML Protocol TS process.

Checkpoints:

Guideline 5. Plan test materials development.

A Working Group may have different levels of involvement in test materials development. Nevertheless at any level a Working Group needs a clear understanding of what development framework it will use for its test materials, and how to insure the quality and usability of the test materials themselves. Before starting a test materials development, it is also recommended the test materials developer (Working Group or third party) looks at Test Guidelines and decides what approach to take for test materials organization, test criteria, etc.

Checkpoints:

Guideline 6. Plan test materials publication.

Once the test materials (TM) development is in progress, a Working Group needs to publish the TM drafts and releases, as part of its QA processes.

When the test materials have reached some advanced milestone of maturity and development (e.g., operationally usable), W3C needs to ensure that:

• their support and maintenance includes a secure repository;
• tests materials are freely available for download;
• any licenses do not preclude the test suite from being used by all interested parties;
• publication of test results is encouraged.

Meeting the needs of TM publication necessarily involves some aspects of TM management, such as repository. These prerequisite aspects are addressed in the following checkpoints, as well as publication details proper.

Checkpoints:

Guideline 7. Plan the transfer of test materials to W3C if needed.

A prior checkpoint about test materials repository (Checkpoint 5.2) recommends that test materials reside in the W3C. If test development was done outside by some external entity or group (EG), and the Working Group (WG) together with EG decided to move test materials (TM) to W3C, the following checkpoints define requirements for the transfer process.

Some aspects of the transfer of a whole test suite from outside W3C parallel aspects of in-house planning and development, and piecemeal contribution and review. Those considering such transfers might not already have the planning, development, and submission infrastructure in place. Previous transfer experiences have shown that it is useful to isolate and collect those aspects in one place, as this guideline does.

None of the checkpoints of this guideline are applicable if the Working Group does not transfer test materials from an external entity.

Checkpoints:

Guideline 8. Plan for test materials maintenance.

The ongoing maintenance of test materials is critical to their long term integrity, and may require a significant commitment of resources. The checkpoints of this guideline address planning and resource commitment requirements to ensure that proper specification tracking happens, and that maintenance and update procedures are provided for.

If the Working Group is not re-chartered, then it is problematic where the resources will come from. In this case, it is up to the Director to determine how the work should be done. Some options include:

• it may be assigned to a Coordination Group;
• it may be assigned to a related Working Group;
• the Team could take care of minor items.

Checkpoints:

3. WG relationship to QA Activity

The QA Activity works closely with other W3C WGs, providing assistance and expertise in helping them achieve their QA goals and deliverables. The QA Activity anticipates different relationships with the various WGs depending on the QA-specific needs of the particular Working Group. For example, the resources and experience of Working Group members as well as their stage in the Recommendation track may influence the type and level of collaboration between the Working Group and QA Activity. Potential relationships between Working Groups and the QA Activity include:

• liaison and consultation
• active reviews
• adjudicate entry and exit criteria
• QA resources supplement
• resolution of external QA requests.

More details about the ways in which the QA Activity may interact with and assist the W3C WGs may be found in the QA Working Group Process Document [QAWGPD].

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

4.1. Normative parts

The following parts of this specification are normative:

• Statements prefaced by the keywords, "Conformance requirements:"
• Statements using an uppercase keyword from RFC 2119.
• Section 3, Conformance.
• The priority level associated with each checkpoint statement.
• Any non-applicability statement(s) associated with a checkpoint.

Text that is designated as normative is directly applicable to achieving and assessing conformance to this document. Informative parts of this specification consist of examples, extended explanations, terminology, and other information that facilitates correct implementation of this specification.

4.2. Extensibility

This specification is extensible. That is, the Working Groups MAY set up quality-related processes and operations in addition to those required for conformance to this specification. Extensions to this specification MUST NOT contradict or negate the requirements of this specification.

The rationale for allowing Working Groups to define extensions to these operational guidelines is that these requirements are considered to be the minimal requirements for successful quality practices within the WGs. Doing more than the minimum is not only acceptable, but beneficial. Extensions also allow Working Groups tailor their quality practices more closely to their specific needs. The guidelines of this specification may not be sufficient to meet the needs of all WGs.

4.3. Conformance requirements & test assertions

Each checkpoint contains one or more conformance requirements. These are prefaced by "Conformance requirements:", highlighted in a different style (on appropriate media), and contain at least one of the RFC 2119 [RFC2119] keywords (MUST, SHOULD, MAY, etc).

This Operational Guidelines document does not enumerate a list of test assertions. Test assertions can be derived from conformance requirements in a straightforward manner.

4.4. Checkpoint priority definitions

Each checkpoint has a priority level based on the checkpoint's impact on the quality practices of a Working Group. The meanings of the priorities are:

[Priority 1]

Critical/essential. These checkpoints are considered to be basic requirements for ensuring that test materials are usable, and are produced in time to ensure the quality of the standard and its implementations. Satisfying these checkpoints is a basic requirement to ensure quality and interoperability of the standard.

[Priority 2]

Important/desirable. Satisfying these checkpoints, in addition to the priority 1 checkpoints, should significantly improve the usability and timeliness of the test materials, as well as the quality of the standard and its implementations.

[Priority 3]

Useful/beneficial. Satisfying these checkpoints, on top of all the others, will further improve the quality, usability, and timeliness of the test materials and the standard itself.

4.5. Conformance definition

This section defines three conformance levels to this guidelines specification:

• A-Conforming: all Priority 1 checkpoints are satisfied;
• AA-Conforming: all Priority 1 and Priority 2 checkpoints are satisfied;
• AAA-Conforming: all Priority 1, Priority 2, and Priority 3 checkpoints are satisfied;

This three-level conformance scheme (A, AA, AAA) was chosen in order to facilitate progressive (good-better-best) pursuit of the goals and benefits of these guidelines.

A checkpoint is satisfied by satisfying all of the individual mandatory conformance requirements. Mandatory requirements are those that use the conformance keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", or "SHALL NOT".

Note that it is allowed (even encouraged) to implement checkpoints in addition to those required to satisfy a particular degree of conformance (A, AA, or AAA). The more checkpoints that are satisfied, the better. However, there is no additional conformance designation for such intermediate collections of checkpoints (i.e., for all checkpoints of a given level plus some, but not all, of the checkpoints of the next levels).

The minimum recommended conformance to this specification is A-conforming -- all Priority 1 (critical/essential) checkpoints satisfied.

4.6. Conformance claims

An assertion of conformance to this specification -- i.e., a conformance claim -- MUST specify:

• The name and URI of the Working Group
• The guidelines title: QA Framework: Operational Guidelines
• The guidelines dated URI: http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/
• The date of the claim
• The degree of conformance satisfied: A-Conforming, AA-Conforming, or AAA-Conforming.

The priority-ordered checklist for this specification ([OPS-ICS]) is the Implementation Conformance Statement (ICS) pro forma for this specification. Any assertion of conformance to this specification MUST link to a completed ICS.

Example of a valid conformance claim:

The QA processes and operations of this FooBar Working Group, http://example.org/FooBar/, conform to W3C's 'QA Framework: Operational Guidelines', available at http://www.w3.org/TR/2003/WD-qaframe-ops-200302110/, on 23-April-2003, AA-Conforming. The Implementation Conformance Statement for this claim is available at http://example.org/FooBar/Test/OpsGL-ICS .

4.7. Conformance disclaimer

The checkpoints of this specification present verifiable conformance requirements about the operational aspects of Working Group quality processes. As with any verifiable conformance requirements, users should be aware of this disclaimer:

Passing all of the requirements to achieve a given conformance degree -- A, AA, or AAA -- does not guarantee that the subject operations and processes are well-suited to or will achieve their intended purposes, nor does it guarantee the quality or suitability of test materials produced under the processes.

5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

• Daniel Dardailler (W3C),
• Dimitris Dimitriadis (Ontologicon),
• Karl DuBost (W3C),
• Peter Fawcett (RealNetworks),
• Kirill Gavrylyuk (Microsoft),
• Dominique Hazael-Massieux (W3C),
• Lofton Henderson (CGM Open),
• Ian Jacobs (W3C),
• David Marston (Lotus Development Corp),
• Sandra Martinez (NIST),
• Patrick Curran (Sun Microsystems),
• Lynne Rosenthal (NIST),
• Mark Skall (NIST),
• Andrew Thackrah (Open Group),
• Olivier Thereaux (W3C),

6. References

6.1. Normative references

[PROCESS]

World Wide Web Consortium Process Document, I. Jacobs, Ed., 18 June 2003, available at http://www.w3.org/2003/06/Process-20030618/ .

[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt .

6.2. Informative references

[CONTRIB]

Contribution of Software or Test Materials to W3C, which defines W3C-approved procedures and terms for submission of Software and Test Materials, available at http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231 .

[EXTERN-TA]

QA activity email thread about third-party participation in test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html .

[MATRIX]

W3C-wide conformance activity survey covering all the Working Groups, "The Matrix", available at http://www.w3.org/QA/TheMatrix .

[TAXONOMY]

QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy .

[OPS-CHECKLIST]

Appendix to this operational guidelines document that presents all checkpoints in a table sorted by guideline. Available at http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/qaframe-ops-checklist .

[OPS-ICS]

Appendix to this operational guidelines document that presents all checkpoints in a table sorted by Priority. Available at http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/qaframe-ops-ics .

[OPS-EXTECH]

QA Framework: Operational Examples & Techniques, L. Henderson, L. Rosenthal, K. Gavrylyuk, D. Dimitriadis, Eds., W3C Note, (initially) September 2003, companion version to this document, latest companion version available at http://www.w3.org/QA/WG/2003/09/qaframe-ops-extech .

[PROTOCOL-WG-TS]

The XML Protocol WG has a TS process document, available at http://www.w3.org/2000/xp/Group/1/10/ts-contribution, and a contribution/submission license (example of a submission legal notice), available at http://www.w3.org/2001/10/test-materials-license.html .

[QA-GLOSSARY]

A comprehensive glossary of QA terms, maintained by the QA Working Group. (Initial version under construction.)

[QAF-INTRO]

QA Framework: Introduction, L. Henderson, K. Gavrylyuk, D. Dimitriadis, L. Rosenthal, Eds., W3C Working Draft, 10 February 2003, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/ .

[QAF-SPEC]

QA Framework: Specification Guidelines, D. Hazaël-Massieux, L. Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 08 November 2002, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/ .

[QAF-TEST]

QA Framework: Test Guidelines, K. Gavrylyuk, D. Dimitriadis, L. Henderson, M. Skall, P. Fawcett, Eds, W3C Working Draft, 20 December 2002, companion version to this document, available at http://www.w3.org/TR/2002/WD-qaframe-test-20021220/ .

[QAIG]

Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/ .

[QAWG]

Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/ .

[QAWGPD]

QA Working Group Process Document, defining both the general and QA-specific processes of the QA Working Group, available at (under construction).

[DOM Working Group TS]

DOM TS Process Document, D. Dimitriadis, Ed., 15 January 2002, available at http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115 .

[REC-TRACK]

Stages and milestones in the W3C Recommendation Track, per the Process Document (Process Document is available at http://www.w3.org/Consortium/Process-20010719/, see section 5.2).

[SCHEMA-WG-TS]

The XML Schema WG has a TS process document, available at http://www.w3.org/2001/05/xmlschema-test-collection.html, and a contribution/submission license (example of a submission legal notice), available at http://www.w3.org/2001/05/test-materials-license.html .

[WCAG10]

Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/ .

[WG-QA-RANGE]

Email proposal by David Marston, on the QA public mail list, for range of Working Group commitment levels to conformance test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html .

7. Change history

2003-09-12, Candidate Recommendation version

Clarifications and improvements in response to Last Call comments. Checkpoint numbers, CPx.y, refer to the checkpoint numbers in the Last Call WD of Operational Guidelines. Similarly, "GLz" refers to Guideline "z" in the Last Call version (all of which are the same for this Candidate Recommendation version.)

• New Introduction section -- a table (for now) -- about correlation of guidelines to stages in WG's life and deliverables.
• Rewrote CP1.1, 1.2, 1.3 to preserve all of LC-version normative requirements, but eliminate the table and (implied) conflicting conformance scale, and integrate the commitment definition with other GL documents.
• Added clarifications to GL1/2 verbiage, to emphasize (ideally) that these are Charter-phase activities.
• Wording of GL1 and GL2 improved to emaphasize that they are about deciding and recording commitment to QA and its staffing.
• To CP1.5 and CP3.1, added reciprocal clarifications to disambiguate the similarity between the two.
• To CP2.2, CP4.1 and CP4.2, added reciprocal clarifications to disambiguate the similarity between them.
• Revised CP4.3 verbiage for clarification (per LC-58), to allow a "TOC" to satisfy the QAPD.
• Cross-reference and clarify relationship of CP5.3 (new5.4) and CP6.2, revise verbiage per consensus with W3C Legal and TM-license task team.
• Clarified CP4.5 (esp. the ambiguous use of "the QA framework"), and its motivation/rationale; changed its priority from P1 to P2; moved it to become first checkpoint of GL5 (so that it becomes new5.1).
• Moved some verbiage out of CP6.1 into GL6, clarified about maintenance & publication.
• Added some clarification and rationale to verbiage of GL7.
• Improved testability of the language in CP3.1, CP3.2, CP6.1, CP4.2, CP2.2, CP6.3, CP7.1, CP7.2, CP7.3.
• Change CP8.2 from P2 to P1, CP3.2 from P3 to P1, and clarify the requirements and relationship between the two checkpoints, and clarify "versioning".
• Better explanation of Checklists and their usage, in Introduction.
• Better exposure of the Examples & Techniques documents, in Introduction, by reference to "QA Framework: Introduction".
• Editorial clarification of wording in GL3, CP5.4 (new5.5), CP6.3, CP6.5.
• Applied new styling throughout, for clearer presentation of Checkpoints and their subsections.
• Complete re-write of (informative) Introductory chapter, for brevity, clarity, and readability.
• Clarification of minimal recommended conformance level (A) in Conformance clause.
• Clarification of rationale/justification for 3-level (A, AA, AAA) conformance scheme in Conformance clause.
• Short section of Use Cases added to end of Introduction.
• Added "promote conformance testing" refinement to the "publish test results" of CP6.5.
• Added new "Document overview" subsection to the beginning of the Introduction.

2003-02-10, Last Call Working Draft

Revised Introduction, Conformance chapters for better conformance to SpecGL. Restructure all checkpoints for better verifiability. Add rationale to many checkpoints. Implement resolution of all closed OpsGL issues. Editorial improvements.

• Finish "add Rationale" (below) for those CP which need one
• extensive editorial improvements to wording.
• In all checkpoints, changed "To fulfill..," verbiage to "Conformance requirements:"
• Changed "Test assertions" subsection of Conformance chapter to "Conformance requirements & test assertions", revised its wording (no test assertions).
• Replace most of Ch.3, " WG relationship to QA Activity", with a reference to the QAWG Process Document.
• Removed Sec 1.3 (Goals of QA Development), revised 1.1, 1.2, 1.4.
• Add to CP5.3 (submission license policy) "Discussion" a reference to W3C-wide submission policies.
• Modified CP6.2 (TM License) to subsume a special TM License under "W3C-approved custom license", while discussion with W3C legal is still pending.
• Clarified wording of CP3.2 (spec versioning/errata support in TM).
• Added "Rationale" to all checkpoints for which it was missing.
• Revised "commitment table" of CP1.1 to clarify levels and add conformance criteria.
• Clarified fulfillment criteria of CP3.2.
• Remove 2nd part ("sufficient" clause) of Conformance Disclaimer.
• Revised checkpoint wording and fulfillment criteria of CP8.1
• edited Discussion of several checkpoints, added Rationale to a couple (CP4.3, 7.3).
• revised Conformance Section to comply w/ SpecGL (extensibility, normativity, conformance claims, etc).
• revised Introduction to comply w/ SpecGL -- added scope, class of product, etc.
• CP5.3, replace requirement for license with requirement for license policy.
• CP5.3, add links to a couple of sample submission licenses.
• separate the normative references from the informative references (and label them).
• add links to sample QA (TS) process documents to GL4 verbiage.
• CP2.3, change requirement on existing WGs.
• CP6.4, the disclaimer must prominent.
• CP6.5, adjust the "may" usage in the fulfillment criteria and discussion.
• clean up discussion in CP5.1
• qualify "public, archived" for mailing list and "public" for /Test/ Web page.
• revise GL3 and CP3.1 to explicitly link QA deliverables to specification versions.
• change post- WG "maintenance" comments in CP6.1 to repository.
• include post- WG maintenance comments in CP8.1.
• clarify CP4.6 (branding) (new4.5), and stipulate "not applicable" if branding is not supported.
• remove "discussion" from list requirements in CP4.4, clarify requirements.
• clarify minimal QAPD contents in CP4.3 and enumerate them in the discussion.
• move content from CP4.1 to Ops Extech, leave brief summary in its place.
• changes to wording in QA commitment table of GL1

2002-11-08, third published WD

Revised Introduction, Conformance for better conformance to SpecGL. Implemented other minor editorial and resolved-issue changes. Make over all checkpoints for better verifiability.

• (since 11/03) Moved a handful of examples to Ops Extech.
• (since 10/31)
  • Make over of GL3-GL8, to isolate the "to fulfill" criterion and apply TA markup;
  • Numerous minor editorial (wording & grammar) improvements.
• Miscellaneous editorial corrections, fix missing anchors, etc.
• Combine CK5.5 into CK5.4 as new MUST requirement.
• In GL1 and GL2, apply test assertion markup, to identify fulfillment criteria.
• In GL2, remove the "In the charter" bits of all checkpoints, and embed old/new requirements in the fulfillment criteria.
• In GL1, remove the "In the charter" bits of all checkpoints, and embed old/new requirements in the fulfillment criteria.
• Identify the checklist as the ICS pro forma for OpsGL.
• Revise "Conformance" chapter per SpecGL (level/degree, test assertions, etc)
• Rewrite descriptions of priority 1, 2, 3 (subsection of Intro)
• Clarify RFC2119 usage ("Terminology" subsection of Intro)
• Reorganize and augment Introduction, w/ subsections for: scope&goals, motivation, understanding&using, priorities, etc.

2002-05-15, second published WD

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

• Ck 4.1 - split into 2 having QA moderator as Pri1 and the task force as Pri2
• Ck 4.4 - advanced to pri1
• Ck 4.5 - removed "if applicable"
• Ck 5.1 - advanced to Pri1, remove accredited, changed to "usable for intended purpose"
• Ck 5.3 - advanced to Pri1
• Advanced Ck 6.1 to Pri1, removed "future", add "freely available"
• Ck 6.2 - advanced to Pri1
• Ck 6.4 - advanced to Pri1.
• Ck 6.5 reworded checkpoint and explanation text. advanced to Pri2
• Ck. 7.1. Added a note on how this checkpoint is related to 5.4 "review process". Clarified what is the assessment quality by referring to the criteria for the review process.
• Added Ck 5.5 with Pri1 as a criteria for the review process - needed for a reference from other places where we talk about "tests quality assessment criteria". Pri1 is needed to match the minimum requirement in the WG QA commitment.
• Ck 7.2 - added note on how this relates to the Ck 2.2 and moved it to Pri1
• Ck 7.4 - removed
• Ck 8.1 - reworded.
• Ck 1.1 - changed the commitment table based on Mark's suggestions: added requirement for some test assertions to the level 2, added requirement for tests review to the Level 3, swapped level 5 and level 6.
• Added 2 more checkpoints in advancement of Ck 1.1: with pri2 to commit to the level 5 and with pri3 to commit to the level 7. Explanation notes are there
• Added first draft for Definitions of the Ck Priorities to the Introduction section.
• Added Conformance disclaimer subsection drafted by Lofton, per WG decision.
• Other rewrites and small editorial changes.

[1] http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops

03-11-2002

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

• Incorporated Lynne's contribution - Ch3 rewrite
• add checkpoint for soliciting QA resources in Call for Participation with P2
• minor rephrasing of statements of chkpt 1.1, 1.2, and 1.3
• rephrase "CR-exit" discussion
• Changed chkpt 2.2 from [1] to have P1
• Changed chkpt 1.2 from [1] to have P1
• Removed 3.1, added more explanation notes to 3.2 (P2) and 3.3 (P3)
• Removed the checkpoint 4.1 from [1] and add it to the Intro section
• Removed the checkpoint 5.1
• Added more explanation, clarified the difference between chckpt 6.3 and 6.5 from [1]
• Added more explanation, clarified the difference between 5.3 and 5.5 from [1]
• Added text to explain why the WG SHOULD NOT publish test materials in TR space
• Added text to explain why the Document License is recommended
• Added mild disclaimer on the use of the "SHOULD", "MUST", etc keywords to the Terminology section
• Other rewrites and small editorial changes

[1] http://www.w3.org/QA/WG/2002/02/qaframe-ops-0225.html

02-28-2002

Major rewrite of the previous draft. Incorporated Lynne's comments about Gd1 and Gd4, split Gd regarding QA process, added bunch of new checkpoints.