W3C

QA Framework: Operational Guidelines

W3C Working Draft 04 August 2003 (r3)

This version:
http://www.w3.org/QA/WG/2003/08/qaframe-ops-20030804
Latest (published) version:
http://www.w3.org/TR/qaframe-ops/
Previous version(s):
Published: http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/
WD-discussion: http://www.w3.org/QA/WG/2003/06/qaframe-ops-20030711
Editors:
Lofton Henderson (lofton@rockynet.com)
Dominique HazaŽl-Massieux (dom@w3.org)
Lynne Rosenthal (lsr@nist.gov)
Kirill Gavrylyuk (kirillg@microsoft.com)
Contributors:
See Acknowledgments.

Abstract

This document defines a common Operational Framework for building conformance test materials for W3C specifications. It presents operational and procedural guidelines for groups undertaking conformance materials development. This document is one in a family of Framework documents of the Quality Assurance (QA) Activity, which includes the other existing or in-progress specifications: Introduction; Specification Guidelines; and, Test Guidelines.

Status of this document

This version is a QA Working Group (QAWG) editor's draft, for illustrating and discussing QAWG-agreed modifications to the Last Call version.

This contains information that will be included in the Status section of the Candidate Recommendation version...

This version is the last in a sequence of editor's drafts (see "Previous versions"). Its purpose is to accompany the Last Call (LC) "Disposition of Comments" (DoC) document, to illustrate to commentors how their comments have been addressed.

This editor's draft is not yet fully pubrules compliant, principally because of links to other members of the QA Framework document family (whose revision is still in progress).

After the completion of the DoC resolution process with Last Call issue originators, the planned next step for Operational Guidelines is to request transition to Candidate Recommendation (CR). It is intended that the CR period will last at least six months, during which the Working Groups will be invited to assess their operations against the Operational Guidelines. At the end of the CR period, feedback from the Working Groups will be incorporated, and the further processing of Operational Guidelines will be decided.

There are numerous changes and improvements in response to Last Call comments. At a high level, the normative requirements of the guidelines and checkpoints are substantially the same. However, lots of text has been edited for clarity and simplicity, and new structuring and formatting (via CSS styling) has been applied to all of the guidelines and checkpoints. This extensive editorial make-over probably makes a 'diff' version impractical, but the more significant impovements are documented in Change history.

Some of the formatting make-over is still in progress. Especially, new markup and formatting is applied to the earlier guidelines and checkpoints, but not to the later ones yet. This will all be completed by the time CR text is published.

The "Status" section of the Last Call version follows...

This section describes the status of this document at the time of its publication. Other documents may supersede this document. The latest version of this document series is maintained at the W3C.

This document is a W3C Working Draft (WD), made available by the W3C Quality Assurance (QA) Activity for discussion by W3C members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.

This Working Draft is published for Last Call review. This version incorporates the closure of all open QAWG (QA Working Group) issues about this specification, as well as editorial improvements to more clearly explain the rationale behind a number of the checkpoints. The Previous Version (3rd published WD) represented convergence on a stable set of the individual checkpoints and their priorities, and was structured so that its conformance requirements (fulfillment criteria) were clearly isolated and identified. For details, please see Change history.

This version supersedes all previous drafts. Future progression of this document beyond Working Draft is planned, but the final status has not been determined at this time. See QAWG issue #18 and issue #71. It is anticipated that this specification will eventually advance to Candidate Recommendation (CR), after successful discussion and resolution of any and all issues that arise during Last Call review. The QAWG has discussed criteria for finishing CR phase and entrance to Proposed Recommendation ( PR). The agreed criteria are: for each Priority 1 and Priority 2 checkpoint, two example implementations that successfully fulfill the checkpoint.

This part of the Framework document family has a companion QA Framework: Operational Examples & Techniques. That informative companion document is currently being progressed as a W3C Note. At least until this guidelines document stabilizes, that Examples & Techniques companion will be maintained and frequently updated in QA Working Group Web space (as opposed to /TR/). Although that document is not the principal subject of this review, the QAWG welcomes feedback on it as well.

The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.

Please use the provided form to make your comments. If for some reason you are unable to use the form, you may email comments to www-qa@w3.org, the publicly archived list of the QA Interest Group [QAIG]. Please note that comments that you make will be publicly archived and available, do not send information you would not want to see distributed, such as private data.

Publication of this document does not imply endorsement by the W3C, its membership or its staff. This is a draft document and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use W3C Working Drafts as reference material or to cite them as other than "work in progress".

A list of current W3C Recommendations and other technical documents can be found at http://www.w3.org/TR/.

Table of contents

  1. Introduction
  2. Guidelines
  3. WG relationship to QA Activity
  4. Conformance
  5. Acknowledgments
  6. References
  7. Change history

Two separate appendices to this document [OPS-CHECKLIST] and [OPS-ICS] present all checkpoints in a tabular form sorted in their original order and sorted by their priorities, for convenient reference. The latter is an Implementation Conformance Statement (ICS) pro-forma for this specification.


1. Introduction

1.1. Scope and goals

The scope of this specification is a set of verifiable requirements for the process and operational aspects of the quality practices of W3C Working Groups. The primary goal is to help the W3C Working Groups (WGs) with the planning, development, deployment, and maintenance of conformance test materials (TM).

For this guidelines document, the term conformance test materials includes:

1.2. Class of product and audience

The class of product that is the target of the requirements in this specification is: all process and operational aspects of Working Group quality practices. This includes

The intended audience for these guidelines is all W3C Working Group members, as well as the actual developers of conformance materials for W3C specifications.

Early adoption of sound quality practices by the Working Groups (WGs) is optimal. Nevertheless, these guidelines are written for newly chartered WGs and existing WGs alike. Working Groups who may already be doing some of these activities should review the document and incorporate the principles and guidelines into their quality practices as much as possible.

1.3. Motivation and expected benefits

As the complexity of W3C specifications and their interdependencies increase, quality assurance becomes even more important to ensuring acceptance and deployment in the market.

These guidelines aim to capture the experiences, good practices, activities, and lessons-learned of the Working Groups, and to present them in a comprehensive, cohesive set of documents for all to use and benefit from. They thereby aim to:

Read more in the "Motivation" and other sections of the "Overview" chapter of QA Framework: Introduction.

1.4. Relationship to other specifications

This document is part of a family of seven QA Framework documents, whose aim is to help the WGs improve all aspects of their quality practices by solidifying and extending current quality practices found within the W3C.

The QA Framework family includes:

The introduction should be read. It is the key to understanding the whole QA Framework family, and especially the pairwise relationship between the guidelines documents and their examples & techniques counterparts.

The QA Framework documents are independently implementable, but still are interrelated and complementary. For example, there is a close relationship between the WG processes for dealing with versions and errata (in this Operational Guidelines), and the test framework architecture for handling multiple versions and errata levels (in Test Guidelines).

1.5. Understanding this document

This specification uses a model of organizing guidelines and implementable, verifiable checkpoints. Each guideline and its checkpoints are organized and structured as follows:

Guideline [Number]. [Statement of the guideline]

[some discussion of the guideline and its rationale]

Checkpoints:

Checkpoint [Number]. [the statement of the checkpoint (its "title")] [Priority†[Level]]
Conformance requirements:

[a statement of the conformance requirements of the checkpoint, i.e., how to fulfill the checkpoint]

Discussion [, Notes, Rationale]

[optional sections of informative notes, rationale, clarifying examples, and cross references to related guidelines or checkpoints]

Examples & Techniques for checkpoint.

Notes:

1.6. Checklists to assess conformance

Two checklists facilitate the use of this guidelines specification. The checklists are in two separate appendices:

The first checklist provides a topical outline of the "Guidelines" chapter of this specification. The second checklist is an Implementation Conformance Statement (ICS) pro-forma for this specification.

Use the ICS checklist to record a conformance assessment of a Working Group to this specification. For each checkpoint, consider the individual conformance requirements of the checkpoint and record in the ICS checklist whether the checkpoint is satisfied, or not satisfied, or not applicable.

1.7. Checkpoint priorities

Some checkpoints are more critical than others for the timely production of high-quality, highly usable test materials. Therefore each checkpoint has a priority level -- Priority 1 (highest), Priority 2, or Priority 3 -- based on the checkpoint's importance. See the priority definitions in the Conformance clause.

1.8. A chronological view of the guidelines

"Earlier is better" is a general truism for QA planning and execution. It is also true that later (than ideal timing) is better than nothing. Accordingly, and in recognition of the diverse realities of W3C's many existing working groups, the Operational Guidelines specifically avoided binding the guidelines to any ideal timeline.

This table presents an example of an ideal chronological correlation, between the individual guidelines and the stages of a WG's life and work. The ideal example assumes test materials (TM) development within the WG, starting at early working drafts, proceeding in parallel with specification development, and culminating with TM publication at the start of CR.

Correlation of Guidelines with phases in Working Group (WG) life
GL
nbr
Topic Stage(s) of WG life Comments
Applies at: Address at:
1 QA-level commitment whole WG life CH Charter inclusion ideally; else address as soon as possible (ASAP) thereafter.
2 QA resource commitment whole WG life CH Charter ideally, else ASAP thereafter.
3 QA & specifications synchronization WD, CR, PR, Rec ASAP post-CH synchronization is most important when first TM exist
4 QA Process definition all post-CH stages ASAP post-CH QA moderator first; start QA Process Document (QAPD)
5 TM development plans at or soon after TM started:
WD
[, LC, CR, PR, Rec]
early-WD plan early, apply plans throughout
6 TM publication plans at/soon after TM published:
[WD, LC,] CR [, PR, Rec] (ideally before CR)
LC plans needed well before anticipated TM publication (which is ideally at/before CR)
7 TM transfer (external source) [WD, LC,] CR [, PR, Rec] (ideally before CR) WD-LC having some TM available benefits spec development
8 TM maintenance plans at/soon after TM published, continuing post-Rec (even post-WG) CR maintenance requirements start at publication and continue indefinitely

Notes:

[@@Ed note. KD has offered to look at turning the table into a diagram. Perhaps it should be something like: horizontal shaded bands representing Charter, 1st WD, 1st PWD, LC, CR, PR, Rec, post-Rec; and each GL would be a vertical column, which would actually be a spanning vertical line, from where it's addressing must happen, to where it applies (maybe w/ a gap, or solid/dash for optimal/acceptable, etc). Kind of like a Gantt chart. Thoughts? ]

1.9. Usage of terminology in this document

The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used as defined in RFC 2119 [RFC2119]. When used with the normative RFC2119 meanings, they are all upper case. Occurrences of these words in lower case comprise normal prose usage, with no normative implications.

Unusual terms in this framework document are defined when first used. Generally useful QA-specific terms will usually be put into the QA Glossary [QA-GLOSSARY] eventually. When used in this specification, terms have the meanings assigned herein, or in the QA Glossary if they are not defined herein.

1.10. Use cases

The following scenarios illustrate typical uses of Operational Guidelines:

2. Guidelines

Guideline 1. Commit to Quality Assurance in Working Group activities.

In order for W3C Web standards to achieve full interoperability and access to all, the quality of implementations needs as much attention as specifications' development. The quality of specifications and the early availability of test materials (TM) are significant contributors to the quality of implementations.

Early commitment to quality practices at a suitable level, followed by identification of specific QA deliverables, milestones, and advancement criteria for the WG's specifications, is a prerequisite for detailed planning, execution, and timely achievement of a suitable quality level across all of a WG's activities.

This guideline and Guideline 2 concern the early statement by the Working Group of degree(s) of QA commitment, commitment to test materials and other QA deliverables, and commitment to needed QA staffing resources. See also the chronological view of the guidelines in Chapter 1.

New Working Groups -- i.e., those that are writing and submitting their charters for approval -- are required in this guideline's checkpoints to make the specified commitments in their charters. Working Groups that are renewing their charters are considered the same as new WGs. Existing Working Groups could satisfy the checkpoints by amendment to their charter, or in other ways, the latter to be specified in QA Framework: Operational Examples & Techniques [OPS-EXTECH].

Checkpoints:

Checkpoint 1.1. Define QA commitment levels for operations, specifications, and test materials. [Priority 1]
Conformance requirements:
Discussion:

This checkpoint is about early commitment of the WG to its intended degree of quality practices in various aspects of its activities. It is not about the actual performance or delivery of quality practices and deliverables.

For existing working groups that make the QA commitment after chartering, various ways to do it are discussed in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 1.2. Commit to test materials. [Priority 2]
Conformance requirements:
Discussion:

This checkpoint is about early commitment of the WG about its test materials intentions. It is not about the specific deliverables, nor about the details of the test materials or their actual production and delivery.

For existing working groups that make the test materials commitment after chartering, various ways to do it are discussed in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 1.3. Commit to complete test materials. [Priority 3]
Conformance requirements:
Discussion:

This checkpoint is about early commitment of the WG about its test materials intentions. It is not about the specific deliverables, nor about the details of the test materials or their actual production and delivery. For existing working groups that make the test materials commitment after chartering, various ways to do it are discussed in the Examples & Techniques document.

Examples & Techniques for checkpoint.

Checkpoint 1.4. Enumerate QA deliverables and expected milestones. [Priority 1]
Conformance requirements:
Discussion:

The W3C Process Document requires that Charters identify deliverables and associate them with milestones. Examples of QA deliverables include sets of use cases, primers, collections of test assertions, test suites (produced or acquired), validators, and test harnesses.

Examples & Techniques for checkpoint.

Checkpoint 1.5. Define QA criteria for Recommendation-track advancement. [Priority 2]
Conformance requirements:
Discussion:

It is natural for test suites and implementations to develop in parallel -- each is a help to the development of the other. A de facto convention of test suite completion before entrance into PR phase (and by implication, completion of CR phase) seems to be a common goal amongst a number of the existing test suite efforts. For example, SVG, UAAG, and DOM have defined this criterion. The test suites were used to demonstrate two correct, interoperable implementations of each feature, as recommended in the W3C Process Document.

Examples & Techniques for checkpoint.

Guideline 2. Commit to resource level for Working Group QA activities.

It is highly beneficial to the success of the Working Group's QA work that it address QA resource requirements from the beginning of the Working Group formation, and reexamine the requirements throughout the WG's life cycle.

This guideline and Guideline 1 concern the early statement by the Working group of levels of overall QA commitment, commitment to test materials and other QA deliverables, and commitment to needed QA staffing resources. See further discussion of the chronology of the guidelines in "Introduction".

The sooner the resource needs are defined and a resource level is committed, the better. Therefore new Working Groups -- i.e., those that are writing and submitting their Charters for approval -- are required to have the resource commitments in their charters. Working Groups whose charters are being renewed are considered the same as new WGs. While "sooner is better", existing Working Groups can still satisfy the checkpoints, in other ways to be specified in QA Framework: Operational Examples & Techniques [OPS-EXTECH].

Checkpoints:

Checkpoint 2.1. Address where and how conformance test materials will be produced. [Priority 1]
Conformance Requirements:
Rationale:

Determining whether the WG plans to produce the test materials themselves or get them from other sources is a first step in the overall planning for QA deliverables. Identifying the scenario for producing these deliverables is a prerequisite to estimating the resources and effort needed for the WG's level of QA commitment.

Discussion:

TM-production scenarios range from all intra- WG effort concurrent with other WG deliverables, to importing completed materials from an external group, and hybrid internal-external efforts. Further discussion and example scenarios are found in Operational Examples & Techniques.

Examples & Techniques for checkpoint.

Checkpoint 2.2. Address QA staffing commitments. [Priority 1]
Conformance Requirements:
Rationale:

For any acceptable level of QA commitment, there will be some staffing requirement. If the WG addresses the staffing level at the same time that it defines the commitment level and general scenario, then that both helps to confirm that the overall plans are realistic, as well as facilitates that the commitments will be successfully realized.

Discussion:

Depending upon the general intent and plan, and how the test materials will be built, the commitment can range from minimal to significant. At a minimum, a QA point-of-contact will be needed by the WG.

Examples & Techniques for checkpoint.

Checkpoint 2.3. Request allocation of QA resources to the Working Group. [Priority 1]
Conformance Requirements:
Rationale:

The WG needs explicit indication of resources available for QA, in order to know whether Working Group capabilities are commensurate with its deliverables and milestones. If the WG emphasizes in its participation calls that QA specialists are needed and welcome, that should facilitate getting competent and interested people to work on the WG's QA commitments.

Discussion:

Once the Charter is prepared, the Director sends a Call for Participation to the Advisory Committee. At this point AC Representatives are asked to provide information about amount and type of resources their organization plans to allocate for the particular Working Group. Note that W3C Process [PROCESS] allows a WG to have several participants per Member, and it is up to the WG to determine details such as any further numerical restrictions, composition by specialty, etc.

Examples & Techniques for checkpoint.

Guideline 3. Synchronize QA activities with the specification milestones.

The benefits of starting synchronization of the specification and test materials development as early as possible include:

Checkpoints:

Checkpoint 3.1. Synchronize the publication of QA deliverables and the specification's drafts. [Priority 2]
Conformance Requirements:
Rationale:

Because each published version of the specification -- WDs, CR(s), PR, etc -- is a changed document, all important dependencies such as test materials need to be updated concurrently.

Discussion:

QA deliverables to be synchronized must include at least the test materials to which the WG has committed -- see Checkpoints 1.2, 1.3, and 1.4. For such test materials, the WG might for example synchronize the first public release of a test suite (TS) with Candidate Recommendation (CR). The Working Group should consider and attempt to synchronize other QA deliverables as well, for example:

Examples & Techniques for checkpoint.

Checkpoint 3.2. Support specification versioning/errata in QA deliverables. [Priority 1]
Conformance requirements:
Rationale:

There will be products that support different versions (editions) and/or different errata levels of the specification after it becomes Recommendation. The requirements of this checkpoint ensure that users will be able to select and apply the appropriate test materials for a given product.

Discussion:

This checkpoint specifically extends the previous one to cover support for specification milestones after Recommendation status. As used in this checkpoint, the scope of "versions" at least includes "editions" -- editorial revisions that incorporate errata and small changes -- rather than major versions with major functional increments (i.e., new Recommendations).

Building specification versioning/errata support into the test materials' infrastructure is one effective technique. Other techniques may also suffice, such as reliance on a code versioning system like CVS (Concurrent Versions System).

Examples & Techniques for checkpoint.

Guideline 4. Define the QA process.

A Working Group QA process encompasses all aspects of QA life within the Working Group, including:

Documented examples of the QA process can be found at DOM TS process, at XML Schema TS process, and at XML Protocol TS process.

Checkpoints:

Checkpoint 4.1. Appoint a QA moderator. [Priority 1]
Conformance Requirements:
Rationale:

Having a single person assume primary responsibility for the WG's quality practices will help ensure that its QA commitments and milestones are met efficiently and on schedule. A single identified QA point-of-contact will also optimize external QA-related communications.

Discussion:

The QA moderator is the overall manager of all of the QA activities in the Working Group, and (by default) principal point-of-contact. At even the lowest acceptable levels of QA commitment, there are sufficient requirements that at least part-time of one person is needed. Depending on the anticipated origin of the test materials, there are several possibilities for the QA moderator, ranging from appointing a WG member to inviting someone from an external organization.

Examples & Techniques for checkpoint.

Checkpoint 4.2. Appoint a QA task force. [Priority 2]
Conformance Requirements:
Rationale:

At higher levels of QA commitment and deliverables, the resource requirements may exceed (part-time of) one person. Identifying and assigning any additional needed QA staff resources is a prerequisite to realistic schedules and QA delivery milestones.

Discussion:

Depending on the level of involvement of a Working Group in the test materials development, a task force assisting the QA moderator can take responsibility for a QA framework development, test materials development, review of contributions, and maintenance. In practice, the task force can range from a subset of dedicated individuals, to some fraction of the time of each WG member. The WG needs to ensure that the size of the task force is adequate to meet the committed QA deliverables and QA level, per the committed milestones.

Examples & Techniques for checkpoint.

Checkpoint 4.3. Produce the QA Process Document. [Priority 1]
Conformance requirements:
Rationale:

A QA Process Document records, in one place, key information about the WG's quality-related logistical and communications setup, contribution and licensing policies, publication and maintenance plans, and other critical process and operational details. The production of this document ensures that these key aspects of the WG's quality practices are explicitly addressed and publicly recorded.

Discussion:

The main goal of a QA Process Document (QAPD) is a single starting place to find important QA information about the Working Group. A single document is preferred, and a template for writing such is included in Operational Examples & Techniques. A table of contents comprised of links to distributed WG documents would be another satisfactory way to satisfy the QAPD.

The QAPD may contain more, but it must address at least the requirements of these other checkpoints:

Examples & Techniques for checkpoint.

Checkpoint 4.4. Specify means for QA-related communication. [Priority 2]
Conformance requirements:
Discussion:

In practice, the mailing list may be a dedicated "test" mailing list, or something like an existing Interest Group mail list could be used for the purpose. A common convention for the QA-related Web page is recommended: name it /Test/ directly under the WG's home page, e.g., http://www.w3.org/Graphics/SVG/Test/.

Examples & Techniques for checkpoint.

Checkpoint 4.5. Define branding policy details. [Priority 3]
Conformance requirements:

This checkpoint is not applicable if the WG does not support branding.

Rationale:

Branding policy (like certification) is potentially one of the most contentious topics that a WG might face. If the WG opts to support branding, then defining the branding policy up front ensures that the WG undertakes test materials development (or acquisition) with a common understanding of their ultimate deployment and use. For end users, early definition of branding policy clarifies the WG intentions for a highly visible aspect of test materials' use.

Discussion:

Some W3C activities support "branding". Branding refers to the use of a conformance icon to indicate some level of conformance to the specification. Examples: HTML validation, CSS validation, WCAG conformance. While these content branding schemes are relatively non-controversial, there are nevertheless issues that should be addressed -- particularly in the contexts of user-agent and API conformance -- before any branding-related goals are articulated.

Examples & Techniques for checkpoint.

Guideline 5. Plan test materials development.

A Working Group may have different levels of involvement in test materials development. Nevertheless at any level a Working Group needs a clear understanding of what development framework it will use for its test materials, and how to insure the quality and usability of the test materials themselves. Before starting a test materials development, it is also recommended the test materials developer (Working Group or third party) looks at Test Guidelines and decides what approach to take for test materials organization, test criteria, etc.

Checkpoints:

Checkpoint 5.1. Define a framework for test materials development. [Priority 2]
Conformance requirements:
Rationale:

A test materials development framework is effectively the top-level plan for test materials production. A sound top-level plan enables the accurate allocation and application of staffing and logistical resources, and the efficient production of quality deliverables that work as intended.

Discussion:

While consideration of a test materials framework might seem to be a detail that belongs only in QA Framework: Test Guidelines [QAF-TEST], in fact such a framework has potential impacts on a WG's operations, processes, staffing, and logistics, and therefore its consideration in the QA Process Document is appropriate.

Examples & Techniques for checkpoint.

Checkpoint 5.2. Ensure test materials are documented and usable for their intended purposes. [Priority 1]
Conformance requirements:
Discussion:

Documentation should be thorough enough that entities unrelated to the test materials' developers -- such as external certification services -- can fully and effectively use the test materials.

Examples & Techniques for checkpoint.

Checkpoint 5.3. Define a contribution process. [Priority 2]
Conformance requirements:
Rationale:

Clear and simple directions for how and to whom to submit materials encourages outside contributions. Explicit instructions on formats and other technical details enables the WG to build processes and procedures for the efficient processing of contributions.

Discussion:

The process document must describe where to submit test materials and whom to notify (e.g., moderator) of a submission. The contribution process describes the format of contributed material, and may contain validation harness, utilities that facilitates tests creation, templates, etc. A well defined submission process is applicable to both internal and external contributors of test materials.

Examples & Techniques for checkpoint.

Checkpoint 5.4. Address license terms for submitted test materials. [Priority 1]
Conformance requirements:
Rationale:

Defining submission license policies in advance will help clarify the WG's expectations to prospective TM submitters, and will facilitate the efficient negotiation of any needed custom submission licenses with submitters.

Discussion:

Unless exempted by custom submission terms with W3C Director's approval, a WG's submission license policies will necessarily conform to standard W3C policies for submitted materials, and specifically those procedures and terms defined in Contribution of Software or Test Materials to W3C [CONTRIB].

Examples & Techniques for checkpoint.

Checkpoint 5.5. Define review procedures for submitted test materials. [Priority 2]
Conformance requirements:
Rationale:

A well-defined and objective review procedure expedites the processing of contributions, and facilitates quick and impartial decisions about their disposition.

Discussion:

These procedures are a follow-on and complement to the contribution process that WGs must define. They should at least include criteria for acceptance or rejection of reviewed tests.

Examples & Techniques for checkpoint.

Guideline 6. Plan test materials publication.

Once the test materials (TM) development is in progress, a Working Group needs to publish the TM drafts and releases, as part of its QA processes.

When the test materials have reached some advanced milestone of maturity and development (e.g., operationally usable), W3C needs to ensure that:

Meeting the needs of TM publication necessarily involves some aspects of TM management, such as repository. These prerequisite aspects are addressed in the following checkpoints, as well as publication details proper.

Checkpoints:

Checkpoint 6.1. Ensure a suitable repository location for test materials. [Priority 1]
Conformance requirements:
Discussion:

This checkpoint does not necessarily require that test materials should physically reside in the W3C. Nevertheless, having the repository at W3C is a recommended way of meeting the checkpoint, because:

  1. At the point that test materials become operationally deployed, challenges to the correctness of both the test materials and specifications normally increase. The Working Group is the best venue for initial processing and adjudication of such queries.
  2. Further to #1, it is more likely that technical report (Recommendation) errata processing and test suite maintenance can be kept synchronized if both responsibilities reside in the same body, i.e., the Working Group.

It is implicit in the "secure" criterion of this checkpoint that, if the Working Group ceases to operate, then the repository arrangements must provide for the continued availability of the test materials.

Examples & Techniques for checkpoint.

Checkpoint 6.2. Define the licenses applicable to published test materials. [Priority 1]
Conformance requirements:
Rationale:

Any W3C-hosted materials must have approved license and use terms. Because there is no single license that is appropriate for all test materials, the WG needs consider the options, select or define a best license for its particular test materials, and clearly inform potential users.

Discussion:

Currently approved W3C licenses that may be applied to test materials are the Document License and the Software License. The Document license has two characteristics that are valuable to test materials:

However, there are situations in which the Document License is inappropriate, because (for example) some Test Materials require modification or completion in order to apply them.

Test Materials may contain any of these three components: test software, test documentation, and test cases. It is possible and sometimes desirable that the WG apply different licenses to different components. If the WG considers that neither the Document nor the Software License is applicable, it should consult with W3C Legal.

Examples & Techniques for checkpoint.

Checkpoint 6.3. Describe how and where the test materials will be published. [Priority 2]
Conformance requirements:
Discussion:

If the test materials are to be published on the W3C site, it is recommended to locate them within the corresponding activity domain. It is strongly recommended not to publish test materials in the TR space for the following reasons:

It is recommended to use one of the practices for publishing test materials, described in Test Guidelines.

Examples & Techniques for checkpoint.

Checkpoint 6.4. Provide a conformance verification disclaimer with the test materials. [Priority 1]
Conformance requirements:
Rationale:

It is common to draw unwarranted conclusions about conformance to the specification from test suite results. A conformance disclaimer clarifies the relationship between test suite results and conformance.

Discussion:

The central principle that a conformance disclaimer for test materials needs to clarify for users of the test materials and consumers of test results is this: passing all tests of the test materials does not guarantee or imply full conformance of an implementation to the specification.

Examples & Techniques for checkpoint.

Checkpoint 6.5. Promote testing and the publication of test results. [Priority 2]
Conformance requirements:
Rationale:

Conformance testing events and publishing test results for implementations promotes interoperability and is helpful to users and vendors alike.

Discussion:

Taking responsibility to publish test results for vendors' products could be a problem for the Working Group. One way to address the problem is for the Working Group to encourage vendors themselves to publish their test results. Such publication should include or describe a test harness that allows reproduction of the results. The Working Group may want to provide the web space to publish collected results.

Examples & Techniques for checkpoint.

Guideline 7. Plan the transfer of test materials to W3C if needed.

A prior checkpoint about test materials repository (Checkpoint 5.2) recommends that test materials reside in the W3C. If test development was done outside by some external entity or group (EG), and the Working Group (WG) together with EG decided to move test materials (TM) to W3C, the following checkpoints define requirements for the transfer process.

Some aspects of the transfer of a whole test suite from outside W3C parallel aspects of in-house planning and development, and piecemeal contribution and review. Those considering such transfers might not already have the planning, development, and submission infrastructure in place. Previous transfer experiences have shown that it is useful to isolate and collect those aspects in one place, as this guideline does.

None of the checkpoints of this guideline are applicable if the Working Group does not transfer test materials from an external entity.

Checkpoints:

Checkpoint 7.1. Perform a quality assessment of any test materials that are candidates for transfer. [Priority 2]
Conformance requirements:

This checkpoint is not applicable if the Working Group does not transfer test materials from an external entity.

Discussion:

The details of any quality assessment are necessarily TM-specific -- it depends the scope & goals of the TM, the type of TM (think "taxonomy"), etc. However, desirable components of a quality assessment would for example include attributes such as clarity of scope, traceability, atomicity, user documentation, etc.

Examples & Techniques for checkpoint.

Checkpoint 7.2. Identify sufficient staff resources to meet the needs of any transferred test materials. [Priority 1]
Conformance requirements:

This checkpoint is not applicable if the Working Group does not transfer test materials from an external entity.

Discussion:

The WG needs to take care that the amount of staff resources is adequate to meet needs of ongoing development and maintenance of the transferred test materials.

Examples & Techniques for checkpoint.

Checkpoint 7.3. For any transferred test materials, resolve all IPR issues with the external party that produced the test materials. [Priority 1]
Conformance requirements:

This checkpoint is not applicable if the Working Group does not transfer test materials from an external entity.

Rationale:

Completion of suitable IPR agreements for any transferred test materials, before the transfer is actually completed, will ensure that the WG can publish and the public can use the test test materials according terms and conditions planned and envisioned by the WG.

Examples & Techniques for checkpoint.

Guideline 8. Plan for test materials maintenance.

The ongoing maintenance of test materials is critical to their long term integrity, and may require a significant commitment of resources. The checkpoints of this guideline address planning and resource commitment requirements to ensure that proper specification tracking happens, and that maintenance and update procedures are provided for.

If the Working Group is not re-chartered, then it is problematic where the resources will come from. In this case, it is up to the Director to determine how the work should be done. Some options include:

Checkpoints:

Checkpoint 8.1. Provide for the long-term maintenance of the contribution and review procedures. [Priority 3]
Conformance requirements:
Discussion:

It is implicit in the criteria of this checkpoint that, if the Working Group ceases to operate, then the WG's maintenance plan must survive and address the ongoing maintenance of the contribution and review procedures.

Examples & Techniques for checkpoint.

Checkpoint 8.2. Specify a test materials update procedure to track new specification versions/errata. [Priority 1]
Conformance requirements:
Rationale:

Defining tracking procedures for specification versions/errata will help to ensure that all needed versions of test materials are available. This is critical to product version tracking of specification versions. It enables products' users to better assess the quality and conformance of the individual product versions.

Discussion:

As used in this checkpoint, the scope of "versions" at least includes "editions" -- editorial revisions that incorporate errata and small changes -- rather than major versions with major functional increments.

Examples & Techniques for checkpoint.

Checkpoint 8.3. Identify a procedure for test validity appeals. [Priority 2]
Conformance requirements:
Rationale:

A well-defined and deterministic procedure for handling challenges will expedite the resolution of challenges, and is critical to their defensibility and acceptance.

Discussion:

Such a procedure is applicable both to external challenges once test materials have been published, as well as to internal challenges during the building of test materials.

Examples & Techniques for checkpoint.

3. WG relationship to QA Activity

The QA Activity works closely with other W3C WGs, providing assistance and expertise in helping them achieve their QA goals and deliverables. The QA Activity anticipates different relationships with the various WGs depending on the QA-specific needs of the particular Working Group. For example, the resources and experience of Working Group members as well as their stage in the Recommendation track may influence the type and level of collaboration between the Working Group and QA Activity. Potential relationships between Working Groups and the QA Activity include:

More details about the ways in which the QA Activity may interact with and assist the W3C WGs may be found in the QA Working Group Process Document [QAWGPD].

4. Conformance

This section defines conformance of Working Group processes and operations to the requirements of this specification. The requirements of this specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification, and apply to the Working Group QA-related documents and deliverables required by this specification.

4.1. Normative sections

The following parts of this specification are normative:

Text that is designated as normative is directly applicable to achieving and assessing conformance to this document. Informative parts of this specification consist of examples, extended explanations, terminology, and other information that facilitates correct implementation of this specification.

4.2. Extensibility

This specification is extensible. That is, the Working Groups MAY set up quality-related processes and operations in addition to those required for conformance to this specification. Extensions to this specification MUST NOT contradict or negate the requirements of this specification.

The rationale for allowing Working Groups to define extensions to these operational guidelines is that these requirements are considered to be the minimal requirements for successful quality practices within the WGs. Doing more than the minimum is not only acceptable, but beneficial. Extensions also allow Working Groups tailor their quality practices more closely to their specific needs. The guidelines of this specification may not be sufficient to meet the needs of all WGs.

4.3. Conformance requirements & test assertions

Each checkpoint contains one or more conformance requirements. These are prefaced by "Conformance requirements:", highlighted in a different style (on appropriate media), and contain at least one of the RFC 2119 [RFC2119] keywords (MUST, SHOULD, MAY, etc).

This Operational Guidelines document does not enumerate a list of test assertions. Test assertions can be derived conformance requirements in a straightforward manner.

4.4. Checkpoint priority definitions

Each checkpoint has a priority level based on the checkpoint's impact on the quality practices of a Working Group. The meanings of the priorities are:

[Priority 1]
Critical/essential. These checkpoints are considered to be basic requirements for ensuring that test materials are usable, and are produced in time to ensure the quality of the standard and its implementations. Satisfying these checkpoints is a basic requirement to ensure quality and interoperability of the standard.
[Priority 2]
Important/desirable. Satisfying these checkpoints, in addition to the priority 1 checkpoints, should significantly improve the usability and timeliness of the test materials, as well as the quality of the standard and its implementations.
[Priority 3]
Useful/beneficial. Satisfying these checkpoints, on top of all the others, will further improve the quality, usability, and timeliness of the test materials and the standard itself.

4.5. Conformance definition

This section defines three conformance levels to this guidelines specification:

This three-level conformance scheme (A, AA, AAA) was chosen in order to facilitate progressive (good-better-best) pursuit of the goals and benefits of these guidelines.

A checkpoint is satisfied by satisfying all of the individual mandatory conformance requirements. Mandatory requirements are those that use the conformance keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", or "SHALL NOT".

Note that it is allowed (even encouraged) to implement checkpoints in addition to those required to satisfy a particular degree of conformance (A, AA, or AAA). The more checkpoints that are satisfied, the better. However, there is no additional conformance designation for such intermediate collections of checkpoints (i.e., for all checkpoints of a given level plus some, but not all, of the checkpoints of the next levels).

The minimum recommended conformance to this specification is A-conforming -- all Priority 1 (critical/essential) checkpoints satisfied.

4.6. Conformance claims

An assertion of conformance to this specification -- i.e., a conformance claim -- MUST specify:

The checklist for this specification ([OPS-CHECKLIST]) is the Implementation Conformance Statement (ICS) pro-forma for this specification. Any assertion of conformance to this specification MUST link to a completed ICS.

Example of a valid conformance claim:

The QA processes and operations of this FooBar Working Group, http://example.org/FooBar/, conform to W3C's 'QA Framework: Operational Guidelines', available at http://www.w3.org/TR/2003/WD-qaframe-ops-200302110/, on 23-April-2003, AA-Conforming. The Implementation Conformance Statement for this claim is available at http://example.org/FooBar/Test/OpsGL-ICS .

4.7. Conformance disclaimer

The checkpoints of this specification present verifiable conformance requirements about the operational aspects of Working Group quality processes. As with any verifiable conformance requirements, users should be aware of this disclaimer:

Passing all of the requirements to achieve a given conformance degree -- A, AA, or AAA -- does not guarantee that the subject operations and processes are well-suited to or will achieve their intended purposes, nor does it guarantee the quality or suitability of test materials produced under the processes.


5. Acknowledgments

The following QA Working Group and Interest Group participants have contributed significantly to the content of this document:

6. References

6.1. Normative references

[PROCESS]
World Wide Web Consortium Process Document, I. Jacobs, Ed., 19 July 2001, available at http://www.w3.org/Consortium/Process-20010719/ .
[RFC2119]
Key words for use in RFCs to Indicate Requirement Levels, March 1997, available at http://www.ietf.org/rfc/rfc2119.txt .

6.2. Informative references

[CONTRIB]
Contribution of Software or Test Materials to W3C, which defines W3C-approved procedures and terms for submission of Software and Test Materials, available at http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231 .
[EXTERN-TA]
QA activity email thread about third-party participation in test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html .
[MATRIX]
W3C-wide conformance activity survey covering all the Working Groups, "The Matrix", available at http://www.w3.org/QA/TheMatrix .
[TAXONOMY]
QA Activity test taxonomy, a classification scheme for conformance test materials, available at http://www.w3.org/QA/Taxonomy .
[OPS-CHECKLIST]
Appendix to this operational guidelines document that presents all checkpoints in a table sorted by guideline. Available at http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/qaframe-ops-checklist .
[OPS-ICS]
Appendix to this operational guidelines document that presents all checkpoints in a table sorted by Priority. Available at http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/qaframe-ops-ics .
[OPS-EXTECH]
QA Framework: Operational Examples & Techniques, L. Henderson, L. Rosenthal, K. Gavrylyuk, D. Dimitriadis, Eds., W3C Note, (initially) 10 February 2003, companion version to this document, latest companion version available at http://www.w3.org/QA/WG/2003/02/qaframe-ops-extech .
[PROTOCOL-WG-TS]
The XML Protocol WG has a TS process document, available at http://www.w3.org/2000/xp/Group/1/10/ts-contribution, and a contribution/submission license (example of a submission legal notice), available at http://www.w3.org/2001/10/test-materials-license.html .
[QA-GLOSSARY]
A comprehensive glossary of QA terms, maintained by the QA Working Group. (Initial version under construction.)
[QAF-INTRO]
QA Framework: Introduction, L. Henderson, K. Gavrylyuk, D. Dimitriadis, L. Rosenthal, Eds., W3C Working Draft, 10 February 2003, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/ .
[QAF-SPEC]
QA Framework: Specification Guidelines, D. HazaŽl-Massieux, L. Henderson, L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 08 November 2002, companion version to this document, available at http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/ .
[QAF-TEST]
QA Framework: Test Guidelines, K. Gavrylyuk, D. Dimitriadis, L. Henderson, M. Skall, P. Fawcett, Eds, W3C Working Draft, 20 December 2002, companion version to this document, available at http://www.w3.org/TR/2002/WD-qaframe-test-20021220/ .
[QAIG]
Quality Assurance Interest Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/IG/ .
[QAWG]
Quality Assurance Working Group of the W3C QA Activity, which may be found at http://www.w3.org/QA/WG/ .
[QAWGPD]
QA Working Group Process Document, defining both the general and QA-specific processes of the QA Working Group, available at (under construction).
[DOM Working Group TS]
DOM TS Process Document, D. Dimitriadis, Ed., 15 January 2002, available at http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115 .
[REC-TRACK]
Stages and milestones in the W3C Recommendation Track, per the Process Document (Process Document is available at http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
[SCHEMA-WG-TS]
The XML Schema WG has a TS process document, available at http://www.w3.org/2001/05/xmlschema-test-collection.html, and a contribution/submission license (example of a submission legal notice), available at http://www.w3.org/2001/05/test-materials-license.html .
[WCAG10]
Web Content Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G. Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at http://www.w3.org/TR/WCAG10/ .
[WG-QA-RANGE]
Email proposal by David Marston, on the QA public mail list, for range of Working Group commitment levels to conformance test materials production, available at http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html .

7. Change history

2003-05-14, WG-discussion Working Draft

Clarifications and improvements in response to Last Call comments. Checkpoint numbers, CPx.y, refer to the numbers in the Last Call WD of Operational Guidelines

2003-02-10, Last Call Working Draft

Revised Introduction, Conformance chapters for better conformance to SpecGL. Restructure all checkpoints for better verifiability. Add rationale to many checkpoints. Implement resolution of all closed OpsGL issues. Editorial improvements.

2002-11-08, third published WD

Revised Introduction, Conformance for better conformance to SpecGL. Implemented other minor editorial and resolved-issue changes. Make over all checkpoints for better verifiability.

2002-05-15, second published WD

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

[1] http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops

03-11-2002

Checkpoints numbering is changed, changes list refers to the numbering from the previous WD [1].

[1] http://www.w3.org/QA/WG/2002/02/qaframe-ops-0225.html

02-28-2002
Major rewrite of the previous draft. Incorporated Lynne's comments about Gd1 and Gd4, split Gd regarding QA process, added bunch of new checkpoints.