This document defines a common Operational Framework for building
conformance test materials for W3C specifications. It presents
operational and procedural guidelines for groups undertaking conformance
materials development. This document is one in a family of Framework
documents of the Quality Assurance (QA) Activity, which includes the
other existing or in-progress specifications: Introduction; Specification
Guidelines; and, Test Guidelines.
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. The latest version
of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity statement.
This is the third public Working Draft, and is significantly changed since
the Previous Version. The Previous Version incorporated the discussions from
several subsequent QA Working Group
[QAWG] teleconference
and face-to-face meetings. Previously the QAWG examined, debated, and agreed upon
the individual checkpoints and their priorities. In this version, the
individual checkpoints have been restructured so that their test assertions
-- i.e., the criteria for fulfillment of the checkpoints -- are isolated and
identified. For details, please see Change
history.
One consequence of making all checkpoints verifiable has been to expose
some underlying hidden issues in how the checkpoints were previously written.
These have not all been addressed by the QA Working Group yet. They will be
addressed and resolved before the next public version of this document.
This version supersedes all previous drafts. It is expected that updated
versions of this document will be produced regularly, along with other
members of the Framework documents family. It is anticipated that the next
public version of this document will be the Last Call version (estimated,
February 2003).
Future progression of this document beyond Working Draft is planned, but
the final status has not been determined at this time. See QA Working Group
issue #18 and
issue #71.
This part of the Framework document family has an accompanying "QA
Framework: Operational Examples & Techniques". As of this version, the
status of that informative companion document has been changed from Working
Draft to Note. At least until this document stabilizes, the Examples &
Techniques companion will be maintained and frequently updated in QA Working
Group Web space (as opposed to /TR/).
The QA Working Group
Patent Disclosure page contains details on known patents related to this
specification, in conformance with W3C policy
requirements.
Please send comments to www-qa@w3.org,
the publicly
archived list of the QA Interest
Group
[QAIG]. Please note
that any mail sent to this list will be publicly archived
and available, do not send information you would not want to see distributed,
such as private data.
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or obsoleted by other documents at any time. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress".
A list of current W3C Recommendations and other technical documents can be
found at http://www.w3.org/TR/.
1. Introduction
1.1 Scope and goals
1.2 Goals of QA development
1.3 Motivation and expected benefits
1.4 Relationship to other specifications
1.5 Understanding and using this document
1.6 Checkpoint priorities
1.7 Terminology
2. Guidelines
G 1. Integrate Quality Assurance into Working Group
activities.
G 2. Define resources for Working Group QA
activities.
G 3. Synchronize QA activities with the milestones for
WG deliverables.
G 4. Define the QA process.
G 5. Plan
test materials development.
G 6. Plan
test materials publication.
G 7. Plan the transfer of test materials to W3C if
needed.
G 8. Plan
for test materials maintenance.
3. WG relationship to QA Activity
3.1 Liaison and Consultation
3.2 Active Reviews
3.3 Adjudicate entry and exit criteria
3.4 QA resource supplement
3.5 Resolution of external QA requests
4. Conformance
4.1 Test assertions
4.2 Conformance definition
4.3 Conformance disclaimer
5. Acknowledgments
6. References
7. Change history
An appendix to this document [OPS-CHECKLIST]
presents all checkpoints in a tabular
form, for convenient reference. This checklist is an Implementation
Conformance Statement (ICS) pro-forma for
this specification.
1. Introduction
1.1 Scope and goals
This document describes the process and operational activities for
building conformance test suites and tools for W3C specifications. The
primary goal is to assist the W3C Working Groups (WGs) by providing guidelines for the
planning, development, deployment, and maintenance of conformance test
materials.
In this guidelines document, the term "conformance test materials" refers
conformance test suites, validation tools, conformance checklists, and any
other materials that are used to check or indicate conformance.
The intended audience for these guidelines is all W3C Working Group
members, as well as the actual developers of conformance materials for W3C
specifications.
While it is optimal that sound quality practices are integrated into WG
activities from the very beginning, these guidelines are intended for newly
chartered and existing Working Groups alike. Working Groups who may already
be doing some of these activities should review the document and incorporate
the principles and guidelines into their quality operations as much as
possible.
Not only are the Working Groups the consumers of these guidelines, they
are also key contributors. It is the aim of these guidelines to capture the
experiences, good practices, activities, and lessons learned of the Working
Groups and present them in a comprehensive, cohesive set of documents for all
to use and benefit from. These guidelines aim to standardize the best of
current practice.
The goal is to enable the WGs to reuse what works rather than having to
reinvent, and to foster consistency across the various Working Group quality
activities and deliverables.
1.2 Goals of QA development
Common frameworks for W3C quality practices are shaped by the high-level
goals for development and deployment of test materials. The goals for the
operational framework guidelines share those high-level goals.
The over-arching goal is stated in the QA Activity Statement:
- to enable W3C Web standards to achieve full interoperability and access
to all, by addressing the quality and correctness of implementations.
Specific goals of test materials development include:
- provide self-assessment tools and materials for implementation
builders;
- help implementation builders to achieve interoperability of
implementations;
- help users to verify conformance of implementations with the stated
conformance requirements of their respective specifications;
The goals of QA work within a Working Group do not encompass:
- measurement of performance, scalability of implementations, or their
assessment against any other "goodness" criteria, except what is stated
as a conformance requirement of their respective specification.
- certification of implementations
While certification may have a positive quality assurance role in
standards-based Web environments, W3C does not presently intend to initiate
or offer any certification services.
1.3 Motivation and expected benefits
As the complexity of W3C specifications and their interdependencies
increase, quality assurance becomes even more important to ensuring their
acceptance and deployment in the market. There has been a growing awareness
and interest in conformance and quality. In approving and initiating the QA
Activity, W3C has endorsed the principle that in order for W3C Web standards
to achieve full interoperability and access to all, the quality of the
implementation must be given as much attention as the standards' development.
The principal factor for improving the quality of implementation is early
availability of conformance test materials.
Although not explicitly stated, the W3C Process Document supports the
development of conformance test materials.
[...] groups may produce technical reports, review the work of other
groups, develop sample code or test suites, etc." (see Process
Document, section 3.)
W3C should make every effort to maintain its Recommendations (e.g., by
tracking errata, providing test bed applications, helping to create test
suites, etc.) (see Process
Document, section 5.2.5, "Ongoing work".)
In an effort to meet these suggestions and address the implementation
requirements of the Process Document, some Working Groups have included the
development of conformance materials as part of their CR-exit and PR-entrance criteria. Examples
include Scalable Vector Graphics (SVG), User Agent Accessibility
Guidelines (UAAG), and Extensible
Style Language - Formatting Objects (XSL-FO).
This makes sense, since it is natural for test suites and implementations to
develop in parallel - each is a help to the development of the other.
There is already a body of contemporary QA experience and activity amongst
the Working Groups. The Matrix identifies more than a score of test suites
and validators, in various states of development. Moreover, many Working
Groups have already established procedures, techniques and tools for
developing test materials (e.g., Document Object Model - DOM). It makes sense to capitalize on
what has already been done and share that with those who are starting out and
those who are already in the process of developing conformance materials.
1.4 Relationship to other specifications
This document is part of a family of QA Framework documents designed to
help the WGs improve all aspects of their quality practices. by solidifying
and extending current quality practices within the W3C. The QA Framework
documents are:
The QA Framework documents are interrelated and complement each other. For
example, there is a close relationship between the processes for dealing with
versions and errata, and the test framework architecture for handling
multiple versions and errata levels. Hence there is interrelationship between
this document and the Test Guidelines. The reader is strongly encouraged to
be familiar with the other documents in the family.
1.5 Understanding and using this document
The Guidelines in the document are organized chronologically. The document
starts with the guidelines that are applicable as early as the formation of a
Working Group (e.g., charter considerations), and continues through the
various process and operational activities necessary in planning, developing,
deploying and maintaining conformance materials.
This document employs the WAI (Web Accessibility
Initiative) model for representing guidelines or general principles for the
development of conformance materials. See, for example, Web
Content Accessibility Guidelines. Each guideline includes:
- the guideline number,
- the statement of the guideline,
- the rationale behind the guideline,
- a collection of one or more checkpoints.
The checkpoint definitions in each guideline define the processes and
operations that need to be implemented in order to accomplish the guideline.
Each checkpoint definition includes:
- the checkpoint number
- the statement of the checkpoint (its "title"),
- the priority of the checkpoint,
- a statement of how to fulfill the checkpoint,
- optional informative notes, clarifying examples, and cross references
to related guidelines or checkpoints,
- a link to the corresponding section in Operational Examples &
Techniques.
Each checkpoint is intended to be specific enough so that someone can
implement the checkpoint as well as verify that the checkpoint has been
satisfied. A checkpoint will contain at least one, and may contain multiple
individual requirements, that use RFC2119 normative keywords. See the "Conformance" chapter for further discussion of
requirements and test assertions.
A separate appendix to this document [OPS-CHECKLIST] presents all checkpoints in a tabular form, for
convenient reference. The checklist is an Implementation Conformance
Statement (ICS) pro-forma for
this specification.
Some checkpoints are more critical than others for the timely production
of high-quality,highly usable test materials. Therefore each checkpoint has a
priority level assigned by the QA Working Group based on the checkpoint's
impact on the quality and timing of the test materials produced by a Working
Group.
-
[Priority 1]
- Critical/essential. These checkpoints are considered to be basic
requirements for ensuring that test materials are usable, and are
produced in time to ensure the quality of the standard and its
implementations. Satisfying these checkpoints is a basic requirement to
ensure quality and interoperability of the standard.
-
[Priority 2]
- Important/desirable. Satisfying these checkpoints, in addition to the
priority 1 checkpoints, should significantly improve the usability and
timeliness of the test materials, as well as the quality of the
standard and its implementations.
-
[Priority 3]
- Useful/beneficial. Satisfying these checkpoints, on top of all the
others, will further improve the quality, usability, and timeliness of
the test materials and the standard itself.
1.7 Terminology
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" will be used as
defined in RFC 2119 [RFC2119]. When used with the
normative RFC2119 meanings, they will be all upper case. Occurrences of these
words in lower case comprise normal prose usage, with no normative
implications.
Unusual terms in these framework documents are defined when first used,
and most generally useful QA-specific terms will eventually be in the QA
Glossary [QA-GLOSSARY].
Guideline 1. Integrate Quality Assurance into Working Group
activities.
In approving and initiating the QA Activity, W3C has endorsed the
principle that in order for W3C Web standards to achieve full
interoperability and access to all, the quality of implementation must be
given as much attention as the standards' development. A principal
contributor to the quality of implementations is early availability of
conformance test materials (TM).
Therefore, conformance test materials -- test suites and tools -- must be
considered as Working Group deliverables the same as the standards
themselves. Accordingly, Working Groups must make explicit commitment to QA
at a suitable level, as well as identify specific QA deliverables,
milestones, and dependencies with other WG deliverables.
New Working Groups -- i.e., those that are writing and submitting their
Charters for approval -- are required to make the specified commitments in
their charters. Working Groups that are renewing their charters are
considered the same as new WGs. Existing Working Groups could satisfy the
checkpoints by amendment to their charter, or in other ways, the latter to be
specified in
QA Framework: Examples
& Techniques
[OPS-EXTECH].
Checkpoints:
Checkpoint 1.1. Commit to at least
"QA
level three".
[Priority 1]
To fulfill this checkpoint, a new or rechartering Working Group
MUST commit, in its Charter, to at least QA level
three. An existing Working Group MUST document a consensus commitment to
at least QA level three in one of various other
ways.
Discussion. Detailed planning and specification of test materials is
inappropriate in the charter because it inflexibly binds the Working Group to
test materials implementation specifics for which flexibility may be needed.
There are numerous possible levels of commitment that a Working Group could
make based on anticipated Working Group resources, existence of outside
efforts, and the Working Group's current stage in the process [REC-TRACK].
Based on the level of
testability of the specification and the extent of test materials, a range of
possible commitments has been identified.
| Level |
testability of specification |
extent of test materials |
| 1. |
Working Group plans regular specification reviews for testability,
following Specification Guidelines
|
|
| 2. |
In addition to the regular specifications reviews mentioned in the
previous level, Working Group commits to develop a set of test
assertions, not necessarily complete, before beginning development of
a test suite. |
Working Group commits that test suite for the specification, not
necessarily complete and thorough, will exist before specification
becomes Recommendation. |
|
3. |
In addition to regular specification reviews, a Working Group aims
to have numerous normative use cases in the body of the
Recommendation. |
In addition to the commitment of the previous level, a Working
Group ensures that the test suite produced is reviewed by the Working
Group or externally and the review process meets the criteria established in this
document. |
| 4. |
Same as above. |
Working Group commits to establish and maintain the test cases
contribution and review process and will produce and review the test
suite, not necessarily complete and thorough, before the
specification becomes Recommendation. |
|
5. |
In addition to the commitments from the previous level, Working
Group will derive the list of testable assertions as an addendum to
specification by the time it becomes Recommendation. |
Same as above |
| 6. |
Same as above |
In addition to the commitment for the previous level, a Working
Group insists on a complete test suite before corresponding standard
becomes Recommendation. |
|
7. |
Same as above |
In addition to the commitment for the previous level, Working Group
plans to maintain the test suite even after Specification becomes
recommendation. Working Group plans to support versions and errata in
the test suite and review appeals. |
This seven-point enumeration is derived from the proposal to the QA mail list, after the 4/2001 QA
Workshop.
The further down the scale (i.e., the higher the number) that the Working
Group commits, the better it is for achieving W3C's ultimate goals of
interoperability and accessibility.
For existing working groups that make the QA commitment after chartering,
various ways to do it are discussed in the Examples & Techniques
document.
Examples &
Techniques for checkpoint.
Checkpoint 1.2. Commit to at least
"QA
level five".
[Priority 2]
To fulfill this checkpoint, a new or rechartering Working Group
MUST commit, in its Charter, to at least QA level
five. An existing Working Group MUST document a consensus commitment to
at least QA level five in one of various other
ways.
Discussion. This checkpoint advances the level of QA commitment required
by the previous checkpoint and therefore supersedes it. Satisfying this level
of requirements for the specification and the test materials significantly
increases confidence in the interoperability of the standard.
For existing working groups that make the QA commitment after chartering,
various ways to do it are discussed in the Examples & Techniques
document.
Examples &
Techniques for checkpoint.
Checkpoint 1.3. Commit to at least
"QA
level seven".
[Priority 3]
To fulfill this checkpoint, a new or rechartering Working Group
MUST commit, in its Charter, to at least QA level
seven. An existing Working Group MUST document a consensus commitment to
at least QA level seven in one of various other
ways.
Discussion. This checkpoint advances the level of QA commitment required
by the previous checkpoint and therefore supersedes it. Satisfying this level
of requirements for the specification and the test materials further
increases confidence in the interoperability of the standard.
For existing working groups that make the QA commitment after chartering,
various ways to do it are discussed in the Examples & Techniques
document.
Examples &
Techniques for checkpoint.
Checkpoint 1.4. Enumerate QA deliverables and expected
milestones.
[Priority 1]
To fulfill this checkpoint, a new or rechartering Working Group
MUST document the QA deliverables and milestones its Charter. An existing
Working Group MUST document QA deliverables and milestones in some consensus
record, in one of various other ways.
Discussion. The W3C Process Document requires for
Charters, that deliverables be identified with milestones. It is vital that
the milestones for QA deliverables are synchronized and even serve as
criteria for WG technical deliverables (specifications).
Examples of QA deliverables include sets of use cases, test suites
(produced or acquired), validators, test harnesses.
Examples & Techniques
for checkpoint.
Checkpoint 1.5. Associate QA criteria with WG
milestones.
[Priority 2]
To fulfill this checkpoint, a new or rechartering Working Group
MUST, in its charter, specify the QA criteria required for the attainment of
major WG milestones. An existing Working Group MUST define the association in
some consensus record, in one of various other ways.
Discussion. A de facto convention of test suite completion before entrance
into PR phase (and by implication, completion of CR phase, if there was a CR phase) seems to be a common
process goal amongst a number of the existing test suite efforts. For example, SVG, UAAG, and others have
defined this criterion. It is natural for test suites and implementations to
develop in parallel -- each is a help to the development of the other. And,
the Process Document does address implementation
requirement:
Advancement of a technical report to Candidate Recommendation is an
explicit call for implementation experience to those outside of the related
Working Groups or the W3C itself." (see Process
Document, section 5.2, about "Candidate Recommendation.)
Preferably, the Working Group should be able to demonstrate two
interoperable implementations of each feature. (see Process
Document, section 5.2.4, about "Entrance Criteria".)
Examples &
Techniques for checkpoint.
Guideline 2. Define resources for Working Group QA
activities.
It is highly beneficial to the success of the WG's QA work that it address
QA resource requirements from the beginning of the Working Group formation.
Starting from the Working Group Charter and later in the Call For
Participation special attention is required to the staffing and other
resource requirements for successful QA work.
The sooner the resources are defined and committed, the better. Therefore
new Working Groups -- i.e., those that are writing and submitting their
Charters for approval -- are required to have the resource commitments in
their charters. Working Groups whose charters are being renewed are
considered the same as new WGs. While "sooner is better", existing Working
Groups can still satisfy the checkpoints, in other ways to be specified in
QA Framework: Examples &
Techniques
[OPS-EXTECH].
Checkpoint 2.1. Address where and how conformance test materials
will be produced.
[Priority 1]
To fulfill this checkpoint, a new or rechartering Working Group
MUST, in its charter, specify who will produce the test materials and how. An
existing Working Group MUST define the association in some consensus record,
in one of various other ways.
Discussion. Test materials can be produced by a range of possible
scenarios, from all intra-WG effort concurrent with other WG deliverables, to
importing completed materials from an external group.
(Ed note. Example/technique here was moved to Operational Examples
& Techniques.)
Examples & Techniques
for checkpoint.
Checkpoint 2.2. Address QA staffing commitments.
[Priority 1]
To fulfill this checkpoint, a new or rechartering Working Group
MUST, in its charter, allocate WG staffing resources commensurate with the QA level and where-and-how plan. An existing Working Group
MUST define the allocation in some consensus record, in one of various other
ways.
Discussion. There will be at least some staffing required from the Working
Group, for any of the acceptable options presented so far. Depending upon the
general intent and plan, and how the test suite will be built, the commitment
can range from minimal to significant.
Examples &
Techniques for checkpoint.
Checkpoint 2.3. Request allocation of QA resources to the
Working Group.
[Priority 1]
To fulfill this checkpoint, a new or rechartering Working Group
MUST, in its Call for Participation, request that participating members
allocate some staffing resources specifically for QA work. An existing
Working Group MUST must make external appeal for QA-specific resources in one
of various other ways.
Discussion. Once the Charter is prepared, the Director sends a Call for
Participation to the Advisory Committee. At this point AC Representatives are
asked to provide information about amount and type of resources their
organization plans to allocate for the particular Working Group. Explicit
indication of resources dedicated for Quality Assurance is required to assess
Working Group capabilities against deliverables and milestones declared in
the Charter.
Examples &
Techniques for checkpoint.
Guideline 3. Synchronize QA activities with the milestones for
WG deliverables.
The benefits of starting synchronization of the specification and test
materials development as early as possible include:
- it gives extensive list of use cases;
- it identifies ambiguities in the specification at the early stages,
which saves cycles in the late phases;
- it provides clear set of testable assertions -- skeleton of the
specification -- which in its turn facilitates development of
interoperable implementations. The latter is a W3C process criterion for
entering the Candidate Recommendation phase.
Checkpoints:
Checkpoint 3.1. Synchronize the publication of QA deliverables and
the specification's drafts.
[Priority 2]
To fulfill this checkpoint, the Working Group MUST publish QA
deliverables concurrently with each Working Group specification publication
milestone.
Discussion. The checkpoint above
required associating QA criteria with each Working Group milestone. The
Working Group milestones are usually bound to the WG's specifications'
Recommendation track milestones. Each version of the specification --
WDs, CR(s), PR, etc -- is a changed
document that requires an update of all dependencies like test materials. A
natural and desirable QA criterion for such a milestone would be publication
of the updated test materials or other QA deliverables.
Examples of QA deliverables might range from a TS (test suite) schedule in early WDs, to TS design document in later WDs, to a first public TS release at CR.
Examples & Techniques
for checkpoint.
Checkpoint 3.2. Support specification versioning/errata in the
QA deliverables.
[Priority 3]
To satisfy this checkpoint, the Working Group SHOULD plan for
specification versioning/errata support in its QA deliverables, and MUST
ensure that the final published test materials support specification
versioning/errata.
Discussion. Because there will be products that support different versions
or even different errata levels of the specification, users must be able to
verify product's conformance with a specific version or errata level. This
checkpoint specifically extends the previous one to cover specification
milestones after Recommendation status. Whether or not a Working Group plans to maintain the
test suite after the specification becomes Recommendation (see the Table of WG Commitment Levels), it has to ensure
support of versioning/errata in the structure of the test materials.
Examples &
Techniques for checkpoint.
Guideline 4. Define the QA process.
A Working Group QA process encompasses all aspects of QA life within the
Working Group, including:
- a QA leader -- the moderator;
- a task force commensurate with QA commitment;
- producing the QA Process Document (QAPD);
- communications;
- development framework;
- and policies such as branding.
Documented examples of the QA process can be found at DOM TS process,
XML Schema TS process (@@link tbd)
, and XML Protocol TS process (@@link tbd) .
Checkpoints:
Checkpoint 4.1. Appoint a QA moderator.
[Priority 1]
To fulfill this checkpoint, the Working Group MUST identify a
person to manage the WG's quality practices.
Discussion. Before starting test materials development, a Working Group
has to appoint several key participants for its QA process. The QA moderator
is the overall manager of all of the QA activities in the Working Group.
Depending on the plan
for the origin of the test suite a Working Group can:
- appoint a test moderator from the Working Group members (and other W3C
members);
- W3C QA staff;
- invite moderator from external organization.
When a test suite is developed outside of the Working Group, a moderator
often already exists in the external group that is the source of the test
suite. It is recommended to invite the moderator to become a Working Group
member, since he/she needs access to all Working Group materials and
resources. See also section on transferring test suite from outside
W3C.
Examples & Techniques
for checkpoint.
Checkpoint 4.2. Appoint a QA task force.
[Priority 2]
To fulfill this checkpoint, the Working Group MUST identify a
QA task force of size commensurate with the QA commitment level and agreed QA
deliverables.
Discussion. Depending on the level of involvement of a Working Group in
the test materials development, a Working Group may need to appoint QA task
force in addition to the test moderator. Such a task force can take
responsibility for a QA framework development, test materials development,
review of contributions, maintenance. See later checkpoints for details on
each of these activities.
Examples &
Techniques for checkpoint.
Checkpoint 4.3. Produce the QA Process Document.
[Priority 1]
To fulfill this checkpoint, the Working Group MUST produce a QA Process
Document which minimally describes:
- test materials contribution and review process;
- licenses applied to test materials;
- test materials publication rules;
- test materials maintenance rules;
- content as required by other checkpoints in this guideline.
Examples &
Techniques for checkpoint.
Checkpoint 4.4. Specify means for QA-related
communication.
[Priority 2]
To fulfill this checkpoint, the WG's QA Process Document MUST
specify at least one mailing list for QA announcements and comments, plus a "Test" Web page.
Discussion. The Working Group needs an archived mailing list for all
comments, announcements, and discussion. . The WG also needs a
"Test" Web page for posting announcements, QA deliverables, test materials
releases, etc.
Examples &
Techniques for checkpoint.
Checkpoint 4.5. Define the QA framework for test materials
development.
[Priority 1]
To fulfill this checkpoint, the WG's QA Process Document MUST
define the framework that will be used for test materials development.
Discussion. The QA framework describes how to
develop, document and use the tests.
Examples &
Techniques for checkpoint.
Checkpoint 4.6. Specify a policy for branding
materials.
[Priority 3]
To fulfill this checkpoint, the WG MUST document in its QA
Process Document whether or not branding will be supported by the WG, and
MUST document the branding details if branding is to be supported.
Discussion. Some W3C activities support
"branding". Branding refers to the use of a conformance icon to indicate
some level of standard conformance. Examples: HTML validation, CSS
validation, WCAG conformance. While these content branding schemes are
relatively non-controversial, there are nevertheless issues that should be
addressed -- particularly in the contexts of user-agent and API conformance
-- before any branding-related goals are articulated.
Examples &
Techniques for checkpoint.
Guideline 5. Plan
test materials development.
As described earlier, a Working Group may have different levels of
involvement in test materials development. Nevertheless at any level a
Working Group needs to have clear understanding of what QA framework it will use and how to insure the
quality and usability of the test materials themselves. As a part of the WG's
QA process, before starting a test materials development, it is recommended
the test materials implementer (Working Group or 3rd party) looks at Test Guidelines and decide what approach to take for
test materials organization, test criteria, etc.
Checkpoints:
Checkpoint 5.1. Ensure test materials are documented and
usable for their intended purposes.
[Priority 1]
To fulfill this checkpoint, the WG MUST ensure that the test
framework contains user documentation that fully instructs the use of the
test materials for the full range of their intended purposes.
Discussion. While W3C does not have any plans to offer any certification
services, developed test materials can be used by external parties including
certification services. Factors of rigor, objectivity, and defensibility are
key determinants of any test materials.
Examples &
Techniques for checkpoint.
Checkpoint 5.2. Define a contribution process.
[Priority 2]
To fulfill this checkpoint, the WG MUST describe in its QA
Process Document where, how, by whom, and to whom test materials submissions
are to be made.
Discussion. The process document must describe where to submit test
materials and whom to notify (e.g., moderator) of a submission. The
contribution process describes the format of contributed material, it may
contain validation harness, utilities that facilitates tests creation,
templates, etc.
Examples & Techniques
for checkpoint.
Checkpoint 5.3. Define the licenses applicable to submitted
test materials.
[Priority 1]
To fulfill this checkpoint, the WG MUST define in its QA
Process Document at least one license agreement that is acceptable to the WG
and under which external parties may submit test materials.
Describe. A Working Group needs to publish a license agreement acceptable
to the WG, under which vendors and other external parties may submit test
materials. Examples of such license agreements may be found at (@@links tbd). Contributors may
have to negotiate license agreements for their specific needs.
Examples &
Techniques for checkpoint.
Checkpoint 5.4. Define review procedures for submitted test
materials.
[Priority 2]
To fulfill this checkpoint, the WG MUST define in its QA
Process Document a procedure for reviewing test materials contributions, and
the procedure MUST minimally address accuracy, scope, and clarity of the
tests.
Discussion. Once the contribution process is defined, the Working Group needs procedures
for reviewing the tests being contributed, including criteria for acceptance
or rejection of reviewed tests.
(Ed note. Example/technique here was moved to Operational Examples
& Techniques.)
Examples &
Techniques for checkpoint.
Guideline 6. Plan
test materials publication.
Once the test suite development is in progress, a Working Group needs to
publish the test suite drafts and releases, as part of its QA processes.
Checkpoints
Checkpoint 6.1. Ensure a suitable repository location for test
materials.
[Priority 1]
To fulfill this checkpoint, the Working Group MUST ensure that
test materials have a repository locations that is secure, reliable, and
freely accessible.
Discussion. Once the materials have reached some advanced milestone of
maturity and development (e.g., operationally usable), W3C needs to assure
that:
- the test materials have proper maintenance and support;
- tests materials are freely available for download;
- if a license is applied, it does not preclude the test suite from being
used by all interested parties;
- the test materials' quality and usability is appropriate.
This checkpoint does not necessarily require that test suite/tools should
physically reside in the W3C. Nevertheless, having the repository at W3C is a
recommended way of meeting the checkpoint, because:
- At the point that test materials become operationally deployed, then
challenges to the correctness of both the test materials and
specifications normally increase, and the Working Group is the best venue
for initial processing and adjudication of such queries against both.
- Further to #1, it is more likely that technical report (Recommendation)
errata processing and test suite maintenance can be kept synchronized if
both responsibilities reside in the same body, the Working Group.
If the
Working Group ceases to operate, then successors responsible for ongoing
interpretation and maintenance (errata) of the standard should be designated
to ensure continued availability of the test suite.
Examples & Techniques
for checkpoint.
Checkpoint 6.2. Define the licenses applicable to
published test materials.
[Priority 1]
To fulfill this checkpoint, the Working Group MUST define in
its QA Process Document the licenses that are applicable to published test
materials.
Discussion. If test materials are produced within the W3C, the Working
Group is recommended to use the W3C Document License. The Document License is
preferred because:
- It prohibits the test materials from being modified by the user upon
download. Therefore it guarantees that the test materials can not be
misused during conformance evaluation of a product.
- It requires that if the test materials are copied or redistributed,
they must contain the link to the original test materials and their
status
If the Document License is not applicable and a user needs to modify the
test materials in order to use them, the W3C Software License can be used.
For publishing test materials acquired from an external group, other licenses
may be applicable (see transferring test
suite).
Note. QAWG has agreed in principle, Issue #49, to
the desirability and draft text of a Test Materials License, based mostly on
the Document License. Discussion with W3C Legal is pending.
Examples &
Techniques for checkpoint.
Checkpoint 6.3. Describe how and where the test materials
will be published.
[Priority 2]
To fulfill this checkpoint, in its QA Process Document the
Working Group MUST describe how test materials will be published, and point
to the Web site at which they will be published.
Discussion. If the test materials are to be published on the W3C site, it
is recommended to locate them within the corresponding activity domain. It is
strongly recommended not to publish test materials in the TR space for the
following reasons:
- Documents published in TR space cannot be changed. Test materials
change and evolve frequently even after the specification becomes W3C
Recommendation. Changes can be triggered by new submissions,
specification errata's, test errors reports, new versions of the
specification.
- TR space is intended for Technical Recommendations. While test
materials can serve as an extensive set of use cases and help in
interpretation of the related specification, they are not intended to
play the role of recommendation.
It is recommended to use one of the practices for publishing test
materials, described in Test Guidelines.
Examples &
Techniques for checkpoint.
Checkpoint 6.4. Provide a conformance verification disclaimer
with the test materials.
[Priority 1]
To fulfill this checkpoint, the Working Group MUST provide
a disclaimer about the use of the
test materials for conformance verification of implementations.
Discussion. Although tests suites may be used for conformance
verification, the Working Group must make users aware that:
- passing all of the tests does not guarantee full conformance of an
implementation to the specification
- failing the test suite means failing tests for the specific feature
they target
An example of a conformance disclaimer may be found in the Conformance chapter of this specification.
Examples &
Techniques for checkpoint.
Checkpoint 6.5. Address the publication of test results for
products.
[Priority 2]
To fulfill this checkpoint, in its QA Process Document the
Working Group MUST encourage the publication of test results, MUST address
sample scenarios of where and how such publication can be done, and MAY
identify a WG-sponsored Web site for publishing collected results or a
directory of results.
Discussion. Publishing test results for the products implementing the
technical specification is useful for users, vendors, specification authors
and the test suite quality itself. Taking responsibility to publish test
results for other vendor's product could be a problem for the Working Group.
In order to solve such difficulties, the Working Group may encourage vendors to publish results
for their implementations themselves. Such publication should include or
describe a test harness that would allow anyone to reproduce the results. The
Working Group may provide the web space to publish collected results.
Examples &
Techniques for checkpoint.
Guideline 7. Plan the transfer of test materials to W3C if
needed.
As stated before in the notes to the checkpoint for test materials repository, it is recommended
that test materials reside in the W3C. If test development was done outside
by some external entity or group (EG), and the Working Group (WG) together
with EG decided to move test
materials (TM) to W3C, the
following checkpoints define the requirements for the Working Group.
All of the checkpoints of this guideline are not applicable if the Working
Group does not transfer test materials from an external entity.
Checkpoints:
Checkpoint 7.1. Perform a quality assessment of any test
materials that are candidates for transfer.
[Priority 2]
To fulfill this checkpoint, as a part of any test materials
transfer process, the Working Group MUST perform and record an assessment of
the quality of the test materials.
Discussion. This checkpoint parallels the checkpoint for the test review process.
During test materials review, the Working Group must follow the criteria defined in the document that is
required by that checkpoint.
This checkpoint is not applicable if the Working Group does not transfer
test materials from an external entity.
Examples &
Techniques for checkpoint.
Checkpoint 7.2. Identify sufficient staff resources to meet
the needs of any transferred test materials.
[Priority 1]
To fulfill this checkpoint, as a part of any test materials
transfer process, the Working Group MUST commit staff resources commensurate
with planned ongoing WG test materials' deliverables, and MUST record that
commitment in some consensus WG document.
Discussion. This checkpoint parallels checkpoint 2.2 for the specific
circumstance of transferring the test materials from the outside to the
W3C.
This checkpoint is not applicable if the Working Group does not transfer
test materials from an external entity.
Examples &
Techniques for checkpoint.
Checkpoint 7.3. For any transferred test materials, resolve all
IPR issues with the
external party that produced the test materials.
[Priority 1]
To fulfill this checkpoint, as a part of any test materials
transfer process, the Working Group MUST have a documented agreement with the
external entity on all IPR aspects that are
applicable to the transferred materials.
Discussion. This checkpoint
is not applicable if the Working Group does not transfer test materials from
an external entity.
(Ed note. Example/technique here was moved to Operational Examples
& Techniques.)
Examples & Techniques
for checkpoint.
Guideline 8. Plan
for test materials maintenance.
The ongoing maintenance of test materials is critical to their long term
integrity, and may require a significant commitment of resources.
Checkpoints:
Checkpoint 8.1. Maintain the contribution and review
procedures throughout the entire life cycle of the test materials and the
standard itself.
[Priority 3]
To fulfill this checkpoint, in some WG consensus document the
Working Group MUST define a plan and commit resources for long term
maintenance of the test materials contribution and review procedures.
Examples &
Techniques for checkpoint.
Checkpoint 8.2. Specify a test materials update procedure to
track new specification versions/errata.
[Priority 2]
To fulfill this checkpoint, in its QA Process Document the
Working Group MUST specify procedures to update the test materials to track
new specification versions and errata levels.
(Ed note. Example/technique here was moved to Operational Examples
& Techniques.)
Examples &
Techniques for checkpoint.
Checkpoint 8.3. Identify a procedure for test validity
appeals.
[Priority 2]
To fulfill this checkpoint, in its QA Process Document the
Working Group MUST identify the communication channel for appeals of test
validity and a procedure for for resolving such appeals.
(Ed note. Example/technique here was moved to Operational Examples
& Techniques.)
If there
is still active work on the test suite even if the Working Group is not
re-chartered to handle the work, then it is up to the Director to determine
how this work should be done. Some options include:
- it may be assigned to a Coordination Group;
- it may be assigned to a related Working Group;
- the Team could take care of minor items.
Examples &
Techniques for checkpoint.
3. WG relationship to QA Activity
The QA Activity works closely with other W3C WGs providing assistance and
expertise in helping them achieve their QA goals and deliverables. The QA
Activity anticipates different relationships with the various WGs depending
on the QA-specific needs of a particular Working Group. For example, the
resources and experience of Working Group members as well as their stage in
the Recommendation track may influence the type and level of collaboration
between the Working Group and QA Activity. Potential relationships between
Working Groups and the QA Activity include:
- liaison and consultation
- active reviews
- adjudicate entry and exit criteria
- QA resources supplement
- resolution of external QA requests.
The QA Activity strives to make their expertise available to the Working
Groups, responding to requests and providing assistance on an as-needed
basis. The degree of assistance and participation of the QA Activity members
may be determined on a case-by-case basis, but a consultancy role should
almost always be possible. Key determinants of the WG/QA Activity
relationship will include level of available QA Activity resources and the
evolution of W3C policy on QA requirements.
3.1 Liaison and Consultation
In an assistive role, the QA Activity provides consultation to all Working
Groups with respect to planning, building, and/or acquiring test materials.
Consultation may be provided informally in response to questions sent to the
QA Activities IG mail exploder, www-qa@w3.org or formally following the QA
Activity's Request for Assistance Process.
In addition to describing how WGs can submit requests to the QA Activity,
it describes the process within the QA Activity for handling these requests
including review procedures and criteria. The Request for Assistance Process
was created to ensure a fair and impartial, quick and accurate response to
requests.
WGs may submit requests for assistance to the QA Working Group for the
following:
- Request for clarification on documents published by QA Activity,
- Request for review of WG quality assurance strategy and plans
- Request for review of WG charter
- Requests for review of WG specification
- Requests for review of WG QA Process Document
- Requests for consultation on specific QA techniques/test suite
architecture
- Requests for help on coordination of joint QA activities or
collaboration with external (to W3C) organizations
Upon receiving one of these requests, a QA Working Group task team will
review the request, appoint reviewers/consultants, inform the WG of the
request's status, establish a liaison with the WG, and working in concert
with the WG, respond to the request.
Other requests will be considered by the QA Activity Working Group on a
case-by-case basis in accordance with the Request for Assistance Process.
Decisions will be based on the nature of the request, (e.g., within the scope
of the QA Activity domain). The WG will be informed of the QA Activity's
decision and provided an opportunity to appeal the decision.
[ED NOTE: Just as other WGs must have a Process Document, the QAWG has agreed to have
a Process document containing the process for formal consultation, active
reviews -- under construction.]
3.2 Active Reviews
At the request of a WG or in response to a general call for participation
(e.g., public working draft or email to W3C Chair to review something) or due
to a W3C Process requirement, the QA Activity Working Group may provide
detailed review and assessment of a WG's deliverable. If such a detailed
review is conducted, it will be conducted with the knowledge of the WG and in
concert with the WG. If appropriate, a QA Activity member will participate in
the WG meetings.
The QA Activity Working Group will follow the procedures set forth in its
Request for Assistance Process for establishing a review team, performing the
review, and communicating with the WG. The QA Framework Guidelines will be
used in the assessment of the WG deliverable, verifying that appropriate
Framework checkpoints are met. Additionally, any relevant WG documents (e.g.,
Charter, QA Process) will also be used.
3.3 Adjudicate entry and exit criteria
WGs may wish to define QA-specific entry and exit criteria for milestones
on the Recommendation track. The QA Activity Working Group can help to
identify these criteria.
3.4 QA resource supplement
At the request of a WG, the QA Activity WG may be able to provide
resources for the planning and building of test materials.
3.5 Resolution of external QA requests
QA-related requests received from organizations or individuals external to
the W3C, will be categorized as to the nature of the request and handled
accordingly. QA-related questions of general interest will be posted and
discussed on the QA-IG mail exploder. QA-related questions on a specific W3C
technology will be shared with the appropriate WGs and/or Horizontal Teams
and a coordinated dialogue and response will be pursued. Similarly, test
material inquiries and offers to donate test materials will be coordinated
with the appropriate WGs, Horizontal Teams, and W3C Management Team. Details
of how these requests are handled within the QA Activity are described in the
Request for Assistance Process.
This section defines conformance of Working Group processes and operations
to the requirements of this specification. The requirements of this
specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this specification,
and apply to the Working Group QA-related documents and deliverables required
by this specification.
4.1 Test assertions
The test assertions of this Operational Guidelines document are found
within the prioritized checkpoints. A checkpoint will contain at least one,
and may contain multiple individual requirements. These requirements are the
test assertions of this specification. A checkpoint is satisfied by
satisfying all of the individual requirements. Failing one individual
mandatory requirement means that the checkpoint is not satisfied. Mandatory
requirements are those that use the conformance keywords "MUST", "MUST NOT",
"REQUIRED", "SHALL", or "SHALL NOT".
4.2 Conformance definition
This section defines three degrees of conformance to this guidelines
specification:
-
A-Conforming: all Priority 1 checkpoints are
satisfied;
-
AA-Conforming: all Priority 1 and Priority 2
checkpoints are satisfied;
-
AAA-Conforming: all Priority 1, Priority 2, and
Priority 3 checkpoints are satisfied;
A Working Group conforms to the "QA Framework: Operational Guidelines" at
degree X (A, AA, or
AAA) if the Working Group meets at least all degree
X conformance requirements.
To make an assertion about conformance to this document, minimally
specify:
- The guidelines title: "QA Framework: Operational Guidelines"
- The guidelines URI: http://www.w3.org/TR/qaframe-ops/
- The degree of conformance satisfied: A-Conforming, AA-Conforming, or
AAA-Conforming.
Example:
4.3 Conformance disclaimer
The checkpoints of this specification present verifiable conformance
requirements about the operational aspects of Working Group quality
processes. As with any verifiable test requirements, users should be aware
that:
- Passing all of the requirements to achieve a given conformance degree
-- A, AA, or AAA -- does not guarantee that the subject operations and
processes are well-suited to or will achieve their intended purposes, nor
does it guarantee the quality or suitability of test materials produced
under the processes.
- Failing to achieve degree A conformance does not mean that the subject
quality processes and/or associated test materials are necessarily
deficient to their intended purposes. It means that the processes have
failed one or more checkpoints that best-practice experience has shown to
facilitate and enable successful quality processes, that in turn have a
high correlation with timely and successful development and maintenance
of the test materials.
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C),
- Dimitris Dimitriadis (Ontologicon),
- Karl DuBost (W3C),
- Peter Fawcett (RealNetworks),
- Kirill Gavrylyuk (Microsoft),
- Dominique Hazael-Massieux (W3C),
- Lofton Henderson (CGM Open),
- Ian Jacobs (W3C),
- David Marston (Lotus Development Corp),
- Sandra Martinez (NIST),
- Jack Morrison (Sun Microsystems),
- Lynne Rosenthal (NIST),
- Mark Skall (NIST),
- Andrew Thackrah (Open Group),
- Olivier Thereaux (W3C),
-
EXTERN-TA
- QA activity email thread about third-party
participation in test materials production, available at
http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html.
-
MATRIX
- W3C-wide conformance activity survey covering all the Working Groups,
"The Matrix",
available at http://www.w3.org/QA/TheMatrix.
-
TAXONOMY
- QA Activity test
taxonomy, a classification scheme for conformance test materials,
available at http://www.w3.org/QA/Taxonomy.
-
OPS-CHECKLIST
- An appendix to this operational guidelines document presents all checkpoints in tabular form.
Available at
http://www.w3.org/TR/2002/WD-qaframe-ops-20021108/qaframe-ops-checklist
-
OPS-EXTECH
-
QA Framework: Operational
Examples & Techniques
, L. Henderson, L. Rosenthal, D.
Dimitriadis, K. Gavrylyuk, Eds., W3C Note, (initially) 08 November
2002, companion version to this document, latest companion version
available at http://www.w3.org/QA/WG/2002/11/qaframe-ops-extech.
-
PROCESS
-
World
Wide Web Consortium Process Document
, I. Jacobs, Ed., 19
July 2001, available at
http://www.w3.org/Consortium/Process-20010719/.
-
QA-GLOSSARY
- A comprehensive glossary of QA terms, maintained by the QA Working
Group. (Initial version under construction.)
-
QAF-INTRO
-
QA Framework:
Introduction
, L. Henderson, K. Gavrylyuk, D. Dimitriadis, L.
Rosenthal, Eds., W3C Working Draft, 08 November 2002, companion version
to this document, available at
http://www.w3.org/TR/2002/WD-qaframe-intro-20021108/.
-
QAF-SPEC
-
QA Framework: Specification
Guidelines
, Dominique Hazaël-Massieux, L. Henderson, L.
Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 08
November 2002, companion version to this document, available at
http://www.w3.org/TR/2002/WD-qaframe-spec-20021108/.
-
QAF-TEST
-
QA Framework: Test Guidelines (not yet published).
-
QAIG
-
Quality Assurance Interest
Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/IG/.
-
QAWG
-
Quality Assurance Working
Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/WG/.
-
DOM Working Group
TS
-
DOM
TS Process Document, D. Dimitriadis, Ed., 15 January 2002,
available at
http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115.
-
REC-TRACK
- Stages and milestones in the W3C
Recommendation Track, per the Process
Document (Process Document is available at
http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
-
RFC2119
-
Key words for use in
RFCs to Indicate Requirement Levels, March 1997, available at
http://www.ietf.org/rfc/rfc2119.txt.
-
WCAG10
-
Web Content
Accessibility Guidelines 1.0
, W. Chisholm, I. Jacobs, G.
Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at
http://www.w3.org/TR/WCAG10/.
-
WG-QA-RANGE
-
Email
proposal by David Marston, on the QA public mail list, for range of
Working Group commitment levels to conformance test materials
production, available at
http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html.
- 2002-11-08, third published WD
-
Revised Introduction, Conformance for better conformance to SpecGL.
Implemented other minor editorial and resolved-issue changes. Make over
all checkpoints for better verifiability.
- (since 11/03) Moved a handful of examples to Ops Extech.
- (since 10/31)
- Make over of GL3-GL8, to isolate the "to fulfill" criterion
and apply TA markup;
- Numerous minor editorial (wording & grammar)
improvements.
- Miscellaneous editorial corrections, fix missing anchors,
etc.
- Combine CK5.5 into CK5.4 as new MUST requirement.
- In GL1 and GL2, apply test assertion markup, to identify
fulfillment criteria.
- In GL2, remove the "In the charter" bits of all checkpoints, and
embed old/new requirements in the fulfillment criteria.
- In GL1, remove the "In the charter" bits of all checkpoints, and
embed old/new requirements in the fulfillment criteria.
- Identify the checklist as the ICS
pro-forma for OpsGL.
- Revise "Conformance" chapter per SpecGL (level/degree, test
assertions, etc)
- Rewrite descriptions of priority 1, 2, 3 (subsection of
Intro)
- Clarify RFC2119 usage ("Terminology" subsection of Intro)
- Reorganize and augment Introduction, w/ subsections for:
scope&goals, motivation, understanding&using, priorities,
etc.
- 2002-05-15, second published WD
-
Checkpoints numbering is changed, changes list refers to the
numbering from the previous WD
[1].
- Ck 4.1 - split into 2 having QA moderator as Pri1 and the task
force as Pri2
- Ck 4.4 - advanced to pri1
- Ck 4.5 - removed "if applicable"
- Ck 5.1 - advanced to Pri1, remove accredited, changed to "usable
for intended purpose"
- Ck 5.3 - advanced to Pri1
- Advanced Ck 6.1 to Pri1, removed "future", add "freely
available"
- Ck 6.2 - advanced to Pri1
- Ck 6.4 - advanced to Pri1.
- Ck 6.5 reworded checkpoint and explanation text. advanced to
Pri2
- Ck. 7.1. Added a note on how this checkpoint is related to 5.4
"review process". Clarified what is the assessment quality by
referring to the criteria for the review process.
- Added Ck 5.5 with Pri1 as a criteria for the review process -
needed for a reference from other places where we talk about "tests
quality assessment criteria". Pri1 is needed to match the minimum
requirement in the WG QA commitment.
- Ck 7.2 - added note on how this relates to the Ck 2.2 and moved
it to Pri1
- Ck 7.4 - removed
- Ck 8.1 - reworded.
- Ck 1.1 - changed the commitment table based on Mark's
suggestions: added requirement for some test assertions to the
level 2, added requirement for tests review to the Level 3, swapped
level 5 and level 6.
- Added 2 more checkpoints in advancement of Ck 1.1: with pri2 to
commit to the level 5 and with pri3 to commit to the level 7.
Explanation notes are there
- Added first draft for Definitions of the Ck Priorities to the
Introduction section.
- Added Conformance disclaimer subsection drafted by Lofton, per WG
decision.
- Other rewrites and small editorial changes.
[1] http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops
- 03-11-2002
-
Checkpoints numbering is changed, changes list refers to the
numbering from the previous WD
[1].
- Incorporated Lynne's contribution - Ch3 rewrite
- add checkpoint for soliciting QA resources in Call for
Participation with P2
- minor rephrasing of statements of chkpt 1.1, 1.2, and 1.3
- rephrase "CR-exit" discussion
- Changed chkpt 2.2 from [1] to have P1
- Changed chkpt 1.2 from [1] to have P1
- Removed 3.1, added more explanation notes to 3.2 (P2) and 3.3
(P3)
- Removed the checkpoint 4.1 from [1] and add it to the Intro
section
- Removed the checkpoint 5.1
- Added more explanation, clarified the difference between chckpt
6.3 and 6.5 from [1]
- Added more explanation, clarified the difference between 5.3 and
5.5 from [1]
- Added text to explain why the WG SHOULD NOT publish test
materials in TR space
- Added text to explain why the Document License is recommended
- Added mild disclaimer on the use of the "SHOULD", "MUST", etc
keywords to the Terminology section
- Other rewrites and small editorial changes
[1] http://www.w3.org/QA/WG/2002/02/qaframe-ops-0225.html
- 02-28-2002
- Major rewrite of the previous draft. Incorporated Lynne's comments
about Gd1 and Gd4, split Gd regarding QA process, added bunch of new
checkpoints.