This document defines a common Operational Framework for building
conformance test materials for W3C specifications. It presents
operational and procedural guidelines for groups undertaking conformance
materials development. This document is one in a family of Framework
documents of the Quality
Assurance (QA) Activity,
which includes the other existing or in-progress specifications:
Introduction; Specification Guidelines; and, Test Guidelines.
This version is a QAWG-review version, for illustrating and
discussing in-progress modifications to the Last Call version.
- (20030512) Implement agreed and proposed changes to GL1 and GL2 --
simplifications and clarifications.
- (20030513) Implement lots of miscellaneous of the LC-60.x
clarifications, plus license clarifications.
- (20030514) Implement remainder of the LC-60.x issue resolutions.
- (20030609) Implement [...generate the list of...] resolutions
See details in Change
history.
The SoTD of the Last Call version follows...
This section describes the status of this document at the time of its
publication. Other documents may supersede this document. The latest version
of this document series is maintained at the W3C.
This document is a W3C Working Draft (WD), made available by the W3C Quality
Assurance (QA) Activity for discussion by W3C members and other interested
parties. For more information about the QA Activity, please see the QA Activity
statement.
This Working Draft is published for Last Call review. This version
incorporates the closure of all open QAWG (QA Working Group) issues about this
specification, as well as editorial improvements to more clearly explain the
rationale behind a number of the checkpoints. The Previous Version (3rd
published WD) represented convergence on a stable set of the individual
checkpoints and their priorities, and was structured so that its conformance
requirements (fulfillment criteria) were clearly isolated and identified. For
details, please see Change
history.
This version supersedes all previous drafts. Future progression of this
document beyond Working Draft is planned, but the final status has not been
determined at this time. See QAWG
issue #18 and issue
#71. It is anticipated that this specification will eventually advance to
Candidate Recommendation (CR), after successful discussion and resolution of
any and all issues that arise during Last Call review. The QAWG has discussed criteria for finishing
CR phase and entrance to Proposed Recommendation ( PR). The agreed criteria are: for
each Priority 1 and Priority 2 checkpoint, two example implementations that
successfully fulfill the checkpoint.
This part of the Framework document family has a companion QA
Framework: Operational Examples & Techniques. That informative
companion document is currently being progressed as a W3C Note. At least
until this guidelines document stabilizes, that Examples & Techniques
companion will be maintained and frequently updated in QA Working Group Web
space (as opposed to /TR/). Although that document is not the principal
subject of this review, the QAWG
welcomes feedback on it as well.
The QA
Working Group Patent Disclosure page contains details on known patents
related to this specification, in conformance with W3C policy
requirements.
Please use the
provided form to make your comments. If for some reason you are unable to
use the form, you may email comments to www-qa@w3.org, the publicly archived list
of the QA Interest Group
[QAIG]. Please note
that comments that you make will be publicly archived and
available, do not send information you would not want to see distributed,
such as private data.
Publication of this document does not imply endorsement by the W3C, its
membership or its staff. This is a draft document and may be updated,
replaced, or obsoleted by other documents at any time. It is inappropriate to
use W3C Working Drafts as reference material or to cite them as other than
"work in progress".
A list of current W3C Recommendations and other technical documents can be
found at http://www.w3.org/TR/.
- Introduction
- Scope and goals
- Class of product and audience
- Motivation and expected benefits
- Relationship to other
specifications
- Understanding and using this
document
- Checkpoint priorities & conformance
levels
- A chronological view of the
guidelines
- Terminology
- Guidelines
- Commit to Quality Assurance in
Working Group activities.
- Commit to resource level for Working
Group QA activities.
- Synchronize QA activities with the
specification milestones.
- Define the QA process.
- Plan test materials
development.
- Plan test materials
publication.
- Plan the transfer of test
materials to W3C if needed.
- Plan for test materials
maintenance.
- WG relationship to QA Activity
- Conformance
- Normative sections
- Extensibility
- Conformance requirements &
test assertions
- Conformance definition
- Conformance disclaimer
- Acknowledgments
- References
- Normative references
- Informative references
- Change history
Two separate appendices to this document [OPS-CHECKLIST] and [OPS-ICS] present all checkpoints in a tabular form sorted in their original
order and sorted by their
priorities, for convenient reference. The latter is an Implementation
Conformance Statement (ICS) pro-forma for this specification.
[@@Ed note. this Introduction is to be
substantially re-worked per resolution ofLC-110]
The scope of this specification is a set of verifiable
requirements for the process and operational aspects of the quality practices
of W3C Working Groups. The primary goal is to help the W3C
Working Groups (WGs) with the
planning, development, deployment, and maintenance of conformance test
materials (TM).
For this guidelines document, the term conformance test
materials includes:
- conformance test suites,
- validation tools,
- conformance checklists,
- and, any other materials that are used to check or indicate conformance
of a implementation to a specification.
The class of product that is the target of the
requirements in this specification is: all process and operational aspects of
Working Group quality practices. This includes
- the WG's planning and
commitment for QA,
- staffing to meet commitments,
- organization and logistics of the quality practices,
- publication and maintenance,
- synchronization of quality processes with other WG activities.
The intended audience for these guidelines is all W3C
Working Group members, as well as the actual developers of conformance
materials for W3C specifications.
It is optimal that sound quality practices are integrated into WG activities from the very beginning.
Nevertheless, these guidelines are written for newly chartered and existing
Working Groups alike. Working Groups who may already be doing some of these
activities should review the document and incorporate the principles and
guidelines into their quality practices as much as possible.
As the complexity of W3C specifications and their interdependencies
increase, quality assurance becomes even more important to ensuring their
acceptance and deployment in the market. There has been a growing awareness
and interest in conformance and quality. In approving and initiating the QA
Activity, W3C has endorsed the principle that in order for W3C Web standards
to achieve full interoperability and access to all, the quality of the
implementation must be given as much attention as the standards' development.
The principal factor for improving the quality of implementation is early
availability of conformance test materials.
Although not explicitly stated, the W3C Process Document supports the
development of conformance test materials.
[...] groups may produce technical reports, review the work of other
groups, develop sample code or test suites, etc." (see Process
Document, section 3.)
W3C should make every effort to maintain its Recommendations (e.g., by
tracking errata, providing test bed applications, helping to create test
suites, etc.) (see Process
Document, section 5.2.5, "Ongoing work".)
To meet these suggestions and address the implementation requirements of
the Process Document, some Working Groups have included the development of
conformance materials as part of their PR-entrance criteria. Examples
include Scalable Vector Graphics (SVG), User Agent Accessibility
Guidelines (UAAG), and Extensible
Style Language - Formatting Objects (XSL-FO).
This makes sense, since it is natural for test suites and implementations to
develop in parallel - each is a help to the development of the other.
There is already a body of contemporary QA experience and activity amongst
the Working Groups. The Matrix [MATRIX]
identifies more than a score of test suites and validators, in various states
of development. Moreover, many Working Groups have already established
procedures, techniques and tools for developing test materials (e.g.,
Document Object Model - DOM). It makes sense to capitalize on
what has already been done and share that with those who are starting out and
those who are already in the process of developing conformance materials.
Accordingly, these guidelines aim to capture the experiences, good
practices, activities, and lessons learned of the Working Groups, and to
present them in a comprehensive, cohesive set of documents for all to use and
benefit from. By standardizing the best of current practice, these guidelines
should allow the WGs to reuse what
works rather than having to reinvent, which in turn should facilitate and
expedite the work of the WGs.
Conformance with these guidelines should also promote consistency across the
various Working Group quality activities and deliverables
This document is part of the QA Framework family of documents. The
QA Framework documents aim to help the WGs improve all aspects of their quality
practices by solidifying and extending current quality practices found within
the W3C.
The QA Framework documents are:
The QA Framework documents include guidelines documents
and examples & techniques documents. [@@tbd -- finish
sorting out the GL/Extech distinction]
The QA Framework documents are interrelated and complement each other. For
example, there is a close relationship between the processes for dealing with
versions and errata, and the test framework architecture for handling
multiple versions and errata levels. Hence there is interrelationship between
this document and the Test Guidelines. The reader is strongly encouraged to
be familiar with the other documents in the family.
The guidelines in this document start with those that are applicable as
early as the formation of a Working Group (e.g., charter considerations), and
continue through the various process and operational activities necessary in
planning, developing, deploying and maintaining conformance materials.
This specification applies the WAI (Web Accessibility
Initiative) guidelines model to Working Group quality practices and the
development of conformance test materials. See, for example, Web Content Accessibility Guidelines. Each guideline
includes:
- the guideline number,
- the statement of the guideline,
- some discussion of and rationale behind the guideline,
- a collection of one or more checkpoints.
The checkpoint definitions in each guideline define the processes and
operations that need to be implemented in order to accomplish the guideline.
Each checkpoint definition includes:
- the checkpoint number,
- the statement of the checkpoint (its "title"),
- the checkpoint priority,
- a statement of the conformance requirements of the checkpoint, i.e.,
how to fulfill the checkpoint,
- optional informative notes, rationale, clarifying examples, and cross
references to related guidelines or checkpoints,
- a link to the corresponding section in Operational Examples &
Techniques.
Each checkpoint is intended to be specific enough so that someone can
implement the checkpoint as well as verify that the checkpoint has been
satisfied. A checkpoint will contain at least one, and may contain multiple
individual requirements, that use RFC2119 normative keywords. See the Conformance section for further
discussion of requirements and test assertions.
Two separate appendices to this document [OPS-CHECKLIST] and [OPS-ICS] present all checkpoints in a tabular form sorted in their original
order and sorted by their
priorities, for convenient reference. The latter is an Implementation
Conformance Statement (ICS) pro-forma for
this specification.
Some checkpoints are more critical than others for the timely production
of high-quality, highly usable test materials. Therefore each checkpoint has
a priority level based on the checkpoint's impact on the quality and timing
of the test materials produced by a Working Group.
- [Priority
1]
- Critical/essential. These checkpoints are considered to be basic
requirements for ensuring that test materials are usable, and are
produced in time to ensure the quality of the standard and its
implementations. Satisfying these checkpoints is a basic requirement to
ensure quality and interoperability of the standard.
- [Priority
2]
- Important/desirable. Satisfying these checkpoints, in addition to the
priority 1 checkpoints, should significantly improve the usability and
timeliness of the test materials, as well as the quality of the
standard and its implementations.
- [Priority
3]
- Useful/beneficial. Satisfying these checkpoints, on top of all the
others, will further improve the quality, usability, and timeliness of
the test materials and the standard itself.
There is a strong association between the individual guidelines and stages
in the life of a Working Group, its specifications, and its QA deliverables.
Certain guidelines either only apply or ideally apply at certain stages.
The following { diagram | table } presents the
correlation. [@@Ed note. TBD. But contents are roughly...
- Guideline 1. Charter commitment ideally. (Later if necessary).
- Guideline 2. Charter commitment ideally. (Later if necessary).
- Guideline 3. ...blah...
- Guideline 4. ...blah...
- Guideline 5. ...blah...
- Guideline 6. ...blah...
- Guideline 7. ...blah...
- Guideline 8. ...blah... ]
The keywords "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT",
"SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" are used as
defined in RFC 2119 [RFC2119]. When used
with the normative RFC2119 meanings, they are all upper case. Occurrences of
these words in lower case comprise normal prose usage, with no normative
implications.
Unusual terms in this framework document are defined when first used, and
most generally useful QA-specific terms will eventually be in the QA Glossary
[QA-GLOSSARY].
In approving and initiating the QA Activity, W3C has endorsed the
principle that in order for W3C Web standards to achieve full
interoperability and access to all, the quality of implementations must be
given as much attention as specifications' development.
The quality of specifications and the early availability of conformance
test materials (TM) are significant
contributors to the quality of implementations. Early adoption of a suitable
commitment level for these two components of a WG's quality practices -- followed by
identification of specific QA deliverables, milestones, and dependencies with
other WG deliverables -- is
prerequisite to detailed planning, execution, and timely achievement of at
least a minimal quality level across all of a Working Group's
deliverables.
This guideline and Guideline 2 concern the early statement by the Working
group of degree(s) of QA commitment, commitment to test materials and other
QA deliverables, and commitment to needed QA staffing resources. See further
discussion of the chronology
of the guidelines in "Introduction".
New Working Groups -- i.e., those that are writing and submitting their
charters for approval -- are required in this guideline's checkpoints to make
the specified commitments in their charters. Working Groups that are renewing
their charters are considered the same as new WGs. Existing Working Groups could satisfy
the checkpoints by amendment to their charter, or in other ways, the latter
to be specified in QA
Framework: Operational Examples & Techniques [OPS-EXTECH].
Checkpoints:
Conformance requirements: the WG MUST:
- define its commitment level to OpsGL -- A, AA, or AAA;
- for any Recommendations that it plans to produce, define its commitment
level to SpecGL -- A, AA, or AAA;
- for any Test Materials that it plans to produce or adopt, define its
commitment level to TestGL -- A, AA, or AAA.
- A new or rechartering Working Group MUST define these commitments in
its Charter;
- an existing Working Group MUST document consensus for the commitments
in one of various other ways.
Discussion. This checkpoint is about early commitment
of the WG to the intended degree of quality practices in the various aspects
of the WG's activities. It is not about the actual performance or delivery of
quality practices and deliverables. For existing working groups that make the
QA commitment after chartering, various ways to do it are discussed in the
Examples & Techniques document.
Examples &
Techniques for checkpoint.
Conformance requirements:
- the WG MUST commit to produce or adopt at least some test materials for
each of the WG's specifications before it becomes Recommendation;
- a new or rechartering Working Group MUST define this commitment in its
Charter;
- an existing Working Group MUST document consensus for the commitment in
one of various other ways.
Discussion. This checkpoint is about early commitment
of the WG about its test materials intentions. It is not about the specific
deliverables, nor about the details of the test materials or their actual
production and delivery. For existing working groups that make the test
materials commitment after chartering, various ways to do it are discussed in
the Examples & Techniques document.
Examples &
Techniques for checkpoint.
Conformance requirements:
- the WG MUST commit to produce or adopt a complete Test Materials before
Recommendation, where complete is defined as: at least one test case for
every identifiable conformance requirement of the specification;
- a new or rechartering Working Group MUST define this commitment in its
Charter;
- an existing Working Group MUST document consensus for the commitment in
one of various other ways.
Discussion. This checkpoint is about early commitment
of the WG about its test materials intentions. It is not about the specific
deliverables, nor about the details of the test materials or their actual
production and delivery. For existing working groups that make the test
materials commitment after chartering, various ways to do it are discussed in
the Examples & Techniques document.
Examples &
Techniques for checkpoint.
Conformance requirements:
- a new or rechartering Working Group MUST document the QA deliverables
and milestones its Charter;
- an existing Working Group MUST document QA deliverables and milestones
in some consensus record, in one of various other ways.
Discussion. The W3C
Process Document requires for Charters, that deliverables be identified
with milestones. It is vital that the milestones for QA deliverables are
synchronized and even serve as criteria for WG technical deliverables
(specifications).
Examples of QA deliverables include sets of use cases, primers, lists of
test assertions, test suites (produced or acquired), validators, test
harnesses.
Examples & Techniques
for checkpoint.
Conformance requirements:
- a new or rechartering Working Group MUST, in its charter, specify the
QA criteria required for the attainment of major WG milestones;
- an existing Working Group MUST define the association in some consensus
record, in one of various other ways.
Discussion. A de facto convention of test suite
completion before entrance into PR phase (and by implication,
completion of CR phase,
if there was a CR phase)
seems to be a common process goal amongst a number of the existing test suite
efforts. For example,
SVG, UAAG, and others have
defined this criterion. It is natural for test suites and implementations to
develop in parallel -- each is a help to the development of the other. And,
the W3C Process Document does address
implementation requirement:
Advancement of a technical report to Candidate Recommendation is an
explicit call for implementation experience to those outside of the related
Working Groups or the W3C itself." (see Process
Document, section 5.2, about "Candidate Recommendation.)
Preferably, the Working Group should be able to demonstrate two
interoperable implementations of each feature. (see Process
Document, section 5.2.4, about "Entrance Criteria".)
Related checkpoints. This checkpoint is related to Checkpoint 3.1, but is concerned specifically
with early, (ideally) Charter-phase commitment to bind QA-related entry/exit
criteria to the major process milestones as defined in the W3C Process
Document. Checkpoint 3.1 is concerned with the operational aspects of
synchronizing specific QA deliverables with successive specification
versions.
Examples &
Techniques for checkpoint.
It is highly beneficial to the success of the WG's QA work that it address QA resource
requirements from the beginning of the Working Group formation. Starting from
the Working Group Charter and later in the Call For Participation special
attention is required to the staffing and other resource requirements for
successful QA work.
This guideline and Guideline 1 concern the early statement by the Working
group of degree(s) of QA commitment, commitment to test materials and other
QA deliverables, and commitment to needed QA staffing resources. See further
discussion of the chronology
of the guidelines in "Introduction".
The sooner the resources needs are defined and a resource level is
committed, the better. Therefore new Working Groups -- i.e., those that are
writing and submitting their Charters for approval -- are required to have
the resource commitments in their charters. Working Groups whose charters are
being renewed are considered the same as new WGs. While "sooner is better", existing
Working Groups can still satisfy the checkpoints, in other ways to be
specified in QA
Framework: Operational Examples & Techniques [OPS-EXTECH].
Checkpoints:
Conformance requirements: a new or rechartering Working Group
MUST, in its charter, address and discuss who will produce the test materials
and how. An existing Working Group MUST define the who & how in some
consensus record, in one of various other ways.
Rationale. Determining whether the WG plans to produce the test materials
themselves or get them from other sources is a first step in the overall
planning for QA deliverables. Identifying the scenario for producing these
deliverables is a prerequisite to estimating the resources and effort needed
for the WG's level of QA
commitment.
Discussion. TM-production scenarios range from all
intra- WG effort concurrent with
other WG deliverables, to importing
completed materials from an external group, and hybrid internal-external
efforts. Further discussion and example scenarios are found in
Operational Examples & Techniques.
Examples & Techniques
for checkpoint.
Conformance requirements: a new or rechartering Working Group
MUST, in its charter, allocate WG
staffing resources for the tasks necessary to meet the total QA commitments
according to the where-and-how plan. An existing Working Group
MUST define the allocation in some consensus record, in one of various other
ways.
Rationale. For any acceptable level of QA commitment,
there will be some staffing requirement. If the WG addresses the staffing level at the same
time that it defines the commitment level and general scenario, then that
both helps to confirm that the overall plans are realistic, as well as
facilitates that the commitments will be successfully realized.
Discussion. Depending upon the general intent and plan,
and how the test materials will be built, the commitment can range from
minimal to significant. At the least, a QA point-of-contact will be needed by
the WG.
This checkpoint is related to Checkpoint 4.1 and Checkpoint 4.2, in that they all deal with
staff for QA activities. However this checkpoint concerns the early,
(ideally) Charter-phase commitment to a level of QA staffing appropriate for
the total QA commitments, whereas the checkpoints 4.1 and 4.2 concern the
actual operational assignment of the staff and establishment of the QA task
force.
Examples &
Techniques for checkpoint.
Conformance requirements: a new or rechartering Working Group
MUST, in its Call for Participation, request that participating members
include QA specialists in their staff-resource allocation to the WG. An
existing Working Group MAY make external appeal for QA-specific resources in
one of various other ways.
Rationale. The WG needs explicit indication of resources
available for QA, in order to know whether Working Group capabilities are
commensurate with its deliverables and milestones. If the WG emphasizes in its participation calls that
QA specialists are needed and welcome, that should facilitate getting
competent and interested people to work on the WG's QA commitments.
Discussion. Once the Charter is prepared, the Director
sends a Call for Participation to the Advisory Committee. At this point AC
Representatives are asked to provide information about amount and type of
resources their organization plans to allocate for the particular Working
Group. Note that W3C Process [PROCESS]
allows a WG to have several participants per Member, and it is up to the WG
to determine details such as any further numerical restrictions, composition
by specialty, etc.
Examples &
Techniques for checkpoint.
The benefits of starting synchronization of the specification and test
materials development as early as possible include:
- it gives an extensive list of use cases;
- it identifies ambiguities in the specification at the early stages,
which saves cycles in the late phases;
- it provides clear set of testable assertions -- skeleton of the
specification -- which in its turn facilitates development of conformant
and interoperable implementations. The latter is a W3C process criterion
for entering the Candidate Recommendation phase.
Checkpoints:
Conformance requirements: the Working Group MUST publish QA
deliverables, including at least the test materials to which the WG has
committed, concurrently with each Working Group specification publication
milestone.
Rationale. Because each published version of the
specification -- WDs, CR(s), PR, etc -- is a changed document,
therefore all important dependencies such as test materials need to be
updated concurrently.
Discussion. QA deliverables to be synchronized must
include at least the test materials to which the WG has committed -- see
Checkpoints 1.2, 1.3, and 1.4. For such test materials, the WG might for
example synchronize the first public release of a test suite (TS) with
Candidate Recommendation (CR). The Working Group should consider and attempt
to synchronize other QA deliverables as well, for example:
- a TS production schedule might be produced and synchronized with first
published WD;
- a TS design document and enumeration of test cases might be
synchronized with later WDs;
- an interoperability matrix/report might be published at the beginning
of PR.
Related checkpoints. This checkpoint is related to Checkpoint 1.5. However,
that checkpoint concerns the (ideally) Charter-phase commitment to bind
QA-related criteria to the status transitions of the WG's specifications, as
defined in the W3C Process Document. This checkpoint concerns the
synchronization of any and all QA deliverables with successive versions of
the specifications, whether they be successive published WDs, CR succeeding
Last Call WD, etc.
Examples & Techniques
for checkpoint.
Conformance requirements:
- the Working Group's test materials MUST support the unambiguous
association of test materials versions to specification versions and
errata levels;
- this MUST be addressed in test materials documentation;
- and the Working Group SHOULD include specification versioning/errata
considerations in any other QA deliverables such as intermediate planning
documents.
Rationale. There will be products that support different
versions (editions) and/or different errata levels of the specification after
it becomes Recommendation. The requirements of this checkpoint ensure that
users will be able to select and apply the appropriate test materials for a
given product
Discussion. This checkpoint specifically extends the
previous one to cover support for specification milestones after
Recommendation status. As used in this checkpoint, the scope of "versions" at
least includes "editions" -- editorial revisions that incorporate errata and
small changes -- rather than major versions with major functional increments
(i.e., new Recommendations).
Building specification versioning/errata support into
the test materials' infrastructure is one effective technique. Other
techniques may also suffice, such as reliance on a code versioning system
like CVS (Concurrent Versions System).
Related checkpoints. This checkpoint concerns versioning/errata support
functionality in the infrastructure or framework of test materials. The
related checkpoint 8.2
is about defining maintenance procedures for test materials to track
specification versions or errata levels.
Examples &
Techniques for checkpoint.
A Working Group QA process encompasses all aspects of QA life within the
Working Group, including:
- a QA leader -- the moderator;
- a task force commensurate with QA commitment;
- producing the QA Process Document (QAPD);
- communications;
- development framework;
- and policies such as branding.
Documented examples of the QA process can be found at DOM TS process,
at XML Schema TS process, and at XML Protocol TS
process.
Checkpoints:
Conformance requirements: the Working Group MUST identify a
person to manage the WG's quality
practices.
Rationale. Having a single person assume primary
responsibility for the WG's quality
practices will help ensure that its QA commitments and milestones are met
efficiently and on schedule, and a single identified QA point-of-contact will
optimize external QA-related communications.
Discussion. The QA moderator is the overall manager of
all of the QA activities in the Working Group, and (by default) principal
point-of-contact. At even the lowest acceptable levels of QA commitment,
there are sufficient requirements that at least part-time of one person is
needed. Depending on the anticipated origin of the test materials, there are
several possibilities for the QA moderator, ranging from appointing a QA
member to inviting someone from an external organization.
Related checkpoints. This checkpoint and Checkpoint 4.2 are related to Checkpoint 2.2 -- they all
deal with staff for QA activities. However Checkpoint 2.2 concerns the early,
(ideally) Charter-phase commitment to a level of QA staffing appropriate for
the total QA commitments, whereas this checkpoint and Checkpoint 4.2 concern
the actual operational assignment of the staff and establishment of the QA
task force.
Examples & Techniques
for checkpoint.
Conformance requirements: the Working Group MUST identify and
assign a QA task force for the tasks necessary to meet the QA commitment
level and the committed QA deliverables, as identified in checkpoints 1.1 -
1.5.
Rationale. At higher levels of QA commitment and
deliverables, the resource requirements may exceed (part-time of) one person.
Identifying and assigning any additional needed QA staff resources is a
prerequisite to realistic schedules and QA delivery milestones.
Discussion. Depending on the level of involvement of a
Working Group in the test materials development, a task force assisting the
QA moderator can take responsibility for a QA framework development, test
materials development, review of contributions, and maintenance. In practice,
the task force can range from a subset of dedicated individuals, to some
fraction of the time of each WG
member. The WG needs to ensure that the size of the task force is adequate to
meet the committed QA deliverables and QA level, per the committed
milestones.
Related checkpoints. This checkpoint and Checkpoint 4.1 are related to Checkpoint 2.2 -- they all
deal with staff for QA activities. However Checkpoint 2.2 concerns the early,
(ideally) Charter-phase commitment to a level of QA staffing appropriate for
the total QA commitments, whereas this checkpoint and Checkpoint 4.1 concern
the actual operational assignment of the staff and establishment of the QA
task force.
Examples &
Techniques for checkpoint.
Conformance requirements: the Working Group MUST produce a QA Process
Document that:
- is a publicly readable document;
- addresses at least all of the topics required by other checkpoints in
these specification guidelines.
Rationale. A QA Process Document records, in one place,
key information about the WG's
quality-related logistical and communications setup, contribution and
licensing policies, publication and maintenance plans, and other critical
process and operational details. The production of this document ensures that
these key aspects of the WG's
quality practices are explicitly addressed and publicly recorded.
Discussion. The primary goal of a QA Process Document
(QAPD) is a single starting place to find important QA information about the
Working Group. A single document is preferred, and a template for writing
such is included in Operational Examples & Techniques. A
table of contents comprised of links to distributed WG documents is another
way to satisfy the QAPD goals.
The QAPD may contain more, but it must address at least
the requirements of these other checkpoints:
- means for QA-related communication (4.4);
- development framework (4.5)
- branding policy (4.6)
- test materials contribution (5.2) and review process (5.4)
- licenses applicable to test materials (5.3, 6.2)
- test materials publication (6.3, 6.4)
- test results publication (6.5)
- test materials maintenance (8.2, 8.3)
Related checkpoints. Satisfaction of this checkpoint requires satisfaction
of these related checkpoints: 4.4, 4.5, 4.6, 5.2, 5.4, 5.3, 6.2, 6.3, 6.4,
6.5, 8.2, 8.3.
Examples &
Techniques for checkpoint.
Conformance requirements: the WG's QA Process Document
- MUST specify at least one public archived mailing list for QA
announcements and submission of public QA comments;
- and, MUST establish a publicly readable "Test" Web page.
Discussion. In practice, the mailing list may be a
dedicated "test" mailing list, or something like an existing Interest Group
mail list could be used for the purpose. A common convention for the
QA-related Web page is recommended: name it /Test/ directly under the
WG's home page, e.g.,
http://www.w3.org/Graphics/SVG/Test/.
Examples &
Techniques for checkpoint.
Examples &
Techniques for checkpoint.
Examples &
Techniques for checkpoint.
As described earlier, a Working Group may have different levels of
involvement in test materials development. Nevertheless at any level a
Working Group needs to have clear understanding of what QA framework it will use and how to insure the
quality and usability of the test materials themselves. As a part of the
WG's QA process, before starting a
test materials development, it is recommended the test materials implementer
(Working Group or 3rd party) looks at Test
Guidelines and decide what approach to take for test materials
organization, test criteria, etc.
Checkpoints:
Conformance requirements: the WG MUST ensure that the test framework
contains user documentation that instructs the use of the test materials for
the full range of their intended purposes.
Discussion. Documentation should be thorough enough
that external entities unknown to the developers of the test materials --
such as external certification services -- can effectively and fully use the
test materials.
Examples &
Techniques for checkpoint.
Conformance requirements: the WG MUST describe in its QA Process Document
where, how, by whom, and to whom test materials submissions are to be
made.
Rationale. Clear and simple directions for where and to
whom to submit materials will encourage outside contributions. Explicit
instructions on formats and other technical details will enable the WG to build processes and procedures for the
efficient processing of contributions, whether the contributions are internal
or external.
Discussion. The process document must describe where to
submit test materials and whom to notify (e.g., moderator) of a submission.
The contribution process describes the format of contributed material, it may
contain validation harness, utilities that facilitates tests creation,
templates, etc. A well defined submission process is applicable to both
internal and external contributors of test materials.
Examples & Techniques
for checkpoint.
Conformance requirements: in its QA Process Document the
WG MUST define a submission license
policy applicable to test materials submitted to the WG by external parties, and the submission
license policy MUST include at least an outline of terms, conditions,
constraints, and principles that will govern acceptable submissions to the
WG.
Rationale. Defining submission license policies in
advance will help clarify the WG's expectations to prospective TM submitters,
and will facilitate the efficient negotiation of any needed custom submission
licenses with submitters.
Discussion. Unless exempted by custom submission terms
with W3C Director's approval, a WG's submission license policies will
necessarily conform to standard W3C policies for submitted materials, and
specifically those procedures and terms defined in Contribution
of Software or Test Materials to W3C [CONTRIB].
[@@Ed note. these examples to be
replaced by some from JR.]
Documented examples of TM submission licenses can be seen in the XML Schema submission license, and in
the XML Protocol submission
license.
Related checkpoints. This checkpoint is about the terms under which the WG
accepts contributions to be integrated into its test materials. Checkpoint 6.2 is about
the license(s) under which the WG will publish and distribute its test
materials.
Examples &
Techniques for checkpoint.
Conformance requirements: in its QA Process Document, the WG
- MUST define a procedure for reviewing test materials contributions;
- and the procedure MUST at least address criteria for accuracy, scope,
and clarity of the tests.
Rationale. A well-defined and objective review procedure
expedites the processing of contributions, and facilitates quick and
impartial decisions about their disposition.
Discussion. These procedures are a follow-on and
complement to the contribution
process that WGs must define.
They should at least include criteria for acceptance or rejection of reviewed
tests.
Examples &
Techniques for checkpoint.
Once the test materials (TM) development is in progress, a Working Group
needs to publish the TM drafts and releases, as part of its QA processes.
When the test materials have reached some advanced milestone of maturity
and development (e.g., operationally usable), W3C needs to ensure that:
- their support and maintenance includes secure repository;
- tests materials are freely available for download;
- any licenses do not preclude the test suite from being used by all
interested parties;
- publication of test results is encouraged.
Meeting the needs of TM publication necessarily involves some aspects of
TM management, such as repository. These prerequisite aspects are addressed
in the following checkpoints, as well as publication details proper.
Checkpoints:
Conformance requirements: the Working Group's test materials
MUST reside in repository locations that are secure, reliable, and freely
accessible.
Discussion. This checkpoint does not necessarily
require that test materials should physically reside in the W3C.
Nevertheless, having the repository at W3C is a recommended way of meeting
the checkpoint, because:
- At the point that test materials become operationally deployed, then
challenges to the correctness of both the test materials and
specifications normally increase, and the Working Group is the best venue
for initial processing and adjudication of such queries against both.
- Further to #1, it is more likely that technical report (Recommendation)
errata processing and test suite maintenance can be kept synchronized if
both responsibilities reside in the same body, the Working Group.
It is implicit in the "secure" criterion
of this checkpoint that, if the Working Group ceases to operate, then the
repository arrangements must provide for the continued availability of the
test materials.
Examples & Techniques
for checkpoint.
Conformance requirements: the Working Group MUST define in its
QA Process Document the licenses that are applicable to published test
materials.
Rationale. Any W3C-hosted materials must have approved
license and use terms associated. Because there is no single license that is
appropriate for all test materials, the WG needs consider the options, select
or define a best license for its particular test materials, and clearly
inform potential users.
Discussion. Currently approved W3C licenses that may be
applied to test materials are the Document License and the Software License.
The Document license has two characteristics that are valuable to test
materials:
- It prohibits the test materials from being modified by the user upon
download, therefore guaranteeing the integrity of the test mateials;
- It requires that if the test materials are copied or redistributed,
they must contain the link to the original test materials and their
status.
However, there are situations in which the Document License is
inappropriate, because (for example) some Test Materials require modification
or completion in order to apply them.
Furthermore, there are additional issues about the usability of the
Document License for Test Materials, which would benefit from more
discussion, resolution, and the development of a W3C Test License tailored
specifically to test materials.
Test Materials may contain any of these three components: test software,
test documentation, and test cases. It is possible and sometimes desirable
that the WG apply different licenses to different components. If the WG
considers that neither the Document nor the Software License is applicable,
it should consult with W3C Legal. .
Related checkpoints. This checkpoint is about the license(s) under which
the WG will publish and distribute its test materials. Checkpoint 5.3 is about the terms under
which the WG accepts contributions to be integrated into its test
materials.
Examples &
Techniques for checkpoint.
Conformance requirements: in its QA Process Document the
Working Group MUST document the planned Web location and method for
publication of test materials.
Discussion. If the test materials are to be published
on the W3C site, it is recommended to locate them within the corresponding
activity domain. It is strongly recommended not to publish test materials in
the TR space for the following reasons:
- Documents published in TR space cannot be changed. Repositories with
test material are updated and evolve frequently even after the
specification becomes W3C Recommendation. Changes can be triggered by new
submissions, specification errata's, test errors reports, new versions of
the specification.
- TR space is intended for Technical Recommendations. While test
materials can serve as an extensive set of use cases and help in
interpretation of the related specification, they are not intended to
play the role of Recommendation.
It is recommended to use one of the practices for publishing test
materials, described in Test
Guidelines.
Examples &
Techniques for checkpoint.
Conformance requirements: the Working Group MUST provide a
prominent disclaimer about the use of the test materials for conformance
verification of implementations.
Rationale. It is common to draw unwarranted conclusions
about conformance to the specification from test suite results. A conformance
disclaimer clarifies the relationship between test suite results and
conformance.
Discussion. The central principle that a conformance
disclaimer for test materials needs to clarify for users of the test
materials and consumers of test results is this: passing all tests of the
test materials does not guarantee or imply full conformance of an
implementation to the specification.
Examples &
Techniques for checkpoint.
Conformance requirements: in its QA Process Document the Working Group
- MUST encourage the publication of test results;
- MUST address sample scenarios of where and how such publication can be
done;
- and, MAY identify a WG-sponsored Web site for publishing
collected results or a directory of results.
Rationale. Publishing test results for implementations
of the WG's specifications promotes
interoperability and is helpful to users and vendors alike.
Discussion. Taking responsibility to publish test
results for vendors' products could be a problem for the Working Group. One
way to address the problem is for the Working Group to encourage vendors to
publish results for their implementations themselves. Such publication should
include or describe a test harness that allows reproduction of the results.
The Working Group may want to provide the web space to publish collected
results.
Examples &
Techniques for checkpoint.
A prior checkpoint about test
materials repository (Checkpoint 5.1) recommends that test materials
reside in the W3C. If test development was done outside by some external
entity or group (EG), and the
Working Group (WG) together with
EG decided to move test materials
(TM) to W3C, the following
checkpoints define requirements for the transfer process.
It will be noticed that some aspects of the transfer of a whole test suite
from outside W3C parallel aspects of in-house planning and development, and
piecemeal contribution and review. Those considering such transfers might not
already have the planning, development, and submission infrastructure in
place. Previous transfer experiences have shown that it is useful to isolate
and collect those aspects in one place, as this guideline does.
None of the checkpoints of this guideline are applicable if the Working
Group does not transfer test materials from an external entity.
Checkpoints:
Conformance requirements: as a part of any test materials
transfer process, the Working Group MUST perform and record an assessment of
the quality of the test materials.
Discussion. This checkpoint parallels the checkpoint for the test review
process (checkpoint 5.2). During test materials review, the Working Group
must follow the criteria
defined in the document that is required by that checkpoint.
The details of any quality assessment are necessarily TM-specific -- it
depends the scope & goals of the TM, the type of TM (think "taxonomy"),
etc. However, desirable components of a quality assessment would for example
include attributes such as clarity of scope, traceability, atomicity, user
documentation, etc.
[@@for OPS-EXTECH, a more comprehensive list or
even checklist/template, such as...] "correctness, traceability, atomicity,
user documentation, maintainer documentation, declaration of scope,
completeness (vis a' vis declared scope), harnesses or interfaces for
application of the TM, configurability, results assessment, results recording
& reporting, automation features, versioning/errata support, declaration
of publication licenses, integrated submission procedures, etc."
This checkpoint is not applicable if the Working Group does not transfer
test materials from an external entity.
Examples &
Techniques for checkpoint.
Conformance requirements: as a part of any test materials
transfer process, the Working Group MUST identify and assign staff resources
for the tasks associated with ongoing test materials' development and
maintenance after the transfer, and MUST record that commitment in some
consensus WG document.
Discussion. The WG needs to take care that the amount
of staff resources is adequate to meet needs of ongoing development and
maintenance of the transferred test materials.
Related checkpoints. This checkpoint parallels checkpoint 2.2 for the
specific circumstance of transferring the test materials from the outside to
the W3C.
This checkpoint is not applicable if the Working Group does not transfer
test materials from an external entity.
Examples &
Techniques for checkpoint.
Conformance requirements: as a part of any test materials
transfer process, the Working Group MUST have a documented agreement with the
external entity that covers the IPR aspects that are
applicable to the transferred materials.
Rationale. Completion of suitable IPR agreements for any
transferred test materials, before the transfer is actually completed, will
ensure that the WG can publish and
the public can use the test test materials according terms and conditions
planned and envisioned by the WG.
Discussion. This checkpoint is not applicable if the
Working Group does not transfer test materials from an external entity.
Examples & Techniques
for checkpoint.
The ongoing maintenance of test materials is critical to their long term
integrity, and may require a significant commitment of resources. The
checkpoints of this guideline address planning and resource commitment
requirements to ensure that proper specification tracking happens, and that
maintenance and update procedures are provided for.
If the Working Group is not re-chartered, then it is problematic where the
resources will come from. In this case, it is up to the Director to determine
how the work should be done. Some options include:
- it may be assigned to a Coordination Group;
- it may be assigned to a related Working Group;
- the Team could take care of minor items.
Checkpoints:
Conformance requirements: in some WG consensus document, the Working Group MUST
define a plan and identify resources for the maintenance of the test
materials' contribution and review procedures throughout the entire life
cycle of the test materials and the Recommendation itself.
Discussion. It is implicit in the criteria of this
checkpoint that, if the Working Group ceases to operate, then the WG's maintenance plan must survive and
address the ongoing maintenance of the contribution and review procedures.
Examples &
Techniques for checkpoint.
Conformance requirements: in its QA Process Document the
Working Group MUST specify procedures to update the test materials to track
new specification versions and errata levels.
Rationale. Defining tracking procedures for
specification versions/errata will help to ensure that all needed versions of
test materials are available. This is critical to product version tracking of
specification versions. It enables products' users to better assess the
quality and conformance of the individual product versions.
Discussion. As used in this checkpoint, the scope of
"versions" at least includes "editions" -- editorial revisions that
incorporate errata and small changes -- rather than major versions with major
functional increments.
Related checkpoints. This checkpoint is about defining maintenance
procedures for test materials to track specification versions or errata
levels. The related checkpoint 3.2 concerns support
functionality for versioning/errata in the test materials' infrastructure,
framework or management system.
Examples &
Techniques for checkpoint.
Conformance requirements: in its QA Process Document the
Working Group MUST identify the communication channel for appeals of test
validity and a procedure for resolving such appeals.
Rationale. A well-defined and deterministic procedure
for handling challenges will expedite the resolution of challenges, and is
critical to their defensibility and acceptance.
Discussion. Such a procedure is applicable both to
external challenges once test materials have been published, as well as to
internal challenges during the building of test materials.
Examples &
Techniques for checkpoint.
The QA Activity works closely with other W3C WGs, providing assistance and expertise in
helping them achieve their QA goals and deliverables. The QA Activity
anticipates different relationships with the various WGs depending on the QA-specific needs of the
particular Working Group. For example, the resources and experience of
Working Group members as well as their stage in the Recommendation track may
influence the type and level of collaboration between the Working Group and
QA Activity. Potential relationships between Working Groups and the QA
Activity include:
- liaison and consultation
- active reviews
- adjudicate entry and exit criteria
- QA resources supplement
- resolution of external QA requests.
More details about the ways in which the QA Activity may interact with and
assist the W3C WGs may be found in
the QA Working Group Process Document [QAWGPD].
This section defines conformance of Working Group processes and operations
to the requirements of this specification. The requirements of this
specification are detailed in the checkpoints of the preceding "Guidelines" chapter of this
specification, and apply to the Working Group QA-related documents and
deliverables required by this specification.
The following sections are normative in this document:
- all the sentences using a capitalized keyword from RFC 2119 (typically,
one paragraph per checkpoint)
- the non-applicability clauses of each checkpoint
- the conformance section (this section)
Any other section not explicitly marked as normative is assumed to be
informative.
This specification is extensible. That is, the Working Groups MAY set up
quality-related processes and operations in addition to those required for
conformance to this specification. Extensions to this specification MUST NOT
contradict or negate the requirements of this specification.
For each degree of conformance claimed (I.e., A, AA or AAA), it is
allowable to implement more than the checkpoints required to satisfy that
degree of conformance. This may be achieved by either satisfying some, but
not all of the checkpoints of the next degree of conformance or by
implementing additional conformance related features beyond what is specified
in this document. For example, claiming to be A-conforming but also
satisfying some of the Priority 2 checkpoints and some Priority 3
checkpoints.
The rationale for allowing Working Groups to define extensions to these
operational guidelines is that these requirements are considered to be the
minimal requirements for successful quality practices within the
WGs. Doing more than the minimum is
not only acceptable, but beneficial. Extensions also allow Working Groups
tailor their quality practices more closely to their specific needs. The
guidelines of this specification may not be sufficient to meet the needs of
all WGs.
Within each prioritized checkpoint there is at least one
conformance requirement, and there may be more than one.
These are prefaced by "Conformance requirements:", highlighted in a different
style, and contain at least one of the RFC 2119 [RFC2110] keywords (MUST, SHOULD, MAY, etc).
This Operational Guidelines document does not enumerate a list of test
assertions. A test assertion can be derived from each "MUST" requirement in a
straightforward manner.
This section defines three conformance levels to this
guidelines specification:
- A-Conforming: all Priority 1 checkpoints are
satisfied;
- AA-Conforming: all Priority 1 and Priority 2
checkpoints are satisfied;
- AAA-Conforming: all Priority 1, Priority 2, and
Priority 3 checkpoints are satisfied;
A checkpoint is satisfied by satisfying all of the individual conformance requirements.
Failing one individual mandatory requirement means that the checkpoint is not
satisfied. Mandatory requirements are those that use the conformance keywords
"MUST", "MUST NOT", "REQUIRED", "SHALL", or "SHALL NOT".
A Working Group conforms to the "QA Framework: Operational Guidelines" at
degree X (A, AA, or
AAA) if the Working Group meets at least all degree
X conformance requirements.
An assertion of conformance to this specification -- i.e., a
conformance claim -- MUST minimally specify:
- The name and URI of the working group
- The guidelines title: "QA Framework: Operational Guidelines"
- The guidelines dated URI:
http://www.w3.org/TR/2002/WD-qaframe-ops-20021108/
- the date of the claim
- The degree of conformance satisfied: A-Conforming, AA-Conforming, or
AAA-Conforming.
The checklist for this specification ([OPS-CHECKLIST]) is the Implementation Conformance
Statement (ICS) pro-forma for
this specification. Any assertion of conformance to this specification MUST
link to a completed ICS.
Example of a valid conformance claim:
The QA processes and operations of this FooBar Working Group,
http://example.org/FooBar/, conform to W3C's 'QA Framework: Operational
Guidelines', available at
http://www.w3.org/TR/2003/WD-qaframe-ops-200302110/, on 23-April-2003,
AA-Conforming. The Implementation Conformance Statement for this claim is
available at http://example.org/FooBar/Test/OpsGL-ICS .
The checkpoints of this specification present verifiable conformance
requirements about the operational aspects of Working Group quality
processes. As with any verifiable conformance requirements, users should be
aware of this disclaimer:
Passing all of the requirements to achieve a given conformance degree --
A, AA, or AAA -- does not guarantee that the subject operations and
processes are well-suited to or will achieve their intended purposes, nor
does it guarantee the quality or suitability of test materials produced
under the processes.
The following QA Working Group and Interest Group participants have
contributed significantly to the content of this document:
- Daniel Dardailler (W3C),
- Dimitris Dimitriadis (Ontologicon),
- Karl DuBost (W3C),
- Peter Fawcett (RealNetworks),
- Kirill Gavrylyuk (Microsoft),
- Dominique Hazael-Massieux (W3C),
- Lofton Henderson (CGM Open),
- Ian Jacobs (W3C),
- David Marston (Lotus Development Corp),
- Sandra Martinez (NIST),
- Patrick Curran (Sun Microsystems),
- Lynne Rosenthal (NIST),
- Mark Skall (NIST),
- Andrew Thackrah (Open Group),
- Olivier Thereaux (W3C),
- PROCESS
- World Wide
Web Consortium Process Document, I. Jacobs, Ed., 19 July
2001, available at http://www.w3.org/Consortium/Process-20010719/ .
- RFC2119
- Key words
for use in RFCs to Indicate Requirement Levels, March 1997,
available at http://www.ietf.org/rfc/rfc2119.txt .
- CONTRIB
- Contribution
of Software or Test Materials to W3C, which defines
W3C-approved procedures and terms for submission of Software and Test
Materials, available at
http://www.w3.org/Consortium/Legal/2002/contribution-grant-20021231
.
- EXTERN-TA
- QA activity email thread about third-party
participation in test materials production, available at
http://lists.w3.org/Archives/Public/www-qa/2001Oct/0060.html .
- MATRIX
- W3C-wide conformance activity survey covering all the Working Groups,
"The
Matrix", available at http://www.w3.org/QA/TheMatrix .
- TAXONOMY
- QA Activity test taxonomy, a
classification scheme for conformance test materials, available at
http://www.w3.org/QA/Taxonomy .
- OPS-CHECKLIST
- Appendix to this operational guidelines document that presents all
checkpoints in a table sorted by guideline.
Available at
http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/qaframe-ops-checklist
.
- OPS-ICS
- Appendix to this operational guidelines document that presents all
checkpoints in a table
sorted by Priority. Available at
http://www.w3.org/TR/2003/WD-qaframe-ops-20030210/qaframe-ops-ics .
- OPS-EXTECH
- QA Framework:
Operational Examples & Techniques, L. Henderson, L.
Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Note, (initially) 10
February 2003, companion version to this document, latest companion
version available at http://www.w3.org/QA/WG/2003/02/qaframe-ops-extech
.
- PROTOCOL-WG-TS
- The XML Protocol WG has a TS process
document, available at
http://www.w3.org/2000/xp/Group/1/10/ts-contribution, and a contribution/submission
license (example of a submission legal notice), available at
http://www.w3.org/2001/10/test-materials-license.html .
- QA-GLOSSARY
- A comprehensive glossary of QA terms, maintained by the QA Working
Group. (Initial version under construction.)
- QAF-INTRO
- QA Framework:
Introduction, L. Henderson, K. Gavrylyuk, D. Dimitriadis, L.
Rosenthal, Eds., W3C Working Draft, 10 February 2003, companion version
to this document, available at
http://www.w3.org/TR/2003/WD-qaframe-intro-20030210/ .
- QAF-SPEC
- QA Framework:
Specification Guidelines, D. Hazaël-Massieux, L. Henderson,
L. Rosenthal, D. Dimitriadis, K. Gavrylyuk, Eds., W3C Working Draft, 08
November 2002, companion version to this document, available at
http://www.w3.org/TR/2003/WD-qaframe-spec-20030210/ .
- QAF-TEST
- QA
Framework: Test Guidelines, K. Gavrylyuk, D. Dimitriadis, L.
Henderson, M. Skall, P. Fawcett, Eds, W3C Working Draft, 20 December
2002, companion version to this document, available at
http://www.w3.org/TR/2002/WD-qaframe-test-20021220/ .
- QAIG
- Quality Assurance
Interest Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/IG/ .
- QAWG
- Quality Assurance
Working Group of the W3C QA Activity, which may be found at
http://www.w3.org/QA/WG/ .
- QAWGPD
- QA Working Group
Process Document, defining both the general and QA-specific
processes of the QA Working Group, available at (under
construction).
- DOM
Working Group TS
- DOM
TS Process Document, D. Dimitriadis, Ed., 15 January 2002,
available at
http://www.w3.org/2002/01/DOMConformanceTS-Process-20020115 .
- REC-TRACK
- Stages and milestones in the W3C
Recommendation Track, per the Process
Document (Process Document is available at
http://www.w3.org/Consortium/Process-20010719/, see section 5.2).
- SCHEMA-WG-TS
- The XML Schema WG has a TS
process document, available at
http://www.w3.org/2001/05/xmlschema-test-collection.html, and a contribution/submission
license (example of a submission legal notice), available at
http://www.w3.org/2001/05/test-materials-license.html .
- WCAG10
- Web Content
Accessibility Guidelines 1.0, W. Chisholm, I. Jacobs, G.
Vanderheiden, Eds., W3C Recommendation, 5 May 1999, available at
http://www.w3.org/TR/WCAG10/ .
- WG-QA-RANGE
- Email
proposal by David Marston, on the QA public mail list, for range of
Working Group commitment levels to conformance test materials
production, available at
http://lists.w3.org/Archives/Public/www-qa/2001Apr/0004.html .
- 2003-05-14, WG-discussion Working Draft
Clarifications and improvements in response to Last Call
comments.
- (Skeleton) new Introduction section -- table or diagram -- about
association of guidelines to stages in WG's life and
deliverables.
- Rewrote CP1.1, 1.2, 1.3 to preserve all of LC-version normative
requirements, but eliminate the table and (implied) conflicting
conformance scale, and integrate the commitment definition with
other GL documents.
- Added clarifications to GL1/2 verbiage, to emphasize (ideally)
that these are Charter-phase activities.
- To CP1.5 and CP3.1, added reciprocal clarifications to
disambiguate the similarity between the two.
- To CP2.2, CP4.1 and CP4.2, added reciprocal clarifications to
disambiguate the similarity between them.
- Raise CP3.2 from P3 to P2, clarify "versioning", clarify
relationship to CP8.2.
- Revise CP4.3 verbiage for clarification, and (LC-58) to allow a
"TOC" to satisfy the QAPD.
- Cross-reference and clarify relationship of CP5.3 and CP6.2,
revise verbiage per consensus with W3C Legal.
- Clarify CP4.5 (esp. the ambiguous use of "the QA framework"), and
its motivation/rationale.
- Moved some verbiage out of CP6.1 into GL6, clarified about
maintenance & publication.
- Added some clarification and rationale to verbiage of GL7.
- Priority changes: CP3.2 from P3 to P2; CP8.2 from P2 to P1; CP4.5
from P1 to P2.
- Improved testability of the language in CP3.1, CP3.2, CP6.1,
CP4.2, CP2.2, CP6.3, CP7.1, CP7.2, CP7.3.
- Change CP4.5 from P1 to P2.
- Change CP8.2 from P2 to P1, CP3.2 from P3 to P1, and clarify the
requirements and relationship between the two checkpoints.
- 2003-02-10, Last Call Working Draft
Revised Introduction, Conformance chapters for better conformance
to SpecGL. Restructure all checkpoints for better verifiability. Add
rationale to many checkpoints. Implement resolution of all closed OpsGL
issues. Editorial improvements.
- Finish "add Rationale" (below) for those CP which need one
- extensive editorial improvements to wording.
- In all checkpoints, changed "To fulfill..," verbiage to
"Conformance requirements:"
- Changed "Test assertions" subsection of Conformance chapter to
"Conformance requirements & test assertions", revised its
wording (no test assertions).
- Replace most of Ch.3, " WG relationship to QA Activity",
with a reference to the QAWG Process Document.
- Removed Sec 1.3 (Goals of QA Development), revised 1.1, 1.2,
1.4.
- Add to CP5.3 (submission license policy) "Discussion" a reference
to W3C-wide submission policies.
- Modified CP6.2 (TM License) to subsume a special TM License under
"W3C-approved custom license", while discussion with W3C legal is
still pending.
- Clarified wording of CP3.2 (spec versioning/errata support in
TM).
- Added "Rationale" to all checkpoints for which it was
missing.
- Revised "commitment table" of CP1.1 to clarify levels and add
conformance criteria.
- Clarified fulfillment criteria of CP3.2.
- Remove 2nd part ("sufficient" clause) of Conformance
Disclaimer.
- Revised checkpoint wording and fulfillment criteria of CP8.1
- edited Discussion of several checkpoints, added Rationale to a
couple (CP4.3, 7.3).
- revised Conformance Section to comply w/ SpecGL (extensibility,
normativity, conformance claims, etc).
- revised Introduction to comply w/ SpecGL -- added scope, class of
product, etc.
- CP5.3, replace requirement for license with requirement for
license policy.
- CP5.3, add links to a couple of sample submission licenses.
- separate the normative references from the informative references
(and label them).
- add links to sample QA (TS) process documents to GL4
verbiage.
- CP2.3, change requirement on existing WGs.
- CP6.4, the disclaimer must prominent.
- CP6.5, adjust the "may" usage in the fulfillment criteria and
discussion.
- clean up discussion in CP5.1
- qualify "public, archived" for mailing list and "public" for
/Test/ Web page.
- revise GL3 and CP3.1 to explicitly link QA deliverables to
specification versions.
- change post- WG
"maintenance" comments in CP6.1 to repository.
- include post- WG
maintenance comments in CP8.1.
- clarify CP4.6 (branding), and stipulate "not applicable" if
branding is not supported.
- remove "discussion" from list requirements in CP4.4, clarify
requirements.
- clarify minimal QAPD contents in CP4.3 and enumerate them in the
discussion.
- move content from CP4.1 to Ops Extech, leave brief summary in its
place.
- changes to wording in QA commitment table of GL1
- 2002-11-08, third published WD
-
Revised Introduction, Conformance for better conformance to SpecGL.
Implemented other minor editorial and resolved-issue changes. Make over
all checkpoints for better verifiability.
- (since 11/03) Moved a handful of examples to Ops Extech.
- (since 10/31)
- Make over of GL3-GL8, to isolate the "to fulfill" criterion
and apply TA markup;
- Numerous minor editorial (wording & grammar)
improvements.
- Miscellaneous editorial corrections, fix missing anchors,
etc.
- Combine CK5.5 into CK5.4 as new MUST requirement.
- In GL1 and GL2, apply test assertion markup, to identify
fulfillment criteria.
- In GL2, remove the "In the charter" bits of all checkpoints, and
embed old/new requirements in the fulfillment criteria.
- In GL1, remove the "In the charter" bits of all checkpoints, and
embed old/new requirements in the fulfillment criteria.
- Identify the checklist as the ICS
pro-forma for OpsGL.
- Revise "Conformance" chapter per SpecGL (level/degree, test
assertions, etc)
- Rewrite descriptions of priority 1, 2, 3 (subsection of
Intro)
- Clarify RFC2119 usage ("Terminology" subsection of Intro)
- Reorganize and augment Introduction, w/ subsections for:
scope&goals, motivation, understanding&using, priorities,
etc.
- 2002-05-15, second published WD
Checkpoints numbering is changed, changes list refers to the
numbering from the previous WD
[1].
- Ck 4.1 - split into 2 having QA moderator as Pri1 and the task
force as Pri2
- Ck 4.4 - advanced to pri1
- Ck 4.5 - removed "if applicable"
- Ck 5.1 - advanced to Pri1, remove accredited, changed to "usable
for intended purpose"
- Ck 5.3 - advanced to Pri1
- Advanced Ck 6.1 to Pri1, removed "future", add "freely
available"
- Ck 6.2 - advanced to Pri1
- Ck 6.4 - advanced to Pri1.
- Ck 6.5 reworded checkpoint and explanation text. advanced to
Pri2
- Ck. 7.1. Added a note on how this checkpoint is related to 5.4
"review process". Clarified what is the assessment quality by
referring to the criteria for the review process.
- Added Ck 5.5 with Pri1 as a criteria for the review process -
needed for a reference from other places where we talk about "tests
quality assessment criteria". Pri1 is needed to match the minimum
requirement in the WG QA
commitment.
- Ck 7.2 - added note on how this relates to the Ck 2.2 and moved
it to Pri1
- Ck 7.4 - removed
- Ck 8.1 - reworded.
- Ck 1.1 - changed the commitment table based on Mark's
suggestions: added requirement for some test assertions to the
level 2, added requirement for tests review to the Level 3, swapped
level 5 and level 6.
- Added 2 more checkpoints in advancement of Ck 1.1: with pri2 to
commit to the level 5 and with pri3 to commit to the level 7.
Explanation notes are there
- Added first draft for Definitions of the Ck Priorities to the
Introduction section.
- Added Conformance disclaimer subsection drafted by Lofton, per
WG decision.
- Other rewrites and small editorial changes.
[1] http://www.w3.org/QA/WG/2002/framework-20020311/qaframe-ops
- 03-11-2002
Checkpoints numbering is changed, changes list refers to the
numbering from the previous WD
[1].
- Incorporated Lynne's contribution - Ch3 rewrite
- add checkpoint for soliciting QA resources in Call for
Participation with P2
- minor rephrasing of statements of chkpt 1.1, 1.2, and 1.3
- rephrase "CR-exit" discussion
- Changed chkpt 2.2 from [1] to have P1
- Changed chkpt 1.2 from [1] to have P1
- Removed 3.1, added more explanation notes to 3.2 (P2) and 3.3
(P3)
- Removed the checkpoint 4.1 from [1] and add it to the Intro
section
- Removed the checkpoint 5.1
- Added more explanation, clarified the difference between chckpt
6.3 and 6.5 from [1]
- Added more explanation, clarified the difference between 5.3 and
5.5 from [1]
- Added text to explain why the WG SHOULD NOT publish test
materials in TR space
- Added text to explain why the Document License is recommended
- Added mild disclaimer on the use of the "SHOULD", "MUST", etc
keywords to the Terminology section
- Other rewrites and small editorial changes
[1] http://www.w3.org/QA/WG/2002/02/qaframe-ops-0225.html
- 02-28-2002
- Major rewrite of the previous draft. Incorporated Lynne's comments
about Gd1 and Gd4, split Gd regarding QA process, added bunch of new
checkpoints.