This is the first (derived from old QAF-Intro, via FPWD QAH Section 6).
Copyright © 2004 W3C ® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
QA Framework Roadmap is an informational supplement to The QA Handbook [QAF]. It provides a general orientation to the whole of the QA Framework (QAF), in the form of usage scenarios, primer material, and target audience identification by WG role.
This is a QAWG Editor's development draft. It has no status or standing, and does not necessarily represent QAWG consensus.
For now, it has only been separated from QA Handbook [QAH], but not yet edited for changes such as the back-burner status of TestGL.
It is uncertain whether this will be progressed as a WG Note (companion to QAH in /TR/), or simply maintained as a QA Web page (outside of /TR/). The following needs work, depending on that decision...
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This document is a W3C Working Group Note, made available by the W3C Quality Assurance (QA) Activity for discussion by W3C Members and other interested parties. For more information about the QA Activity, please see the QA Activity statement.
This is the FPWD of this document. This material was originally developed for the now-discontinued QA Framework: Introduction, @@QAF-INTRO@@, and due to positive endorsement was subsequently extracted and preserved as an appendix within FPWD of The QA Handbook [QAH].
The QAWG does not expect this document to become a Recommendation. Rather, after further development, review and refinement, it will finally be published and maintained as a Working Group Note.
Publication as a Working Group Note does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
The QA Working Group Patent Disclosure page contains details on known patents related to this specification, in conformance with W3C policy requirements.
Comments on this document may be emailed to www-qa@w3.org, the publicly archived list of the QA Interest Group. Commenters please note that your comments will be publicly archived and available, do not send information that should not be distributed, such as private data. Please give any feedback by 12 June 2004, so that it may be considered by QAWG at its next face-to-face meeting.
-- Here repeat the Abstract
-- Here list QAH, QAF-SPEC, and QAF-TEST. With caveat about QAF-TEST progression.
The following list identifies the parts of the The QA Framework (QAF) that might interest those who fill the various roles within a Working Group. While each part of QAF is targeted itself at a specific principal audience, the various parts might have somewhat broader interest and applicability.
[TBD. Could this section maybe be simplified further, into a checklist of sorts (checklist to either replace or supplement the fuller prose? ]
[TBD. This section will need close coordination especially with Test Guidelines, when that is first published.]
This chapter, in the style of a primer, introduces the use of the QA Framework document family. It progresses through significant milestones in a Working Group's life, from writing a Charter through publishing Recommendations, and looks at associated test suite and other quality practice activities.
Because QA is properly an integral part of the activities of each Working Group, each Working Group has to consider and commit to a set of QA deliverables appropriate to its work. A spectrum of possibilities are discussed and illustrated in the early planning and commitment (@@fix link@@) module of The QA Handbook.
If a Working Group is being newly formed, and if the it is able to anticipate and agree at Charter time on deliverables like test suites, then it should consider documenting those QA deliverables in its Charter, just as it does all other deliverables. Again, see the early planning and commitment (@@fix link@@) section of The QA Handbook. A Working Group being re-chartered is a similar case to a newly formed group, although the scope and direction of its work might be clearer.
For an already-chartered Working Group undertaking new test and other QA-related activities, if these deliverables are not documented in the Charter already, then there are a couple of options. The W3C Process [PROCESS]describes how to amend a Charter to accommodate significant new deliverables, if it wishes to take this route.
Once the Working Group is off and running, and assuming that it has planned on some test- or other quality-related deliverables, the next step is to chose and document the processes and logistics that it will use for its QA activities. These include such typical details as:
The sections within the Day-to-Day operations (@@fix link@@) module of The QA Handbook give good-practice advice about how to do this, plus examples and a handy template for writing a QA Process Document.
There is a tight bond between how the specification (Recommendation) is written on the one hand, and on the other hand its testability and its suitability as a basis for interoperable implementations.
New specification. QA Framework: Specification Guidelines [QAF-SPEC] should be applied from very beginning. Among the key topics that it addresses are:
Consider the advice of QA Framework: Specification Guidelines [QAF-SPEC] even at the stage of planning the structure and presentation style of the spec. Along with W3C "pubrules" and W3C Manual of Style, spec authors and editors should refer to the spec guidelines throughout their work, on topics like testable language, clarity, conciseness.
New Edition of specification. A new Edition of the same functional level of a specification is typically used for incorporation of errata (e.g., XML 1.0 Second Edition). Normally, this should not be considered a good time to align a specification to QA Framework: Specification Guidelines [QAF-SPEC] — the changes associated with such alignment could significantly disrupt and restructure the specification.
New Version of specification. A new Version of the specification refers to a significant functional change and enhancement. This presents a good opportunity to improve the testability and implementability of the specification, as just described for new specifications.
This stage in the specifications life has two significant aspects:
When the specification is published in TR space, then W3C Members not participating to the Working Group and the general public begin to review and comment. It would be valuable that such reviewers consult and understand the material in QA Framework: Specification Guidelines [QAF-SPEC] — it gives and informed set of evaluation criteria about the conformance, testability, and interoperability aspects of the specification.
Working Group participants and especially Test Task Force participants should refer to the good-practice pieces about advancement criteria and synchronization (@@fix link@@) (between specs and test materials) of The QA Handbook. Projects enter The Matrix [MATRIX] at Last Call Working Draft (if not sooner). A de-facto process convention is emerging, that there should be significant conformance test materials before finishing Candidate Recommendation phase. This timing coordinates with the explicit process requirement of two interoperable implementations.
There are several scenarios for how the Working Group "builds" its conformance test materials:
Intra-group build. Before starting the development, the Test Task Force participants would benefit from a familiarity with the material in QA Framework: Test Guidelines [QAF-TEST]. There is useful information for both high-level planning — e.g., does the Working Group want breadth-first Basic Effectivity or a fully detailed suite? — as well as specific details for building the individual test cases. Another aspect of building test materials is an acceptance procedure for the individual bits, as they are built. This is addressed in the review-procedures guidance of QA Framework: Test Guidelines [QAF-TEST].
Import completed test materials. Several high-quality test suites have been developed outside of the relevant W3C Working Group, and then transferred to the group. Groups which are considering such a transfer should refer to test materials acquisition (@@fix link@@) module of The QA Handbook. Clearly, the quality of the candidate test materials should be carefully assessed, and for this the Test Guidelines can provide useful assessment criteria.
Assemble contributions. Some test suites have been built by implementing processes to assemble significant chunks of material from outside (or internal participant) contributions. Again, The QA Handbook test materials acquisition (@@fix link@@) module addresses the steps needed to complete such a transfer — they are the same as the preceding paragraph about transferring completed test materials. In addition, there should be careful quality assessment of contributions, for which QA Framework: Test Guidelines [QAF-TEST] can be helpful.
Typically, a Working Group Test Task Force will want to publish releases of test materials, particularly as the specification advances through its final maturity levels (e.g., Proposed Recommendation) towards Recommendation. Test material publication is addressed in QA Framework: Test Guidelines [QAF-TEST].
One hurdle on the way to publication is legal — deciding and agreeing on suitable publication licenses. Advice on navigating this potential quagmire is presented in the licensing module (@@fix link@@) of The QA Handbook.
As soon as the test materials become public, then there is definite need for a procedure to process challenges to correctness, make determinations, and appeal decisions. Test appeal process advice is presented in QA Framework: Test Guidelines [QAF-TEST].
Publication of test materials often comprises an implicit (or explicit) invitation for contributions. The considerations described in "Assemble Contributions" (@@fix link@@) are equally applicable here.
When the specification reaches Recommendation, there is typically a concurrent publication of the test materials. This might be considered a "final" publication, or ongoing development may still be planned according to one of the mechanisms discussed above. In any case, a maintenance procedure must be in place for the test materials. Firstly, there are tie-ins between approved specification errata and applicability of particular tests — mechanisms for this are discussed in QA Framework: Test Guidelines [QAF-TEST]. Secondly, there is the above-discussed need for both challenge/review/appeal processes. Finally, even if the Working Group ceases active development of test materials, it may want to continue to consider submissions, and review and integrate them per the test-contribution advice of QA Framework: Test Guidelines [QAF-TEST].
It is possible that the Working Group and its Test task force may disband after its charter expires. The various aspects of this situation are introduced (@@fix link@@) in The QA Handbook, and dealt with in somewhat more detail in QA Framework: Test Guidelines [QAF-TEST].
The following QA Working Group and Interest Group participants have contributed significantly to this document: