Caveat: this was discussed at 20040329 telecon. Some questions were answered and changes agreed that are not yet reflected in this outline.
Editor's notes to self: the following subsections are for editor's (my) benefit for now -- context, audience, goals & purposes, requirements. Later, some of this might migrate into QAH. But for now I've written them out a little bit, just to help keep focus in the following outline(s).
The QA Handbook (QAH) combines the best features of the former QAF Introduction and QAF Operational Guidelines into a non-normative handbook about the process and operational aspects of the quality practices of W3C's Working Groups. The aim is to flag avoidable pitfalls, and to provide techniques, tools, and templates that will facilitate and accelerate the work of the WGs.
QAH is intended for: chairs and staff contacts of W3C Working Groups. Many of these are savvy about W3C process and culture, and the realities of running a W3C Working Group; a few are novices.
To a large extent, these folks share the same concerns and pressures of spec editors as articulated in this Wiki page.
Therefore, the QAH looks at things that impact these aspects of WGs' quality practices:
To a lesser degree, the QAH will introduce the other QAF components and other QA resources.
Issue: to what degree should QAH introduce the rest of QAF?
Look at the requirements listed in QaFramework/SpecGlUsability (Wiki page). I'll initially assume that those requirements applicable to editors also apply to the QAH chairs/staff audience. (TBD -- let's eventually review each of those, and also consider whether there are others to add).
Problems to solve: 3 Ops problem topics are/were: not enough (motivated) staff; stuff being done too late; licenses.
Overview of outline: I see QAH dividing naturally into a few main pieces. Tbd whether these are sections of one document, independent documents, etc.
So there's an introduction/roadmap and 4 modules. Each of the modules, #2 - #5, should consider for inclusion those bits that are applicable in the following "proto-module"...
Actual Topic/Principle/Module title would replace "Proto-module".
[Here at start are some temporary editor's comments about how module relates to previous Ops GL/CP, etc.]
First issue: do we want QAH's intro sections to introduce: all of QAF? or only that old-OpsGL process & operational stuff that will survive in QAH?
[Introductory and Roadmap module]
[The first piece of QAH is the introduction. It is a quick roadmap that clarifies who this is for, what it aims to do, and what the remaining bits are. It might contain the some parts of old QAF-intro, like modified version of the usage scenarios. Might contain a modified/simplified OpsGL chronology diagram ???]
Issue: would this section itself be better as an external, referenced document? It doesn't involve too much content from QAF-intro, so it could go either way -- in-line or external.
The rest of the pieces of QAH would be something like 3-4 operational modules that are derived from old OpsGL/ET material. For each one, we should consider inclusion of each of the components of the following proto-module (module template)...
[Operational module #1. Subservient to and one level below the Roadmap.]
[Overview of what to put into this module: implement QAWG's general consensus (see f2f minutes) that old-CP1.4 is the important one, and the other CPs of GL1 & 2 are details. CP1.5 (Rec-track criteria), CP2.1 (internal or external?), CP2.3 (request alloc of people for QA) are worth mentioning and should survive as details ... "Good practice" or "Recommended practice" or ..." CP2.2 (staff commitments) could be dropped, ditto for CP1.1 (A/AA/AAA to Ops, Spec, Test). CP1.2 (commit to TM) and CP1.3 (commit to complete TM) seem like details. See also checklist.]
Possible story (changed name): Xyz10 launched a intense test effort late in the spec development. During CR, the still-growing Test Suite kept uncovering ambiguities and flaws in the spec. XYZ10 cycled three times in CR. For XYZ20, the WG decided up front that any functionality proposed for inclusion had to be accompanied by test cases. [...sadly, story goes on... everyone felt that they didn't have time to do it, so they ignored the rule. Approaching LC, there is no test suite... Nice thought tho'...]
Principle: Plan for and commit to your QA deliverables as early as possible.
What does this mean? In theory it means [...tbd...]. In practice, it means attention to a handful of points:
Why care? It's not always easy to anticipate your needs during the rush and pressure of Chartering. But the more you can do so, the less likely that you'll get big surprises later on, or run into problems that will delay your progress towards Rec.
Examples. [Here, or in the above list, or ???]
How can I do it?
Related:
(or, Defining and documenting your quality processes.) (or ...)
Ed note. Still in progress defining the details here. Will identify most of existing QAPD items. But since TestLite is going to take a bunch of 'em, will need to figure out how to interface with it on those that straddle the respective turfs.
[Operational module #2. Subservient to and one level below the Roadmap.]
[Overview of what appears in this module:old-CP4.1 (qa moderator); drop CP4.2 (appoint a task force); *** CP4.3 (QAPD) is central***; CP4.4 (email & web); CP5.4, CP6.2 (license stuff) and maybe CP4.5 (branding) might be documented in QAPD, but reference to IANAL/license module for details; CP6.1 (repository) drop or roll into Web site stuff; CP5.1, CP5.3, CP5.5 (TM-related) might be documented in QAPD, but reference to @@TestLite@@ for details; CP6.3, CP6.5 (TM-related) are unclaimed by TestLite, but there's an issue whether they ought to be -- might be documented in QAPD, but reference to @@TestLite@@ for details. CP8.1, CP8.2, CP8.3 are about TM maintenance -- like some other TM ones, they might get primary treatment in @@TestLite@@ but be cross-referenced from here and documented in QAPD. See also checklist.]
Possible story. After doing it up front in one of its first TM development efforts, a test development team was so convinced of the value of a good QAPD that, in its several subsequent W3C test development efforts, one of the first things it did was copy-paste-edit a previous QAPD for the new effort's QAPD.
Principle. In one place, collect all of the information needed to define your work process, inform others how to communicate with you on QA topics, and direct everyone how to find and use your QA resources. A QAPD is an easy way to do this.
Why care? Over and over we hear the message from successful WG test suite projects -- define work process: process document, framework, issues list, email list, faq [Ex. DOM, Xquery, ...] Conversely, there are examples of disorganized collections of test cases with no one apparently in charge, no way to find their status, correctness, (KD's example of "can't find anyone who's responsible for amorphous undocumented heap of "tests".)
What does it mean? In theory it means [...tbd...your QA work processes need to be clear to people need to know how to communicate with you, where to look for your suites, how to get and use them, etc.].
In practice, it means (easiest solution) producing a QAPD [CP4.3] recording the details of your QA-related work processes: [this section should use lots of examples]
Your QAPD is also a handy central location to record your licensing and branding policies (if any):
If you are developing test materials, then there are several topics associated with TM development process that you will find described in some detail in @@TestLite@@, but which might conveniently be documented in your QAPD:
Finally, if you have started to think yet about "life after WG", then you will need to decide what happens to both your test materials and associated processes. You will find these maintenance-related topics described in some detail in @@TestLite@@, but you might conveniently be locate the information in your QAPD:
[Issue: is this an okay approach in the last two bullet lists, i.e., mention the topics, suggest that they can be documented in QAPD (they're in template now), and point to TestLite for details?]
How can I do it? Simple: Use QAPD template. It will guide you through everything you need, and then some. Or copy and edit one of these @@examples@@.
Related:
Success criteria:
[Operational module #3. Subservient to and one level below the Roadmap.]
[Overview of what to put into this module: all the useful stuff associated with CP5.4 (submission licenses) and CP6.2 (publication licenses). See also checklist.] [Issue: Should CP4.5 (branding policy) be in here also? It does tend to get entangles with IPR, enforcement, etc whenever we talk about it.]
Cartoon? [Anyone have any good lawyer cartoons/jokes?]
Possible story. Xxxxxy WG has a great collection of test cases, but they can't agree now on publication licenses that are acceptable to everyone, and so they are stalled while they work it out [in CR? (ask MB who this is and what the details are].
Principle. Get agreement up front about License issues around your planned test materials.
What does this mean? Theory: submitted materials (by default, more or less routine W3C submission license); and published materials. Note possibility of subdividing and applying different sorts of licenses. In practice:
Why care? Get it right early, or it may stall your Rec-track progress indefinitely. While this might seem to be a routine piece of Day-to-day operations, it has proved to be sufficiently problematic within W3C that it deserves to be a Principle by itself..
How can I do it?
Related:
[Operational module #4. Subservient to and one level below the Roadmap]
[If both of these survive the cut, they could: each be in a module of their own; or a miscellaneous module with a couple of principles; or...other? If both are dropped, problem solved. If one goes away, problem solved -- re-title the module appropriately.]