QA Handbook FPWD Issues

Reference: The QA Handbook(First Published Working draft)

Table of Contents

• Dan Connolly
• Jeremy Carroll
• Dominique Hazaël-Massieux
• Other:  terminology like "QA deliverables"
• Other:  LH-identified issues in the QAH

Dan Connolly

http://lists.w3.org/Archives/Public/www-qa/2004May/0043.html

Synopsis:  likes overall direction; suggests that "IPR" should be replaced with clearer, more specific terms like patent, copyright, trademark, trade secret, or else with "legal issues" where a generalization is needed.

Proposed resolution:Agreed.

Resolution:The occurrences of "IPR" will be changed to something more specific. In particular, in the QAH context, QAWG thinks that "license" or "licensing" is appropriate in each case. As an aside, the QAWG was not unanimous that "IPR" should always be avoided. In particular, when we searched for guidance in other W3C documents, we found that there are numerous occurrences in the W3C Process Document.

Jeremy Carroll

http://lists.w3.org/Archives/Public/www-qa/2004May/0044.html

Synopsis:  looks very good, approve overall direction, detailed comments to follow

http://lists.w3.org/Archives/Public/www-qa/2004Jun/0000.html

Synopsis:  Numerous detailed comments, enumerated in the following subsections:

Overview of comments

This is a personal review of the QA Handbook WD dated 10 May 2004.

Overall I am happy with this WD and believe that this represents a significant forward step over the previous round of QAF publications. I offer the QA WG my congratulations on having turned things around so quickly.

Only three of these comments are not intended as formal comments according to the W3C process, these are marked with a *

Comments enumerated

1. Status of this Document

   * Links to latest version of QA Introduction and QA Ops Guidelines (sorry I have not checked this, unfortunately I am offline as I am writing this up, and cannot check immediately). I suggest the latest version links of QA Intro and QA Ops should either point to QAH or to a dummy page indicating that they have been superseded by QAH. If this is already the case, sorry for having raised this in error.

   Proposed resolution:this was in fact done in the published QAH WD.

   Resolution:moot.

2. 1.3 "QAH takes this approach for several reasons
   ....
   ... too authoritarian and fierce"

   I believe this rationale is interesting only to those relatively few readers who have reviewed the older QAF and wish to understand the change. Hence, while it was very much in place in this WD, I suggest it should be deleted in the next WD.

   Proposed resolution:agree in principle. Question: should the 2nd bullet be turned around to read positively, and kept? E.g., "in diverse, people-centric processes ...blah... is better received and more effective than stringent rules." (Where we'd come up with good characterization for "...blah...", like "user-friendly advice and guidance".)

   Resolution:agreed. The actual resolution is to change the 2nd bullet into a positive statement, free of its historical baggage.

3. 1.3 "If it looks helpful, follow it. If not, don't -- there's no consequence"

   Suggest delete "there's no consequence" - you don't really believe it, and, FWIW, neither do it. Choosing to not follow a suggestion is likely to result in lower quality, but that decision may be the right one to make in particular circumstances. This observation could be made directly in the text. e.g. "These suggestions will generally be helpful and enhance the quality of the work of a WG. However, each suggestion should be applied or not depending on the specific situation of the WG."

   Proposed resolution:agree, incorporate proposed wording.

   Resolution:agreed, will incorporate proposed wording.

4. 1.4 QAH Scope, last para

   In my view it would be better to be more upfront about the degree of emphasis on test and testability in the QAF,and note that WG's will need to balance this emphasis with other quality issues which are either only partially addressed or not addressed at all in the QAF, e.g. timeliness. (related to * comment on 6.1)

   Proposed resolution:agree in principle. Exact resolution tbd. This relates to the terminology issuesaround the use of fuzzy terms like "QA deliverables", and also his framework-name commentabout "QA Framework" versus "Test & Testability. Framework"

   Resolution:QAWG agrees that the QAF has a heavy emphasis on test materials production, but that is not its entire scope. Accordingly, QAWG still thinks it inappropriate to change "QA" to "test" or "test & testability" (in "QA Framework", or indeed in "QAWG"). QAWG was unable to think of another designation or title that is both more accurate and brief enough. And also, QAWG worries that the disruption of a general change away from the now-familiar (in W3C) "QA" at this point would not be worth the increase in accuracy. We do agree that the test&testability emphasis should be clearer, and will adjust the text in the QAH Scope (and other places).

5. 2. Story at beginnning

   I like it.

   Proposed resolution:thanks for the positive feedback.

   Resolution:thanks for the positive feedback.

6. 2. "Different kinds of QA deliverables might include"

   Suggest also

   "+ tests of new, changed or contentious functionality"

   this would capture the test driven spec development, that I presented in Cannes.

   Proposed resolution:accept comment, add this to the bullet list.

   Resolution:accepted, add this to the bullet list.

7. 2 Note:

   I have not yet reviewed the draft charter template (I now doubt I will be able to do so before 12 Jun)

   Proposed resolution:it is nothing but a 1-paragraph placeholder (Editor's note) in this first WD. It will be fully developed for next WD publication.

   Resolution:n/a.

8. 2. How can I do it?

   second bullet point suggest rephrase, I particularly disliked the wording "now is a good time for QAH to acknowledge that "

   Perhaps s/Having adovcated/While often/ and s/now is a good time for QAH to acknowledge that/

   the current wording seems somewhat battle-scarred to me.

   Proposed resolution:accept comment, use suggested wording.

   Resolution:agreed, will use suggested wording.

9. 3. Day-to-day QA operations

   first story

   I think TM is being used without being defined here.

   Proposed resolution:editorial fix.

   Resolution:In general, we are revising the text to largely avoid use of non-obvious abbreviations.

10. fourth para: "The W3C Process Document ..."

    suggest delete "all important"

    Proposed resolution:okay, editorial fix.

    Resolution:okay, will fix.

11. * 3.1 QA Moderator para

    suggest s/a single identified/an identified/

    e.g. WebOnt had two people filling this role without problem.

    Proposed resolution:okay, editorial fix.

    Resolution:okay, will fix.

12. 3.1 Note:

    I have not reviewed QAPD template, again I fear that I am out of time for this round

    Proposed resolution:in content, it is reasonably close to a usable template in this first WD. It will be more developed for next WD publication, including quick example links, and also more closely coordinated with the QAH text (and TestGL text?).

    Resolution:n/a.

13. 4. on the Document License 

    I wondered about copy&paste issues.  At least one function of a test suite is as a set of examples which should work. Hence it is tempting to do a view source and copy and paste from them. I note that this might be something that should be discouraged, particularly when the tests are exercising corners of the spec rather than the main body No suggested change, however, I wonder if you might want to add something on this.

    Proposed resolution:to be discussed by QAWG, along with legal issues in general.

    Resolution:@@check@@ The W3C AB is pondering a policy, and we will revise the entire "legal" section to be consistent when that policy becomes available. As we edit QAH, we'll keep in mind whether some consideration of this fits.

14. 5. Acquiring test materials

    Hmm, on my print out I have scribbled "links to test" alongside the first "good practice" box and the following two paras. However, I can't think what I was on about :(

    Maybe that the two lists in the two paras following the "good practice" box should consist of items each hyperlinked into a relevant part of the test guidelines where the issue is expanded.

    Proposed resolution:that's a good suggestion. When (or if) TestGL is developed and coordinated to a similar state of maturity as QAH, it should be done.

    Resolution:Good suggestion. Note, however, that QAWG has resolved for resource reasons to put TestGL on the back burner while QAH and SpecLite are advanced. So QAH references to TestGL will have to be put into "future planned" tense.

15. 5. TM

    The abbreviation TM is expanded here as Test Materials, but it has already been used extensively

    Proposed resolution:as a general policy, the editor likes to expand an acronym once in each major section of the document -- reconsider whether this should be done (tbd).

    Resolution:In general, we are revising the text to largely avoid use of non-obvious abbreviations.

16. 6.1 WG spec editors and authors

    typo: missing "of" in "understanding the Principles"

    Proposed resolution:editorial fix.

    Resolution:will fix.

17. * 6.1 WG-TS Moderator versus "WG's QA projects"

    Remaking one of my comments on the CR version of QAF, that it is really a test and testability framework and the name QA framework is unhelpfully broad. My prefered solution is to rename the QAF to be a "Test and Testability Framework". I suspect that the QAWG is unwilling to concede this. This issue is partially addressed in section 1.4 scope, and as I have already suggested could be made clearer there. In this section suggest s/WG's QA projects/WG's TM/

    Also not clear what TS in WG-TS participant or WG-TS moderator is meant to stand for. Maybe s/TS/TM/

    Proposed resolution:tbd. This relates to the terminology issuesaround the use of fuzzy terms like "QA deliverables", and also his scope issue.

    Resolution:As discussed above (#4), we agree that the QAF has a heavy emphasis on test materials production, but that is not the entire scope. Accordingly, QAWG still thinks it inappropriate to change "QA" to "test" or "test & testability" (in "QA Framework", or indeed in "QAWG"). We agree that the test&testability emphasis should be clearer, and will adjust the text in here (and other places). All terms like "QA projects" will be individually reviewed and replaced as appropriate with more specific or accurate terms. (Finally, note that QAWG resolved to move this appendix to an external, referenced document, "QA Framework Roadmap".)

18. 6.1 Last words

    I am out of time on checking that QAIG has chartered deliverables. I thought in general IG's did not have deliverables, and so the last sentence jarred.

    Proposed resolution:tbd (QA Team -- do you know the Process considerations here?)

    Resolution:good catch, will fix.

19. 6.2 Primer

    In the current WD this primer felt redundant, because the WD itself was sufficiently short. However, I suspect that when all the TBDs are done, the QAH will maybe double in length (hopefully not quite); and then the primer will become more significant. I suggest that while the main copy is part of the QAH single html file, that it may well be useful for many readers to have the primer as a short standalone file.

    I note that having reviewed your documents on a number of occassions now, I am not the right person to tell whether the primer was redundant or not.

    Proposed resolution:to be discussed by QAWG.

    Resolution:QAWG resolved to move this appendix to an external, referenced document, "QA Framework Roadmap".

20. 8. Refs

    QAF-TEST

    I don't believe this doc has made it to Candidate Rec as suggested in the note. Anyway the note will go in the next revision (after the next version of QAF-TEST has been published).

    Proposed resolution:editorial fix.

    Resolution:will fix.

Dominique Hazaël-Massieux

Related issues: scope issueof JC, and also his framework-name commentabout "QA Framework" versus "Test & Testability Framework"

Resolution:The QAH editor will look at each occurrence during next QAH editing cycle, and try for more specific and meaningful language.

Other:  LH-identified issues in the QAH

1. TestGL coordination: if, when, and how.

   Resolution:decided at Santa Clara, QAH will have loose reference to a "planned future TestGL" while TestGL is on back burner, and meanwhile QAPD template will continue to have places for those items to be addressed.

2. Templates completion (Charter and QAPD).

   Resolution:to be worked on for the next couple months, for next publication.

3. Fill in missing examples.

   Resolution:LH to make a list, referring to anchors in QAH draft, and put it out on QAIG plus W3C Chairs for volunteers.