This document contains replies to comments against the 2-jan-2002 draft of "QA Framework: Process & Operational Guidelines". The format is as simple and explanation is as short as possible. Editors deeply acknowledge every comment made to us, we're sorry in advance if our resolution(s) did not meet your expectations and are happy to discuss any of them. Please review the current draft and comment more!
-Kirill Gavrylyuk (for QA Editors team)
Ian's comment on TOC layout
Taken, TOC is flatened, main chanpter is called Guidelines with Guidelines serving role of subsections. Note, that Chapter 3 (relationship with QA WG) remained without Guidelines/Checkpoints
Ian's comment on "10 levels of WG commitment to QA". To transform to a table with levels being rows growing along 3 axes: WG staff, Spec content, Test Materials extent
Partially taken, transofrmed into a table, levels are growing along 2 axes - Spec testability and test materials extent. Plans to shuffle them more (may be remove couple levels) and/or add 3rd axis "Test materials quality".
Ian's comment on the last level in "10 levels of WG commitment to QA" - includes the phrase "complete test suite." - not a known notion.
Left as it is for now, to be fixed/reworded later. Plan to refer to not written 4a "Technical Guidelines", don't want to elaborate on this in Proc&Ops.
Ian's comment: Provide boilerplate text (informal) for the disclaimer: passing all of the tests does not guarantee full compliance of implementation to spec, and failing the test suites means failing specific tests for specific targeted features.
Done.
Ian's comment: I don't think it's necessary to have a checkpoint requiring the process document to point to the test materials; this seems like an ordinary thing one would do..
Changed to: "In QA Process document, describe the way test materials will be published and point to the corresponding web page"
Ian's comment: I don't think it's necessary to have a checkpoint on usability of test materials for two reasons: (a) people will make use of the most usable instance available.(b) unless you explain what makes the test suite usable, the checkpoint won't be helpful. If you have points for making a test suite usable, add a guideline on test suite usability and include specific checkpoints under it.
Left unchanged. Examples: current test materials published by XML Schema WG, test materials in OASIS XML test suite have pretty low usability. We will add reference to 4a "Technical Guidelines" where we will explain what does it mean
Ian's comment: It's not necessary to say too much about documenting resource commitments in the charter; that's already required by the Process Document. But it makes sense to say "Don't forget QA."
Left unchanged. Process Document doesn't tell anything about level of commitment to QA. Amount of resources depends on the level of Commitment
Ian's comment: I don't think it's necessary to have a checkpoint that existing charters should be updated. That requirement only applies once, when the QA documents are approved within W3C.
Taken. Removed this checkpoint.
Ian's comment: Most of the rest of the prose in the document can be inserted as notes after checkpoints or guideline introduction material, or even subsections of "1. Introduction".
Taken. We believe it is fixed now.
Ian's comment: The checkpoint about w3c providing a secure and reliable repository can be deleted.
Taken. Checkpoint is changed.
Lynne's comment and following discussion with Daniel and others on "Test matrerials home"
Taken. Please see if the wording of current Guideline 2, Chk2.1/2.2 and notes to them. I tried to capture all the ideas.
Lynne's comment/objection for reason #2 in "test materials home": ...I also disagree with Reason #2, - although some test suites are developed by ad-hoc entities others are built and supported by 'real' organizations that won't go away. And, although the people who built the tests may disappear (same as what happens in a W3C WG), the organizations will still exist (e.g. NIST, Open Group, ETSI)
Taken. Corresponding readson is removed.
Comment (from Lynne?) Text from explanation to Guideline 3 to be moved to introduction.
Changed, but left. We believe it needs to be there in context of the current document.Please take a look.
Comment (from Daniel) on assesment of the quality for the test suite developed externally.
Right now there is a wording in notes to Chk 2.2 and Chk 7.1.