[Note from QAWG issues editor: These Last Call (Intro, OpsGL, SpecGL) comments, as well as General and TestGL comments, were received from DI WG on 20030314. Following is the comments body of the message, modified only to add a label and anchor on each comment. -LH.]

Thank you to Olivier Thereaux for visiting the 3/2003 DIWG f2f meeting and presenting to us the work of the QA group.

Our feedback to your work is made out of two parts. The first part recapitulates the points that were brought up in our f2f meeting. In it we respond to you as a group. In the second part we respond to your documents.

General Comments to the QA Working Group

• [GC-1] We are happy to have you. Your work is important and will affect the quality of our deliverables. We also realize, however, that each process that you recommend, while contributing to the final quality of our deliverables, also adds overhead to our efforts. The way to resolve this conflict is by authoring good tools.
• [GC-2] We would like to have A DIFF application that highlights the differences between consecutive versions of a document and by doing so helps a reader to pay attention to the novelties and progress of the work.
• [GC-3] To support uniformity across the consortium and help editors, a set of generic templates should be provided. Such templates will delineate the structure of all possible documents (i.e. specification) and will provide standard headings like “Status of Document”, “Introduction”, “Audience” and ”Glossary.”
• [GC-4] Can you recommend a process by which the QA people are NOT the people who work on the deliverable? Reviews requested by other groups play such a role to some extent. These reviews, however, take place quite late in the life of a document. Maybe earlier steps inside the group can improve the deliverables. As a group, we have exposed ourselves to feedback by publishing our documents to the public before submitting them as final documents. We will appreciate any comments, ideas and suggestions from your group about this subject. Providing guidelines to how to conduct these reviews will improve the reviewing process.
• [GC-5] The DIWG ability to review your operational and test guidelines is limited due to the current nature of our work. Our coming work on the CC/PP specs will give us the opportunity to follow your recommendations and provide you with more feedback.
• [GC-6] All of us are aware of the challenges of multiple glossaries. You addressed this topic in your plenary presentation but, of course, much more thinking and work need to be invested in this important area. Does each document have its own glossary? When does a document give a definition of an expression besides the glossary’s definition and how are the inline and the glossary definitions kept synchronized? You mentioned in your plenary presentation the possibility and permission to give, depending on context, different meanings to the same expression. How do we increase the reader’s awareness of the co-existence of multiple meanings and emphasize the “locality” of a given meaning? Which process or, better yet, tool supports us in maintaining and expanding all the glossaries simultaneously?

Feedback Relating to the QA Documents

General

• [FR-1] Can we measure the QA Operational, Specs, and Testing guidelines against the ISO 9000 process? Has this standard been considered as a possible framework?
• [FR-2] QA processes are inevitably seen as imposing an additional cost on those that need to perform the extra work necessary to comply with the process. However, a QA Process is created to achieve an improvement in the "product" to which it is being applied. It would be beneficial to include some quantification of the expected cost and benefit associated with the process. This will help to improve acceptance of the process, will provide measures by which the process itself can be assessed, and provide a basis to assess potential improvements in the process. Especially it will allow working groups to measure the inevitable delay in producing specs against the benefits of the QA process.

All QA framework Documents: Motivation

• [FR-3] There is no need to convince the readers that QA is an important aspect of every deliverable that aspires to have industrial strength. A too lengthy and detailed explanation of the motivation states the obvious and as such might give amateur impression. On the other hand it is necessary to share with the reader how the processes and standards recommended by the QA group support and improve the current processes of W3C. The new QA documents provide an opportunity to highlight the W3C processes either by including them explicitly in the documents or by mentioning them and linking to them.

Examples and Techniques

• [ET-1] Other than for the introduction, each of the three main documents is paired with its “Examples and Techniques” companion. “QA Framework: Operational Guidelines” is paired with “Operational Examples and techniques”; “QA Framework: Specification Guidelines” is paired with “Specification Examples and techniques”; and “QA Framework: Test Guidelines” is paired with “Test Examples and techniques.” All these documents are listed in the Introduction document. Pointing out to the pairing and adding some prose on how the two members of each pair relate to each other will help the reader. Section 4.1.2 of the Introduction document might be a good place for it.
• [ET-2] We highly recommend following the QA recommendations and providing more examples for each guideline and checkpoint.
• [ET-3] All three documents have numbered guidelines and checkpoints. It will be helpful to label them in a manner that will easily distinguish them from each other even out of the context of their individual documents. To identify them uniquely across the consortium, a QA identifier can be added. I.e. QA-Op-G1 for Operational Guideline 1, and QA-Sp-G1 for Specification Guideline 1. Similarly, checkpoints should not be only 1.1, 1.2, etc, but QA-Op-1.1 for the first checkpoint of QA-Op-G1 and QA-Sp-3.5 for the fifth checkpoint of QA-Sp-G3.
• [ET-4] Is there any time line for the release of “Specification Examples and Techniques” and “Test Examples and Techniques?” The “Operational Examples and Techniques” document is very helpful.

QA Framework: Introduction: Motivation

• [IN-1] We recommend to replace every occurrence of “the ultimate mission of W3C with “ the ongoing mission of W3C” recognizing that we are participating in a process without an end.
• [IN-2] The expression “unimplementable language” is quite meaningless. It is not the language that is implementable or not but rather the things that are claimed or aimed at.

QA Framework: Introduction: Target Audience – The Working Groups

• [IN-3] Section 3.5.2 gives a list of the groups that make up the audience. The details of this list are not as fully expressed in the Audience section.
• [IN-4] Sections 4.1 and 1.3 -- both deal with target audience. For cleaner and clearer structure they need to be combined.

QA Framework: Specifications Guidelines

• [SG-1] “Classes of product” of Guideline 2, “Profiles” of Guidelines 4, “Modules” of Guideline 5, and “Functional levels” of Guideline 6 are all related, close to each other and maybe synonymous at times. A discussion in which all these concepts are discussed and compared will be helpful.
• [SG-2] This might be the place to suggest each document follow some standard headings.

QA Framework: Test Guidelines

• [TG-1] The W3C has some requirements about testing: how many tests; internal/external; etc. They should be included in these Guidelines.
• [TG-2] By which process is the integrity of the tests assured?
• [TG-3] Guideline 3 gives a non-exhaustive list of methodologies. They are all obvious to QA people. To all other mortals, links to articles that elaborate on these methodologies will be helpful. Especially it will be helpful to bring examples that implement these methodologies.

We would like to close with appreciation to the importance of your work and with hope that it will help to improve the quality of the W3C deliverables and their uniformity as well as provide the working groups with tools for better and more efficient work.

Shlomit Ritz Finkelstein on behalf of the Device Independence WG.