[Note from QAWG issues editor: These Last Call (Intro, OpsGL, SpecGL)
comments, as well as General and TestGL comments, were received from DI WG on
20030314. Following is the comments body of the message, modified only to
add a label and anchor on each comment. -LH.]
Thank you to Olivier Thereaux for visiting the 3/2003 DIWG f2f meeting and
presenting to us the work of the QA group.
Our feedback to your work is made out of two parts. The first part
recapitulates the points that were brought up in our f2f meeting. In it we
respond to you as a group. In the second part we respond to your
documents.
General Comments to the QA Working Group
- [GC-1] We are happy to have you. Your work
is important and will affect the quality of our deliverables. We also
realize, however, that each process that you recommend, while
contributing to the final quality of our deliverables, also adds overhead
to our efforts. The way to resolve this conflict is by authoring good
tools.
- [GC-2] We would like to have A DIFF
application that highlights the differences between consecutive versions
of a document and by doing so helps a reader to pay attention to the
novelties and progress of the work.
- [GC-3] To support uniformity across the
consortium and help editors, a set of generic templates should be
provided. Such templates will delineate the structure of all possible
documents (i.e. specification) and will provide standard headings like
“Status of Document”, “Introduction”, “Audience” and ”Glossary.”
- [GC-4] Can you recommend a process by
which the QA people are NOT the people who work on the deliverable?
Reviews requested by other groups play such a role to some extent. These
reviews, however, take place quite late in the life of a document. Maybe
earlier steps inside the group can improve the deliverables. As a group,
we have exposed ourselves to feedback by publishing our documents to the
public before submitting them as final documents. We will appreciate any
comments, ideas and suggestions from your group about this subject.
Providing guidelines to how to conduct these reviews will improve the
reviewing process.
- [GC-5] The DIWG ability to review your
operational and test guidelines is limited due to the current nature of
our work. Our coming work on the CC/PP specs will give us the opportunity
to follow your recommendations and provide you with more feedback.
- [GC-6] All of us are aware of the
challenges of multiple glossaries. You addressed this topic in your
plenary presentation but, of course, much more thinking and work need to
be invested in this important area. Does each document have its own
glossary? When does a document give a definition of an expression besides
the glossary’s definition and how are the inline and the glossary
definitions kept synchronized? You mentioned in your plenary presentation
the possibility and permission to give, depending on context, different
meanings to the same expression. How do we increase the reader’s
awareness of the co-existence of multiple meanings and emphasize the
“locality” of a given meaning? Which process or, better yet, tool
supports us in maintaining and expanding all the glossaries
simultaneously?
Feedback Relating to the QA Documents
General
- [FR-1] Can we measure the QA Operational,
Specs, and Testing guidelines against the ISO 9000 process? Has this
standard been considered as a possible framework?
- [FR-2] QA processes are inevitably seen as
imposing an additional cost on those that need to perform the extra work
necessary to comply with the process. However, a QA Process is created to
achieve an improvement in the "product" to which it is being applied. It
would be beneficial to include some quantification of the expected cost
and benefit associated with the process. This will help to improve
acceptance of the process, will provide measures by which the process
itself can be assessed, and provide a basis to assess potential
improvements in the process. Especially it will allow working groups to
measure the inevitable delay in producing specs against the benefits of
the QA process.
All QA framework Documents: Motivation
- [FR-3] There is no need to convince the
readers that QA is an important aspect of every deliverable that aspires
to have industrial strength. A too lengthy and detailed explanation of
the motivation states the obvious and as such might give amateur
impression. On the other hand it is necessary to share with the reader
how the processes and standards recommended by the QA group support and
improve the current processes of W3C. The new QA documents provide an
opportunity to highlight the W3C processes either by including them
explicitly in the documents or by mentioning them and linking to
them.
Examples and Techniques
- [ET-1] Other than for the introduction,
each of the three main documents is paired with its “Examples and
Techniques” companion. “QA Framework: Operational Guidelines” is paired
with “Operational Examples and techniques”; “QA Framework: Specification
Guidelines” is paired with “Specification Examples and techniques”; and
“QA Framework: Test Guidelines” is paired with “Test Examples and
techniques.” All these documents are listed in the Introduction document.
Pointing out to the pairing and adding some prose on how the two members
of each pair relate to each other will help the reader. Section 4.1.2 of
the Introduction document might be a good place for it.
- [ET-2] We highly recommend following
the QA recommendations and providing more examples for each guideline
and checkpoint.
- [ET-3] All three documents have numbered
guidelines and checkpoints. It will be helpful to label them in a manner
that will easily distinguish them from each other even out of the context
of their individual documents. To identify them uniquely across the
consortium, a QA identifier can be added. I.e. QA-Op-G1 for Operational
Guideline 1, and QA-Sp-G1 for Specification Guideline 1. Similarly,
checkpoints should not be only 1.1, 1.2, etc, but QA-Op-1.1 for the first
checkpoint of QA-Op-G1 and QA-Sp-3.5 for the fifth checkpoint of
QA-Sp-G3.
- [ET-4] Is there any time line for the
release of “Specification Examples and Techniques” and “Test Examples and
Techniques?” The “Operational Examples and Techniques” document is very
helpful.
QA Framework: Introduction: Motivation
- [IN-1] We recommend to replace every
occurrence of “the ultimate mission of W3C with “ the ongoing mission of
W3C” recognizing that we are participating in a process without an
end.
- [IN-2] The expression “unimplementable
language” is quite meaningless. It is not the language that is
implementable or not but rather the things that are claimed or aimed
at.
QA Framework: Introduction: Target Audience – The Working Groups
- [IN-3] Section 3.5.2 gives a list of the
groups that make up the audience. The details of this list are not as
fully expressed in the Audience section.
- [IN-4] Sections 4.1 and 1.3 -- both deal
with target audience. For cleaner and clearer structure they need to be
combined.
QA Framework: Specifications Guidelines
- [SG-1] “Classes of product” of Guideline
2, “Profiles” of Guidelines 4, “Modules” of Guideline 5, and “Functional
levels” of Guideline 6 are all related, close to each other and maybe
synonymous at times. A discussion in which all these concepts are
discussed and compared will be helpful.
- [SG-2] This might be the place to suggest
each document follow some standard headings.
QA Framework: Test Guidelines
- [TG-1] The W3C has some requirements about
testing: how many tests; internal/external; etc. They should be included
in these Guidelines.
- [TG-2] By which process is the integrity
of the tests assured?
- [TG-3] Guideline 3 gives a non-exhaustive
list of methodologies. They are all obvious to QA people. To all other
mortals, links to articles that elaborate on these methodologies will be
helpful. Especially it will be helpful to bring examples that implement
these methodologies.
We would like to close with appreciation to the importance of your work
and with hope that it will help to improve the quality of the W3C
deliverables and their uniformity as well as provide the working groups with
tools for better and more efficient work.
Shlomit Ritz Finkelstein on behalf of the Device Independence WG.