Action Items· Minutes, AM· Minutes, PM·
QA Homepage· Latest News· QA Resources· QA IG· QA WG· QA Calendar·
See the Agenda. (Detailed Agenda)
Roll Call
Rob Lanphier is no longer a member of the Working Group; he is just an observer. A discussion ensued concerning the need for more members. Although it was agreed that the QA Working Group's efforts are becoming much more visible, the consensus was that having more members would allow the Group to maintain our ambitious schedule and large amount of deliverables.
Karl Dubost talked about his activity to develop an overview about what's going on the QA mailing list. It will be interesting for people (journalist, web community as large, occasional readers) to have an abstract of what has been discussed on the mailing list. The goal is both to inform people who are not involved in our activity as well as to encourage people to join the activity.
The original proposal was to create a pool of 3 people in charge of writing this overview each week, alternatively. A team member of the QA activity will review the overview. It was decided to include the Interest Group and any relevant Working Group traffic in the overview. A discussion ensued concerning whether weekly was too often to issue the overview. Monthly was also proposed but that was thought not often enough. It was finally decided to produce the overview bi-weekly. It was decided to try this for 6 months and then re-evaluate the utility of the overview.
The following liaison reports were given:
A discussion then ensued regarding how to get Working Groups to allocate enough resources to do an effective job in testing and in meeting the QA requirements. There were two alternatives proposed:
FPWD published Feb. 1. There has been no feedback since the FPWD. We also have loose outlines for a) Specification Guidelines and b) P&O Examples and Techniques. The target date is March 11 for Working Group publication of the first 4 parts Intro, P&O, P&O examples and Spec Guidelines.
FPWG of first 4 parts (intro, P&O, P&O examples, spec) by beginning of April. Target for everything (all 7 parts) is September. The target for the second working draft of all parts is Jan. 2003.
Peter and Mark volunteered to be contributors to the fourth document (Test Materials.) Kirill will be the Editor.
Kirill then discussed his revisions to the Process and Operational Guidelines document. Kirill revised Guideline 1 according to Lynne Rosenthal's suggestions. Kirill accepted Lynne's rewrite of Chapter 3 - WG relationship to QA Activity. Kirill will cut and paste Lynne's rewrite of Chapter 3. Lynne's suggestions for the rewrite of Guideline 4 were incorporated by Kirill's reorganization of the document.
Scribe: Peter Fawcett
Three new observers present since Lunch:
Olivier started off with a summary of tools for the working group.
First there are some web pages to assist in publishing documents for the qa web space. These include tools for generating templates for new web pages, a validator to check that the page is valid for the site and a link checker. (NOTE: Should I include the urls?, these sites seem public as will be the email... is that a concern? I have the various uri's if they are required.)
There is a new page for new WG members that contains lots of useful information. There was some discussion as to wether this document should be in public or private space, no resolution of the issue was reached but it was decided to leave it as it is for now.
Finally there is a new search tool for the email archives. This can be found at http://www.w3.org/Search/Mail/Member/search. The one trick demonstrated was if you use 'mon year' (ex 'Feb 2002') for the lookfor: field it will give messages from just that month/year.
Nadia gave a demonstration of the test harness she has been working on. It uses n3 markup to describe a series of tests and will then generate a pass/fail/NA web interface with comments on each test case. Two forms of web interface can be created, one uses frames to have both the test case and the pass/fail controls in the same window. The other uses a list of the cases with their pass/fail results in one window and another window/web application to display the test case.
The results of the tests are output in EARL.
This tool isn't finished yet so it isn't publicly available yet. There was a suggestion that this tool could be used for validating checkpoint types guidelines as well.
Issues Processing:
Issue #47: Should test materials be published in TR space? We revisited this, is it really closed. Decision was Yes. But we need to clarify why we feel this way. PF was given an action item (ACTION: A-2002-03-1-6 ) to go through the mail archive and write up a summary of the justification.
This brought up the topic of the rating/endorsement of test materials. There has been an email thread along this line in the IG mailing list. The concern seems to be that the WG is claiming that test suites should be rated by some criteria of goodness. The WG does not feel that this is the case, what we are proposing is to rate is the level of conformance to a set of guidelines. It was decided that this issue related to our already existing issue #44 but that issue #44 was too general so there is an action item to LH (ACTION: A-2002-03-1-7 ) to split issue #44 into two new issues. One concerning wether to and how to rank test materials and one concerning the endorsement of test material. As for the mail thread, it was decided to continue as it is and not issue a WG opinion for now.
Issue #44: Should W3C endorse externally produced test suites? MS pointed out that conformance doesn't mean that a test suite is perfect, it just means it passed some set of criteria. DD One problem with conformance is that our guideline wont be ready for 6 months so we wont be able to hold test materials from current working groups to what ever future standard we set.
LH We aren't able to judge test materials for 'good ness' but we can comment on or judge test materials on their publishing methods. Is it public and freely available, does it have a system from handling errata and so on.
MS We can't check the depth of a test suite because any significancy complex system will have a nearly infinite number of combinations to test but we can check test suites against some set of check point criteria.
Distinction made between High level check, method of publishing, and low level check, some judgment of the worth of the test materials.
DH/LH It is easier to hold test suites designed with in the WG process to some standard that it is with test materials designed out side of the of the WGs.
dd Outside test materials/suites should only be included in matrix if they agree to be tested for compliance.
Decision was Yes to conformance testing for test materials, No to Endorsement of test materials and Only include outside test materials that agree to be tested for conformance.
Break for Coffee.
New Guest: Ian Jacobs
LH proposed postponing discussion of issue #55 till after the WG has had time to read the new document. All agreed to Thursday March 7 at the usual time. ACTION: A-2002-03-1-8 LH Get bridge for Thursday (March 7) Telicon.
Issue #23: Tests for MAY/SHOULD assertions.
MS felt that we should have a way of testing the 'may/should' in the guidelines.
There was some discussion about the language and the difference between Priority levels and normative language. For an implementation to pass with normative language it must pass the MUST tests but not the MAY or SHOULD tests. With the guideline and checkpoint type Conformance Levels if a WG commits to a level of conformance they Must pass all checkpoints up to the required priority level. If a WG commits to 'AA' then they must pass all Priory 1 and 2 checkpoints.
Issue #56 was opened by MS. Issue being that we need an objective test suite for testing checkpoints in guidelines document. All agreed and issue is Closed.
Issue #23 was closed once every one agreed on removing all normative language from the guidelines documents. Also agreeing that priority levels need to be testable to verify a given level of conformance.
Issue #54: Several structure and content issues about Proc&Ops Guideline 6. This is a multi part one. it is too long - Not an issue any longer it's been broken up.
checkpoint 6.1 is not verifiable and shouldn't be there as a checkpoint - All agree it should be made a note rather than a checkpoint.
checkpoint 6.2 looks like a duplicate of 6.5, or 6.5 + 6.2 - in any case, it looks like it's already covered by following checkpoints; - Language needs to be clarified. These are separate checkpoints, one dealt with the way to except an outside test suite, the other being how to review material once accepted. As they are now one appears to be a subset of the other.
checkpoint 6.7 is similar to 3.2 and 3.3. - WG doesn't think that this is an issue.
Close Issue.
Issue #52: Does the QA Working Group need a "Process Document"?
LH Good beginning would be covering process of handling stuff in chapter 3 of the Guidelines document.
DD WAI WG doesn't have process document, they do have special alias mail archive of comments from the group for the record.
KD process document is more for us, our process than for the outside.
LR Agrees.
WG Agrees we need process document.
ACTION: A-2002-03-1-9 KD Claims ownership of process document for QAWG.
Issue #53: What is the process for handling requests from the WGs? Postpone resolution of Issue #53 till after we have a draft of the process document to discuss.
WG agrees.
Issue #49: Should there be a global (W3C-standard) license for use and distribution of test materials?
KG pointed out that there is really two licenses involved, one for documents/suites submitted to a WG from outside and a second for documents/suites published by the WG.
Should we come up with a template for first type.
Second type should usually use Document License.
dd Takes a stand that we should NOT dictate what license to use.
LH disagrees.
Suggestion to put item to J. Reagel from legal, LH discussed email discussion he has already had.
PF suggested adding language to justify why WG was suggesting Document license in most cases but why Software license may be needed.
WG agreed. Issue closed by adding language to guide line to explain justification for suggestion.
DD Pointed out to be aware of existing pubrules and style guides when working on our guidelines.
WG decided next Face to Face should be in June 13-14 seemed to be the only dates available for some people.
ACTION: A-2002-03-1-10 KD To look into possibility of finding place for Face to Face in Montreal.
Next Telicon: Thurs 3/7 then back to regular schedule on 3/14.
The next meeting will be held in Montreal June 13 and 14.
The meeting was adjourned at 1720.
