Venue and Logistics· Agenda· Minutes·
QA Homepage· Latest News· QA Resources· QA IG· QA WG· QA Calendar·
The fifth joint QAWG/QAIG face-to-face meeting was held on March 6th and 7th, 2003, in Cambridge, MA, USA, during the W3C technical plenary meeting.
Read the logistic page and detailed agenda for this meeting.
Below is an excerpt from the QAWG Action Items Table withaction items given during the meeting (and still opened after the meeting). The actual Action Items Table has up-to-date information about current AIs.
| Id | Owner | Action Item | Deadline |
|---|---|---|---|
| AI-20030306-1 | lofton | Send QA contact for Karl in the matrix | |
| AI-20030306-2 | Karl | Make an individual file by WG in n3. | |
| AI-20030306-3 | Lofton | Send contact info on Kirill to JR so he can contact him | 2003-03-07 |
| AI-20030306-4 | PF and PC | Improve wording of Section 3.1 of the Process document and come up with specifics and measurable items | 2003-03-14 |
| AI-20030306-5 | MS | Define what an issue is | 2003-03-28 |
| A-2003-03-07-1 | Peter | Send comments to the form for OpsGL. | 2003-03-10 |
| A-2003-03-07-2 | Lynne | Include Ian Jacob's comments into the form for SpecGL. | By2003-03-10 |
| A-2002-03-07-3 | Patrick | Review to QAWG charter | 2003-03-15 |
| A-2002-03-07-4 | Lofton | Update the QA PD template and message to chair list | 2003-03-20 |
| A-2002-03-07-5 | David | Draft resolution to issue 23. | |
| A-2002-03-07-6 | Lynne | Write checkpoint to reflect resolution of issue 107 | 2003-03-20 |
| A-2002-03-07-7 | Patrick | Circulate a set of suggested list of attributes that will lead to collapsing the checkpoints in GL 1. | 2003-03-14 |
| A-2003-03-07-8 | Peter | Combine GL 2 and 3 | 2003-03-14 |
| A-2003-03-07-9 | Patrick | Restructure the guideline 4. | |
| AI-20030307-10 | Patrick | Next draft TTF Charter with detailed deliverables and strategies from the f2f | 2003-04-12 |
The Raw minutes/notes for this half day were sent to the QAWG mailing-list. This document summarizes them in a more readable form.
Jon Gunderson for the UAAG working group, and Matt Oshry / Paolo Baggia, for the Voice Browser Workign group, came to make a presentation of their work and experience with test suites, test frameworks, etc.
From these presentations, we could conclude that there is a need for:
The QAWG could think about providing all these as part of the mission of the "Test Task force" for example.
Lofton, Karl, Dom and Olivier visited groups during the first two days of the all-groups meeting to discuss with them about their QA needs and expectations, as well as talk about the QA Framework (with help from the QA Outreach Kit put together by Patrick.
The awareness of the QA effort seems to be growing, some groups had read the QA framework, and had a lot of questions. Other were not yet very aware of it, but overall the situation is promising.
The groups expressed various needs, among others:
The Team has a project review scheduled for Team inreach... We should continue this outreach effort, which has been extremely valuable so far.
Some points we could focus on when visiting groups :
The Raw minutes for this half day were sent to the QAWG mailing-list. This document summarizes them in a more readable form.
Joseph Reagle was our guest to discuss test material license.
here may be confusion with respect to fair use. JR will develop a document with all the issues will then send to Chairs for feedback. Next, the document will be sent to W3CM and MIT legal. Anything in JR's present document that's not representative of the issues?
WG agrees that the present document accurately reflects concerns.
Patrick asked whether the document was a statement of requirements, and said he would develop a statement of requirements for Sun.
The group reviewed the current QAWG process document, trying to finalize it.
The addition of "topic experts" to the already existing WG members and invited experts was questioned and discussed, and finally the group decided to follow the process (Invited experts at the discretion of the chairs), and to drop that part of the document.
We need a deadline when sending notices
Clarifying current practices, and clarifying relationship between the process document and the logistics document (link former to latter).
We revisited questions such as "What is an issue, when does a comment become an issue, and who is supposed to decide that it is one".
Another interesting decision : Issues should include alternatives.
The raw minutes for this half day were sent to the QAWG mailing-list. This document summarizes them in a more readable form.
Peter initiated the meeting providing a brief summary of the results from the break out team's review of the Process document using the process document template. The activity resulted in a set of comments that are going to be incorporate in the OpsGL comments form. Patrick took this action. Lynne will incorporate Ian Jacobs's comments into the SpecGL comments form.
As a result of this review, Lofton recommended that a review of the charter should also be done. Patrick took the action of performing the review.
David Marston presented the results of the XQuery and XSL outreach. There were two areas that generated discussion; reporting results and associated claims of (near) conformance. It was recommended to have an example of a disclaimer or a boilerplate to distance the WG from any ill-formed claims concerning the results. In regards to the test assertions a discussion on completeness was generated. The WGs were also concerned about separate test assertions as opposed to tagged content in the Rec, because they abhor saying the same thing in two places. Tagging was believed to raise concerns, some words are more important than others. David added that if this is an issue for some people it could be something we need to recheck. The concern was centered on the side of the test assertions that ties to the specification, no much was discussed in the area of the test assertions leading to test cases. Other than this, the group was basically asking clarifying questions.
Issue raised in email thread (DM, 2001-11-07, 2nd pgph).
MS: There are no requirements for interoperability. We need specific checkpoints.
LR: The requirements for interoperability are being addressed in other specifications. If they are represented in those specifications they are going to be tested.
LF: This is an old issue, just noticed the e-mail that was generated.
DM: The intention of this issue was related to pipelining of XML, meaning expected interoperation between different specs.
MS: There is a problem with the way we are capturing the issues.
LR: This type of information should not be included in the TestGl at this point. We need to keep this document simple.
KD Agree. The TestGl needs to be simple and this will not prevent writing a separate document to address clarification items.
PC Also agree to keep the document simple. Put some words in the document about extensibility/optionality. Propose to address this in the SpecGl, not in TestGl.
LH : Propose to cut and paste clarification into the body of the issue. Also propose to go with Lynne's proposal.
The issue is closed.
LH: A similar issue on SpecGL.
LR: This deals with the comment I raised about providing an overview of the test development process, addressing general concepts and the test development process.
PC: Are you referring to the question of what is conformance testing?
LH: This kind of information should not be part of this document. The intent of this document is not to be a tutorial.
MS: It should be part of the scope.
KD: Recommend modeling the introduction after the WCAG introduction. A primer should make it easier to read the specification, not understand (explain) the specification.
DM: QA framework introduction could be used.
LR: Close the issue. The scope statement should include general clarification for the document.
Issue closed.
LR: Explained her concern on TestGl not addressing a test suite development based on the state of the specification. Writing a test suite for CR might have a different objective than writing a tests for a recommendation. For example: for CR the objective may be to test the specification, whereas a test suite for a Recommendation targets interoperability of the implementations. The TestGl is not allowing flexibility for CR.
DM: Verbiage should be included in the document to cover these areas. The test suite development is a constant process of checking the specification, from WDs onward, including after Rec (exposing needed errata). The test cases can also be used as clarifying examples, and may help the WG write an airtight spec. But the same tests are still useful after Rec if they still are "correct" against the verbiage that the spec editors ultimately put in the Rec.
SM: The process of developing test cases for a test suite should not change; the working group is responsible for identifying the areas to test based on a particular state in the specification development process.
LR: I agree, but the way this document is written it does not allow for CR. Something should be included in the introduction or scope. I am just not happy with the structure as it stands right now.
LH: A checkpoint will be appropriate to address this issue.
PC: It is a matter of coverage and completeness. Coverage is barely defined in the TestGl.
LR: It is up to the working group to determine the coverage. I am not particular on how is done, usually test suites for CR are smaller and more focused.
DM: WG delivers only one test suite. Sounds like you are talking about different test suites. Recommend adding a checkpoint.
LR: A WG may have multiple test suites for different parts of a specification or for different applications of the specification. The introduction or scope should also include something about the fact that a working group might have multiple test suites for a particular specification.
LH Propose to accept Lynne's proposal of adding a checkpoint to cover this issue. The components of what needs to be address should included in the checkpoint
Issue Closed : Lynne will write a checkpoint to for resolution to this issue.
This issue was discussed at the f2f March meeting, after more discussion it was decided that Mark Skall will raise the issue in SpecGL to nail down the circumstances and David will to try to draft a resolution.
Issue closed.
Prior to the Guideline section, provide an overview of the test development process, addressing general concepts and the deliverable path, i.e., spec test assertions test cases reporting maintenance. If possible the guidelines should follow this progression. Also, describe the key aspects of a test suite: traceability, verdict criteria, self-explanatory, valid, short/atomic.
This comment was discussed during the issues section (79) it was agreed that the scope statement should include general clarification.
The comment was accepted by the group.
This comment is making reference to the order of satisfying the checkpoint. Lynne added that it is when the test suite is finished, that all the checkpoints need to be satisfied. She further recommended that we should make that statement up front. The group agreed.
Although we can't assume that the OpsGL and SpecGL were followed, but if they were, some of the CPs may already be satisfied we should provide a table cross-referencing where this occurs. Lynne explained that we can not assume that a checkpoint was already done. The editors should include a note to make a table for cross-reference. Given the status of document a note should be included indicating that it was already done.
LH : This table is not trustworthy until LC.
Since we advocate developing test materials for CR, we should explain that test materials for CR may serve a different purpose and have different coverage than test materials for Rec and this is O.K.
Already discussed as part of issue 107.
SVG and CSS have good test documentation, explaining not only how to build tests, but some of the test suite principles.
Lofton recommended there be a list of possible discrete deliverables in TTF. The group agreed.
Lynne added that she had a difficult time with the order of the guideline. We should adopt the logical process from the tester. Organize it better.
PF: This is what GL 1 does, probably needs to be made more explicit. Changing the wording of the guideline will help.
It should also mention something about EARL.
LR: All this stuff got merged at the end. It should stand out, there are also too many check points.
DM: What kind of action you think is expected from the results?
LH: Providing a result management tool, then include an example in Ex/Tech. Also, the guideline should not mention EARL.
LR: There is more to say at a higher level related to the sorting and reporting capabilities in ck 5.2. A checkpoint should suggest reporting of result, something to make reporting consistent across implementations.
KD: More of describing what to do and how is organized.
LR: Also have a concern with the word framework in the title of guideline 5, it seems to be focusing on recording a framework.
How extensive does this set need to be? For example, DOM must use valid components of other specifications, so would it be necessary to list all those specifications? Is this target set limited to those specifications that are explicitly being tested rather than those specs that are utilized?
Everyone agreed that the language need to be worked out and the checkpoint need to be formatted the same way the SpecGL and OpsGL was done. The whole guideline needs to be restructured.
These CPs are related and breaking them into separate CPs has resulted in confusion as to the difference between them What is the difference between "defined ambiguously" (cp1.7) and "contradictory behaviors" (1.9)? Suggest combining, as: Identify behavior: undefined, ambiguous, and contradictory.
Patrick will circulate set suggested list of attributes that will lead to combining the checkpoints. Lofton expressed concern about the priorities, but David commented that after combining the checkpoint will become a priority one. Patrick added that the way this is going to be presented is not imposing is providing choices.
Document the testing methodology. What is the difference between these? I think there is a difference, but my simple mind is having trouble making sense of all this.
Patrick agreed that the distinction is not clear and suggested combining GL-2 and 3 and speaking more in term of the language. If not combined they should be reversed in order.
Peter took the action to combine the Guidelines.
What does "testing techniques" mean? How is this different from test automation and framework? How much of a search for these techniques needs to be done?
Comment is moot due to previous action.
Lynne added that the default is not to provide automated frameworks and that we are being unrealistic. Patrick suggested that all the automation checkpoints need to be collapsed into one checkpoint; he added that if we are going to do automation we need to provide good guidelines. Lynne also suggested using the ideas from the VML presentation to include test case management.
The group agreed to restructure GL 4 to include automation and test case management.
These comments are going to be covered by the restructure of the guideline 4, mapping the process from beginning to end by looking at the structure. Patrick took this action. Action due two weeks from Monday.
Demonstrate results reporting has sorting and filtering. Why is this P1? Although nice to have, why is sorting and filtering required?
Comment already discussed.
The raw minutes for this half day were sent to the QAWG mailing-list. This document summarizes them in a more readable form.
The future (re-chartered) QAWG will probably be mostly about TTF and outreach.
If we want to carry the TTF charter further, we need to identify possible deliverables. A first list was made:
Patrick noted that Sun is working on Tools to do some test assertion analysis of specifications. If possible he may be able to leverage it and move it into back into our group/w3c.
Once the charter is done, we can start sending invitations. Patrick agreed to be the co-editor of the TTF charter, and will work on the list of possible deliverables. The group will try to make progress on the TTF issues by starting some small things in its deliverables.
The group went through Lofton's comments. Transcript of the discussions follow.
Move to adjourn... agreed. Meeting adjourned around 4PM on March 7th.
Created Date: 2003-03-25 by Olivier Thereaux