Venue and Logistics· Agenda· Minutes·
QA Homepage· Latest News· QA Resources· QA IG· QA WG· QA Calendar·
These minutes have been reviewed by the participants and may be considered quite stable.
The fifth joint QAWG/QAIG face-to-face meeting was held on January 6th to 8th, 2003, in Redmond, WA, USA, hosted by Microsoft. All participants would like to thank Microsoft for the meeting infrasctructure and arrangements, as well as lunchs and reception. Participants would especially like to warmly thank Kirill Gavrylyuk, Microsoft, for his organization and hosting of the meeting.
Read the logistic page and detailed agenda for this meeting.
Below is an excerpt from the QAWG Action Items Table withaction items given during the meeting (and still opened after the meeting). The actual Action Items Table has up-to-date information about current AIs.
| Id | Owner | Action Item | Deadline |
|---|---|---|---|
| AI-20030108-1 | Kirill | Editors (Kirill, Dimitris, Mark, Peter) of Test GL will review EARL Spec. | 2003-01-22 |
| AI-20030108-2 | Dimitris | Describe what the "assuring uniformity" in third bullet of deliverable of TTF means. | ? |
| AI-20030108-3 | Dimitris | Produce new TTF charter draft | 2003-01-31 |
| AI-20030108-4 | KD | Investigate place in Paris (alternate to Greece) to host June QAWG meeting | 2003-03-06 |
| AI-20030107-1 | Karl | Counter-argument to Joseph about the need for scope of use restrictions in license for distributed TM | 2003-01-10 |
| AI-20030107-2 | Dom and Lofton | the ICS becomes the one sorted by priority | 2003-02-10 |
| AI-20030107-3 | Lynne | See if it makes sense to link the certification Note from the Intro | 2003-01-10 |
| AI-20030107-4 | Lofton | Check for history of CP 3.2 in OpsGL | 2003-01-08 |
| AI-20030107-6 | Lynne | Reread the document and introduce consistent useage of "strict conformance" | 2003-01-13 |
| AI-20030107-7 | Dom | Check that each DoV has a checkpoint about relationship with other DoV | 2001-01-31 |
| AI-20030106-1 | Olivier | Proposals for Glossary discussion and presentation at Tech Plenary | 2003-01-09 |
| AI-20030106-2 | Karl, Mark, and Wendy (owned by Karl) |
coordinate with WAI and develop a how-to draft for entering definitions into the glossaries | 2003-02-17 |
| AI-20030106-3 | Lofton | Executive Summary of OpsGL | 2003-04-15 |
| AI-20030106-4 | Lofton | Get template or examples of executive summaries from Janet | 2003-04-15 |
| AI-20030106-5 | Karl | Ownership of Kit. Write a summary of what needs to be done for the kit | 2003-01-08 |
| AI-20030106-6 | Olivier (and Janet) | Find someone to interact with MMI | 2003-03-01 |
| AI-20030106-7 | Olivier | Plan for outreach with WGs during Tech Plenary | 2003-02-01 |
| AI-20030106-8 | Karl | Go through the Matrix and present the list of the WGs matching the above criteria | 2003-01-08 |
| AI-20030106-9 | Olivier | Plan/make a speech for the Tech Planery meeting | 2003-03-01 |
| AI-20030106-10 | Lofton | Draft/send an invitation for review to the WG selected. | 2003-01-24 |
| AI-20030106-11 | Karl | Send publication request for last call documents | 2003-02-08 |
| AI-20030106-12 | Olivier | Last Call comments form | 2003-01-21 |
| AI-20030106-13 | Lofton | SOTD for the Last Call for OpsGL | 2003-02-08 |
| AI-20030106-14 | Dom | SOTD for the Last Call for SpecGL | 2003-02-08 |
| AI-20030106-15 | Karl | Draft LC announcement | 2003-02-05 |
| AI-20030106-16 | Karl | Send LC announcement | 2003-02-10 |
| AI-20030106-17 | Karl | Complete the ET documents for OpsGL and SpecGL | 2003-02-02 |
| AI-20030106-18 | Mark / Lynne | Submit to KD the Techniques for SpecGL | 2003-01-22 |
Review of agenda for first morning
Jump to : Context, Discussion and Proposed course of action.
Olivier and Wendy give a summary of the situation in QA, WAI and beyond.
Olivier explains: In QA, it is important to have consistency across specifications with respect to the wording and meaning of the words. We could look at existing glossaries and pull together into 1 document or start by creating a global glossary. We need to discuss our approach and what we need to do for Tech Plenary.
Do we plan for the unification of glossaries or plan a discussion that would lead to a task force for a glossary unification effort.
LH: Issue 19 was reopened : should there be glossaries in each spec? How do definitions in each spec relate to a central QA Glossary? See summary and proposal sent to www-qa.
The WAI model is a good one. For each spec, there would be a section called Definition, with a definition section specific to that spec. The QA Glossary would have a terse general, central definition of the term. The specific context sensitive definitions in the specs can't contradict the central definition.
WC: To implement this, is the plan to have everything in 1 place and pull from there, or to have terms distributed and pulled into 1 place? It is important to make sure that there is consistency between what QA does and WAI.
KD: What is important is to agree on the model and then we can figure out the Pubs Rules for it. Once we have this, we can encourage other groups to do this as well.
OT: QA could orchestrate, coordinate the effort.
WC: Glossary would be a good topic for Wednesday's Plenary. We can have an open discussion and start gathering requirements. WAI is planning on spending several hours discussing glossaries. They just don't have room to host it.
OT: We need to do 3 things - interest people in having every group input to the glossary, get them interested in formulating the glossary model, and to do a Call for Participation.
KD: There maybe a problem with markup. Maybe we could define simple tools to help build markup for glossaries. If you want to encourage people, you need to give them the tools to do it.
MM: We will need a process to mediate between groups with different definitions. Ultimately we want definitions that are consistent across groups.
LH: Propose that we take our model (his proposal) and build it, showing how this would fit into a universal glossary and this could be used by other Groups.
LH: There is the concept and there is the problem of resources.
OT: Will take some of this responsibility.
DD: I may have a candidate to work on the Glossary, in February.
WC: There may be some resources from WAI and I18N. we should pool resources.
OT: For Tech Plenary, we will discuss with WAI on Friday, and talk more with
Daniel
OT is to seek out interest in glossaries. WC will help. Also, discuss with Janet getting a session at Tech Plenary. The goal for Tech Plenary is to have open discussion, form a group, and if possible have a prototype.
Issue 19 closed. Affirm that each spec can have a Definitions section with context sensitive definition and it relates to the QA Glossary.
LH: Review of success criteria in Charter / what we are doing and what we have not done.
JD: There is already coordination with Comm. The Comm role is almost a production role with respect to getting the documents published. Comm fully supports QA.
KD: There may be a misunderstanding with the Comm team role. The part of quality having to do with publishing our documents is met. We need to define the mechanism to have the WGs apply the documentation we are writing into the Comm process. Having a QA person in each WG would be great, but we don't have this yet.
JD: As good ideas mature as best practices in QA, it would be good to have those incorporated into Pub Rules. To be in Pub Rules, we try to make rule as clear as possible, implementable, and to make sure all document templates are kept up-to-date.
KD What happens when a change is made to Pub Rules?
JD: Some things go smoothly, others don't. Get suggestions from Chairs, the Team, WG members; we don't usually get suggestions from the public. The Comm Team and Web Master go over the proposed suggestion to determine how easy to implement, the burden to implement, etc.
KD: Conformance section is a nightmare. Some groups refuse to include conformance, think the IETC MUST, SHOULD statement is enough. Requiring the inclusion of a conformance section through Pubs Rule would be too much a shock for some groups.
LH: Do we think we will be rechartered? Approaching last call on 3 of our 4 documents.
JD: Rechartering process straight forward. Look at this 3 months out. Domain leader becomes aware of expiration of charter, the status of the documents and the amount of time needed to continue. QA was chartered before the CPP (patent policy) was in place. If there are deliverables with IPR, then the charter needs to go before the membership again.
LH: Does the group think it will continue or pack up and go home? Implicit that we go on.
OT: Do we recharter with current work plan or recharter with new work plan.
LH: Should recharter with emphasis on testing task force, recognize that heaviest effort on writing the framework is done.
KD: To reach Rec stage, must have 2 implementations. So need 2 WGs to implement the guidelines.
Jump to Intro, Discussion and Resolution.
OT: Right now, when we discuss with WGs about QA, the answer is wither "go away" or "OK, what can you do to help". We should work on avoiding the first answer, and have material ready for when the second answer comes. Currently, we don't have either.
lesson learned within WAI : review requirements document
Need an "outreach kit": the kit should include success stories: showing how it can shorten WG's process or improve their documents and implementations (discussion whether this is really the case).
need to identify the target audience. Mainly WGs, but some documents for AC/managers. Within WGs, new groups are an interesting target.
Outreach Topics for Tech Plenary
Olivier will coordinate outrach at tech plenary (both session and direct outreach)
Karl owns the Kit, will summarize what needs to be written. Will try to have most of the kit ready by technical plenary.
deadlines:
Break for lunch. Sushi.
LH: Was there any response from Sun on status of the participation in the QA WG.
KD: Sun's response was that Sun will participate but no specifics were given.
LH: Any other companies that promised participation but did not participate?
KD: Many companies did not commit resources but committed to review the documents.
Olivier gives a short report on public outreach.
Jump to Discussion and Resolution : last call plan.
LH: We believe that documents (Intro, SpecGL and OpsGL) are ready for the the LC. We need to determine what to do for Ops ET and Spec ET. The deadlines we had (Dec 20) were slipped.
LH: Discussing reviewers. By the time the meeting is finished, we need to have a resolution on which documents and when will go to the Last Call. We need to determine the list of the WGs we'd like to ask specifically for the feedback.
criteria for groups selection:
KD: action item to go through the Matrix and present the list of the WGs matching the above criteria - deadline 8th Jan
PR entrance criteria:
KD: The Techniques part of the document will be informative, "This is a good way to do it".
LH: Once we fill the Techniques sections, are we satisfied that we have a proper support for the GL document? - Agreed yes.
Will also align the markup to have similar looking.
KG: How do we intent to publish ET documents?
LH: They stay in WG space, have dated redirect.
ACTION: KD: To complete the ET documents for OpsGL and SpecGL by February 2nd
MS/LR: Action Item to submit to KD the Techniques for SpecGL
LH: Some of the checkpoints do not have rationale anymore.
MS: I find rationale to be useful. What it would take to add it?
LH: A day -1.5 days of work. Not all the checkpoints in SpecGL have a rationale either.
KD: Rationale is like a business case.
LH: If I change the document to add Rationale for all the checkpoints by Jan 21st would the document be completed? No additional WG review necessary?
All: Yes. MS: I trust LH.
LH: Could we pull all of the Ch3 from the OpsGL?
Resolution: Leave a brief header section referring to the QA WG Process document. Most of the info from Ch3 is already in the QA WG process document.
Adjourned for the day,5:10pm.
Updating and discussing opened Action items.
#2 (drop it, it may alien more than help)
Preferred resolution: "Use doc license if possible, software or W3C approved custom license if necessary" ; propose TM license as an example of license with scope of use restrictions; final resolution still pending discussion between Kirill and Joseph
Resolved to add a link in the discussion to W3C-wide contribution license
101: closed since there doesn't seem to be an obvious reason to move to this new scheme.
resolved to create a new checklist ordered by priority -> AI to Dom and Lofton, deadline 2003-02-10 (LC publication); the ICS becomes the one sorted by priority
originator to be changed to AT; resolved as it is adequately handled by the existing distribution licenses;
It doesn't relate to the framework; AI Lynne to see if it makes sense to link the certification Note from the Intro, deadline on 2003-01-10
we don't want to do it; out of scope
this is certification's business, out of scope for the framework; in OpsGL, 6.4 already prevents from abuse of conformance claims based on test materials
Does not belong to OpsGL; SpecGL and TestGL are more relevant; SpecGL handles it with acknowdleging specs as class of products and with CP 10.3; -> recast for TestGL (needs to see if there is a need for a new item in the taxonomy for multiple standards test suite)
Originator to be changed to Mark; going through keywords different from MUST:
(KG): similar to XSLT framework. XQuery will contain appr. 10.000 tests when released. [PowerPoint presentation by KG, will be made available to the WG].
Test Markup is basically copy pasted from the specification, no specific test markup language.
Test Case is a scenario.
What is a test assertion (TA)? Does SpecGL include TA? Should SpecGL require TA for all specifications? (discussion on what a TA is, on the one hand, and how it is syntactically represented, on the other)
(MS) what level of precision/specificity is required for it to be testable? all MUSTS could be argued to be TA. On the other hand, it may require a higher level of specificity
(KG) Not every TS is an independent atomic entity. It's similar to a function in a program.
(discussion on what a specification author should do: write clear specs only, or also supply test assertions?)
(DH) we need to have a checkpoint on requiring requirements in specifications.
(MS) if we put TS in as chekpoints, what priority should they be?
(DM) 3 for the short term future, intending to make it a stricter priority later on.
(LH) should we create a new checkpoint for TA? Change "to fulfil" into "conformance requirements". Change the title of guideline 13 into something like "identify all conformance criteria".
(all) agreed.
Action item on Lofton to do so.
(DH) Prio 2 or 3 for providing TA?
(DM) 3, only for the short term, ideally migrate to 2
(LH) would say 2/3 but not 1
Prio 2/3 for TA?
(resolution): prio 2
(DH) One of the checkpoints I have a hard time understanding. Currently it says "distinguish requirements from product-specific features"
(DM) when you have anything but strcit conformance, you want to divide the possibility of having more than the required features into extensions/more than the minimum (but somehow within the anticipated range of variants the spec allows for).
(MS) How could you not have extensions? You're still not allowing extensions, which is the defintion of strict conformance.
(DM) What about natural languages?
(MS) As long as you prohibit extensions, yes
(DM) The maximum set of features in that case is not the same as the minimum
(MS) if you are correct, I suggest to take out "strict conformance"
(DH) First thing, conformance requirement as written is not clear enough, then the question of clarifying the role of extensions, strict conformance etc
(DM) We could take out the term "strict conformance" but we have a definition in 3.2. If we only want to impact 3.3, we just say something about anticipated variability that does not get implemented by way of extension (implementation-dependent feature)
(DH) the whole checkpoint should beb rewritten
(MS) Anything that is variable within the spec, is not a variant, but an extension. What's the reason for having this checkpoint?
(DM) Reason for anticipating extensions/non-extensions, a way of saying implementing two languages inst. of one does not constitute an extension
(MS) how is this enabling ckpoint 9 to work?
(LR) I don't see the reationale for having it
(MS) agree
(LH) I see the distinction between extension and extended capability of standard functions, not sure of seeng the value of having an extra checkpoint
(DH) agree to remove chkpoint 3.3?
(all) yes
(DM) If we drop 3.2 entirely, we have nothing in the guideline that says anything about conformance policy, whereas guideline 9 still requires this
(DH) Any objection to remove chkpoint 3.2?
[no objections]
(DH) should we merge guideline 3 and 10?
(DM) we split it intentionally to allow for the difference regarding Dimensions of Variability, 3 discusses this, 10 writes about it
(LH) should any of the verbeage from 3.2 migrate to 9 (suggestion by DM to move the second paragraph of 3.2 to 9)?
(LH) Do we move the definition of "strict conformance" to GL9?
Yes
(LH) Does strict-by-class concept of Chkpoint 3.2 get moved to GL9?
Yes
Action item: Editors must reread the document and introduce consistent useage of "strict conformance"
(LH) definition should allow for extended features and so forth
[David Marston leaves]
(DH) proposal: add a checkpoint to guideline 9 saying that specifying relationships between extensions and other DoV
Yes, including specific examples for strict-by-class
Action item on editors to check that each DoV has a checkpoint about relationship with other DoV
Issue 109 closed
processing Alex Rousskov's comments and issues: 002: high priority issue, and 009, 010, 011, 012, 016, 019, 022, 023, 025: substantive 001: done. Use conformance requirement instead. 005: Lynne suggests taking the second section of Alex's rationale.
DH: in favor of the second option.
LR: It should be in the ET. I agree with Dom.
LH: I agree with David Marston that it should be part of the fullfilment criteria.
DM: From a personal point of view, I use hardcopy a lot.
LH: It's an easy requirement to fulfill.
DH: I agree that it should be usable for the hardcopy version but it should not be in the requirement. If we go in that level of details, we'll have to review our checkpoints
KD: Explain the difference between online version and printed version. The only normative version is online. Hardcopy is a plus, and guides for these versions should be in Manual of Style. We should contact Susan Lesch for specific recommendation.
LH: explains
DH: The point is too narrow and very specific version of the document which is not normative.
Poll: No wins.
Resolution: We will not include this requirements. But it will be in the discussion verbiage of the checkpoint.
LH: It does apply to consumers as well.
DH: How we define deprecated
LR: Define it.
LH: explain the deprecated version (---D---) (----N---)
Resolution: The point is only for producers.
(Scribe comment: not sure to understand the issue)
DM: Discussion if the WG should write down about the decision of the WG and not the spec itself.
DH: The SpecGL is about spec not process of the WG.
LH: G3 has changed a lot since, maybe it doesn't make sense anymore. I will see the new draft of the G3 and will reopen it if necessary.
withdrawn because of the change of the Guideline 3.
Is it important to have this disclaimer or not? Does it mean that a SpecGL fails if people can't comply to Level A? What does that mean to conform to Level A? The question is "If you follow SpecGL Level A, your spec is not necessary good. If you do not follow SpecGL Level A, your spec can still be good" LONG DISCUSSION to reach the consensus.
Resolution: Remove the fail part of the Conformance Disclaimer.
Definition should in the general glossary and not in the spec itself. In the spec itself there should be only a specific wording for the scope of the spec.
Resolution: The definition of the Guideline 14 is taken for the general definition in the Spec GL.
Resolution: Dropped because the CK has already been changed.
SHOULD in 2.1 migrates to a MUST
SHOULD in 4.4 migrates in the discussion part and lowercasing it. LH do not agree.
Spelling mistake at the begining if -> If which had lead to a discussion about use of SHOULD and MUST in the spec. How should we use with CK and priority the RFC2119, Keywords? What are the implications on the spec itself?
SHOULD in 8.1 migrates to a MUST
MAY in 5.1 migrates to lowercase. (bug)
Historical presentation.
There's still a need for test Description Markup but EARL is to markup the result of testing.
You can use it also as a bug report language.
There is a good list of implementations already.
The next step would be to have applications to combine the results issued of an analysis to make for example compatibility chart, or dated reviews compared side by side. Possibly could create a tool for maintaining a Website with dated version of maintenance. It would be good to organize a Workshop on the language to be a uniform Reporting Language or Test Framework.
Dimitris had no answers about his try to gain to build task force from W3C. He recommends to start as soon as possible to avoid the development of particular tools inside each WG. The scope of the TTF should be limited to a small set of tools.
Wendy asks for review of the EARL specs.
AI-20030122-01 Kirill Editors (Kirill, Dimitris, Mark, Peter) of Test GL will review EARL Spec. 2003-01-22
Meeting Tech Plenary with ERT on Thursday Afternoon
LH : Will not finish specGL today. Will finish in next 2 telcons. Thus, cannot do resolution to go to last call. That will take place a week from Monday. One open item left for OpsGL. Will do at Monday's telcon (fill in commitment table of GL1.) Rest of Monday wii be devoted to finishing SpecGL. Everyone should look at draft Action Item list and make sure we understand and agree with it.
Dd Charter is modeled on QA WG charter. Charter is for a task force to ensure that guidelines in TestGL get followed by WGs When they decide to produce test materials.
LH Is this to narrow for our mission?
LR - She objects to enforcement role.
LH Mission should be to ensure that every rec has a good test suite.
Dd Should be enforcement.
MS We don't have power to enforce.
MS Ensure is proactive. We can't ensure (guarantee).
KD We can tell WGs when they do not successfully have 2 interoperable implementations.
DH People won't spend a lot of time worrying about the charter.
Consensus New bullet develop tools, templates and tool kits of general usefulness to help WGs develop test materials.
Consensus Tools to help WGs conform to our documents is the charter of QAWG, not TTF.
LH Enforce the use of practices will get changed to Ensure the use of practices required by the QAWG Guideline documents.
KG Should be "help to ensure".
MM Should say "advocate and help implement".
Dd Deliverables include test technologies and techniques
LR Should the tools reside in QAWG? Should TTF develop templates to facilitate tool development? Do we envision TTF building things?
Dd We should not do maintenance. Don't know about templates.
LR Charter should allow us to optionally develop tools to help WGs build test materials or to help WGs conform to our documents.
Consensus It's desirable for TTF to build tools, resources allowing.
MM Even a how to will help.
Dd We shouldn't be out source for building tests.
LH Should completion of TestGL be in charter for TTF? Consensus No.
LH Third bullet of deliverable should be "Assuring uniformity" rather than just assuring.
Duration of TTF It depends on whether TTF is chartered as separate from QAWG.
LH We should say that we anticipate that this mission will take at least 2 years (mission will be long-term) and TTF is just a mechanism for starting it up.
Dd Should be a WG.
LH Doesn't matter what it is but if we make work interesting, people will join.
MS People will only join if they get management support and they need to show relevance, not interest.
Dd We will not attract new people if it's in the same WG.
MS This will not allow us to attract new members; we will need to do it.
LH That supports not having a WH because we will not have the resources to start it.
LH Success criteria should be worded to reflect the mission statement.
Task Group Proposal was already discussed in Tokyo. We don't need to discuss it again now. DD has already done F2F solicitation DOM WG is happy with idea but no one seems to devote resources.
LH If we have high-level W3C support, they should help get us support. W3C should meet in person with companies.
MS, DH That's not the way W3C works or should work.
MS We should plan that the only resources are already in QAWG; if we get more, great, but deliverables should not assume that.
TTF Next Steps: next draft of charter, circulated to IG list; call for participation, investigation by KD.
Points of consensus:
An http test suite was demonstrated.
Next meeting, at Technical plenary in Cambridge, Ma. Then meeting tentatively in Athens.
Dd - No connectivity, no meals, hotels may not be cheap.
KD Had same situation in Montreal; did not provide meals; try to get support from universities, etc.
LH We need an alternative for the Athens meeting. I propose France as an alternative
TestGL KG will have reduced time to contribute; cannot be lead editor. PF may have time to be lead editor(need 4 hrs/week commitment).
Meeting adjourned at 1700.
Created Date: 2003-01-15 by Olivier Thereaux