Minutes of QAWG and QAIG face-to-face meeting, Tokyo, October 2002

WARNING : Minor additions and clarifications may be added to these minutes. .


Action Items

Table of Contents

October 8th, morning

Lynne Rosenthal
Lofton Henderson, Olivier Thereaux

Thank you to KEIO team for meeting arrangements and for providing the meals and reception, and especially to Olivier Thereaux for meeting organization and logistics.

Review of Agenda

Status of Spec ET

KD waiting until SpecGL more stable before producing ET. With work with DH to ensure a timely production. Usually published as a Note rather than in REC track (see issue 68).

Week in QA: Should this be continued?

Started bi-weekly, and evolved to monthly. we made multiple calls for feedback, but those were unanswered.
Is it worth continuing or not?
  • it is used to publicize work,
  • valuable to provide a summary,
  • should be monthly
  • If target is WG, do we need as much detail?
  • Can we get statistics on who else is reading this? We Can ask on chairs list. Many may not know about this. Karl takes action item to ask chairs.
  • Can add as link on newsletter to members.
Agreed that it was a good idea, to do monthly, LR agreed to capture Month in review and work with OT and KD to format and publish it.

Telcon schedule

should we make this weekly and change time? (for next quarter)
DH : we need to be more efficient. Not talk about issues unless it has owner and enough members to reach consensus on issue.
before every telcon, provide list of every issue to be discussed, with owners. KG noted that we had weekly telcon during summer and still ran out of time.
Teleconference moved to Weekly
Schedule :Monday 11 AM Eastern US
Effective week after next (Oct. 21)
Length of call 1.5 hours (decided day 3)
Naming the scribe prior to the meeting
PEOPLE must be here on time.
issues must have an owner (and preferably a proposed resolution) before the meeting

Handling of issues triggered by IG

Should issues raised by IG dominate discussion?
IG should be allowed to open new issues.
MS: Should wecreate a timeline for major issues to be raised?
LH: SpecGL was reorganized with invitation to respond.
OT: IG should be there to comment, open issues should be discussed in public. Issues need to be acknowledged and discussed.
DH: Don't need the IG approval to close an issue.
KG: need a process, do more business oriented: assign to a person, don't fight, review email, put into issues list, acknowledge receipt, discuss in WG, reply to originator with resolution.
MS: we decide what is an issue and what is not.
Should look at: Is the quality of the deliverable good? Not necessarily issues.
Kirill G. suggests process : do more business oriented: assign to a person, don't fight, review email, put into issues list, acknowledge receipt, discuss in WG, reply to originator with resolution. Once an issue comes in, discuss who owns it, analyze it, talk with sender, next telcon assign priority and owner.
DH: lead editor takes initial responsibility for splitting, if necessary, create list of issues and assign owner at next telcon.
Suggested issues process adopted. WG members may or may not discuss issues with "reporters" on the IG list, depending on time available, but no obligation to do so.

Process for QA WG and IG

see above discussion. We need a process document. Had part of one in Ops GL, Karl created another part, the decision above is yet another part...
OT: Propose that we group all our processes
LR agreed to produce first draft-detailed outline of Process document by 31 Oct, but not agree to be permanent editor.


OT: QA team accepted continuation of work on EARL (asked by WAI). Wondering about possible role of QAWG/IG. Is political benefit enough compared to amount of additional work?
KG: many vendors not support EARL.
LH: any group that creates test materials will have to have reporting, so it is within scope.
OT: There is a new tool for reporting being produced by team.
DD: Notes that DOM decided not to use EARL. There will be people that won't want to support it.
OT: when EARL reaches stability it will be hosted by QA Activity, not clear if it will be developed or maintained or only hosted.
DD: one of test groups would be early assessment of technology.
Topic Closed


LH: 3 types of reviews:

  1. reviews in order to progress the documents,
  2. requested reviews e.g., SOAP would like our review.
  3. ?

We need to respond to review requests Qeustion is, Who will do it and how?

LR: original proposal, was that the WG chair handles.
KD started to write a process document, sent a long time ago.

LH: can chairs assign someone to handle or respond that there is no one to handle it we lack interest and resources.

DH: No requirement put on us, when there is a Last Call review requests are sent to all WG chairs. But, it is good to have comments on Last Call documents.

KG: decouple reviews from comments on documents, but can not justify time on review of matrix.

KD sits on all CR and PR calls (as Conformance Manager and as QA Team). Tries to go through issues and not be too harsh, but identify what isn't clear or logical.

LH: do archive of reviews exist? Can we produce list of what reviews were done and what was said.
KD can start such a list.

Outreach (to chairs and public)

DD: summary of last week's telcon, discussed what existing WG going to do. What will happen to the future of the WG. How to get more participation? The WG needs to have a more business-like profile, QA is a W3C deliverable if you don't have QA, you don't publish.

OT: Increased support from team, but not from members. Need to make more noise, so people realize that if these guidelines become mandatory, they will be stuck with what we produce, good or bad.

DH: what experience have other WGs had in publicizing, how get participation.

MS: different model. QA is intended for W3C people, internal document. Other model, audience is implementers. Our guidelines should be treated differently. Need a modification schedule since won't really get comments until these documents are used.

DD: initial audience is not only W3C internal, but testing guidelines will be used by others. MS: but only mandatory (for conformance) by W3C.

OT: Think of the W3C Process (and how it is reviewed/approved) as an example: Proposed, reviewed by members, and then enforced. And it gets reviewed, since people know that it will be enforced. Members will want more things on tests. Do we move on now or later?

DD: WGs will need more help with testing, in the future they may need more help with specs.

DH: If guidelines are good, they will be used beyond W3C as evidenced by WAI and their guidelines. Agree with MS, that will get feedback at Last Call, that is when people will read specs.

MS: not inferring that there won't be outreach affects. By the money aspects. People don't make money by developing test suites. applaud Microsoft's commitment to participating in QA and test development.

HF: a carrot is if you do, a stick is if you don't. For outreach - if you go through this process and measures, this is what you get: a better spec. Specify the benefits. And, by the way, if you don't you don't get published (that's the stick).

DD: sees comments like - who will review my products, do I get a 'brand' for conformance, etc.

DH: who is developing test suites. Other groups developing test suites: XML Core, DOM, Query, XSL Schema. DOM developed test suites since needed 2 implementations.

DH: Need to have OpsGL should speak about what kind of conformance you should reach for the level of your deliverables, e.g, what you need for exiting CR. LH: there is something defined. Something else is required to make WGs to conform to OpsGL.

OT: Propose to move OpsGL to First Call? Propose: Advertise work. Who should make list of items to advertise and how? Need to have a briefing.

DD: volunteer to help create how to advertise to other groups.

OT: took item to contact WAI to understand how they work.

DD: what will customer get in the end? Need to look at them as customers to get them to buy our QA products. Always advertising QA to DOM WG.

LH: hesitates in doing outreach to SVG, when don't have the documents or resources.
OT: but can't get caught up, that can't do anything till we have the documents.

KD: working with Janet Daley on outreach to members.
OT: working with Judy, Brewer to learn from them.
DD: creating "how to" advertize QA to groups

Chairs meeting presentation

Had one for TechPlenary, last March. Last meeting was July. Talk repeated to AC.

DH: targeted outreach. Check out WGs that are doing test development and target those members.

LH: have companies that stated commitment, but don't. And companies (David Marston) that participate, without commitment. Get someone with contacts at AC level to approach AC members.

OT: Identify who we want to contact and get what we want to say.

KG: Several carrots to pursue. Provide web space to publish testing results. Microsoft would publish results and that will get others to publish.

DD: People would jump if W3C would certify.

OT: This is an open issue in W3C.

MS: Would David Turner from Microsoft say some words about QA at the AC meeting, since Microsoft is so supportive of QA?

DD: Provide QA space with links to published vendor test results.

KD: Question regarding certification will be raised at AC meeting.

LH: Once meeting since we decided the QA guidelines are mandatory. Can we use this as part of an appeal for participation? Maybe Janet or Daniel can talk to some AC members.

DD: if target a few companies and not the full membership, then they may want something in return. Can do in parallel with talking to entire membership.

LH: agree with DH, that need to focus and target specific people.

DH: have been advocating QA at various meetings, Team, Chair, etc. (will continue)

October 8th, afternoon

Kirill Gavrylyuk


DOM: Would it be useful to talk to other WG on how to invite more members?
Lofton: Who would you contact in the WGs?
Dom: People who showed any interest in the QA.
Lofton: What is the nature of the contact?
Dom: Trying to recruit people from the WGs who have test suite efforts.
LR: I could commit to something like this. We have 4 people that are developing the test suites for the W3C WGs. They reviewed the TestGL document. They can help WG to understand how close the documents are to the target.
Olivier to talk to the WAI leader to learn how they do the outreach.
Karl to contact COMM team on the QA outreach, getting new members to come in.



XMLSpec allows to automate: tests validation; test assertions <-> tests mapping.

DOM test suite allows to validate that the test case uses API according to the spec. Does not support test assertion <->test case mapping yet, but can be done.

SVG: difficulty to identify test assertions.

Discussing the questionnaire results
50/50 of the XML Spec and "XHTML + classes".
Need to communicate the results + some statistic back to the editors.
How did the QA WG come up with our XHTML classes?

Lofton/Kirill: we did an adhoc analysis of the techniques around, and had a desire to get things started fast.

The goal of the questionnaire: to understand why people are not going with the XML Spec.

Lofton:there is an inertion in the W3C to move to XML Spec.

DOM: XML Spec had 3 versions in the last 5 months

Dom: what is the final goal of having different versions of the spec, rendered from the single XML Spec?
We can provide modules for XML spec to markup TA - great idea. But we can not convince all people to use XML Spec + XSLT rather then other format + Perl scripts for example

Dimitris: Should we make the XML spec technology mandatory or rather spell out goals and make only them mandatory?

Dom: I don't think the technology can be made mandatory, but the goals can.

All: How do we advertise the technology? Through the COMM team? Through the examples and techniques document? The risk is that not everybody would read it.

Action Items
- Generate the feedback/statistics and communicate it to the editors (Dimitris, end of October)
- Baby-step project: extension for an XML Spec for the test assertions. Approach Scott Boags regarding his suggestions. (Dimitris)

Test group (first discussion)

Dimitris: Presents the proposal for forming the Test Group. Recap of the history on this topic and the proposal being sent.

Dimitris: As the documents mature, we need to put more effort into the testing technologies. I also think that it is more productive to use the concept of the division of labor.

Mark: Supported the idea of the division of labor.
Dimitris: formulates the deliverables and the conclusion.
All: Yes we agree that the problem of resources exists.
Dom: proposed to take these issues to the point of rechartering (next year).
Dimitris: Not clear what to do with the test questions till then.

October 9th, morning

Mark Skall

KG led the discussion on TestGL. There are three key issues need to be addressed to make TestGL a better document.

Process v. Results

The TestGL is in a very immature state. KG talked about what the goal of the document is and asked what we mean by a good test suite. Are we too heavy in process? Should we focus more on strategy? The current Guidelines and checkpoints are very process oriented and we need to try to reformulate them.

KG discussed why he wrote the TestGL in this manner. It was done because we want to know how test suites (TS) achieve their objectives. How do we know test suites cover everything they were supposed to? How do we know implementations pass the test suite? What do we verify to determine the test suite is good? The original Guidelines were organized to help answer these questions.

The key issue is do we develop TestGL to help determine what is a good test suite v. how to develop a good test suite (more tactical)?

LR's comments said there were too many onerous requirements. We should provide high level guidance and leave details to example and techniques.

MS Let's look at an example new checkpoint 1.3. Would that move to the Examples and Techniques document (ExTech)? The higher level requirement would be that finished test cases should be grouped (mapped) a certain way, but ExTech might illustrate that TS can be grouped a certain way.

DH, LH Don't need to do GLs in a specific order.
DD agree with LH. We need to address modularity issue. Can do it with spec in hand.

KG TestGL shold be strategic, not tactical. Guidelines 6 and 7 should be part of ExTech or OpsGL.
DH GL 6 and 7 should remain in TestGL.
LH Require TS to contain provisions for operational testing.
MS GL 6 is strategic, but GL 7 is tactical.
LR Agree.
LH Disagree.KG First 5 guidelines define the exit criteria for TS.
MS, DD GL 6.5 is also exit criteria.
DH GL 6 should be in ExTech.
Guideline is what
ExTech is how.

DH When developing TS you look at ExTech, not GL. We want GL to be mandatory, but not ExTech.
DD Propose simplistic solution. The only remaining issue is where to put what.
Process, what and GL go together.
Operations, how and ExTech go together.
We need to restructure document to accomplish this model.
KG is GL 6 what or how, especially 6.4 and 6.5?
MS - GL 7 has strategic content. "Encourage public availability of TS and of results of testing" and move details into ExTech...

Do not scare implementers

DD ExTech is for implementers of TS. GLs for managerial and WG chairs.
MS and LH TS developers must use GLs.
MS and LH agree!
MS Change will not scare implementers because details are not required anymore.
LR - Will satisfy 2), especially if we make clear that sequence is not required.
LR NIST didn't develop TS according to TestGL.
MS That's not the issue. The issue is "should we have" and the answer is no.
LR Won't have complete coverage to exit CR. Is this for exit of CR or later?
KG Checkpoints in this GL verify that TS verify quality.
MS Does a requirement exist that "every requirement should be tested" anywhere in our documents?
KR That requirement should not be in TS GL.
DH It should be in TestGL. Need measurement.

We will introduce a table into TestGL similar to OpsGL. This will ensure that all requirements are tested.
TestGL editors will restructure document (will be assigned at editors telcon). Quality items will be added in the form of a table by the TestGL editors.
Need another TestGL re-write by editors. Request for review of the re-write by the editors and by theWG. Get WG feedback, review feedback, incorporate changes, and propose to WG to publish as first WG (Nov. 20).

Issue 68, status of specGL

Issue 68 (extech vs GL)

Issue 68 was then discussed. Issue 68 - What should be the status of the "ExTech" parts, and how should they relate to their "Guidelines" parts? Karl only needs one week notice to produce the SpecExTech. The recommended solution is informal, referenced, companion documents maintained in /QA/WG/ space instead of /TR/. As they stabilize we will move them as notes to TR, at editor's discretion.

QA framework reviews when will we fill in last column? The assignment is to fill in the set of techniques in documents. We already have many examples, but not a lot of techniques. This is a sufficient but not necessary condition.

LH and lead editors will have telcon to choose dates for the fourth column.


DH ran the following portion of the meeting. DH went around the table to ask each participant "What is current state of SpecGL?"

DD Better. We need to work on GLs 14 and 15.
MS Let's move forward to get input from a wider source.
LR It's hard to read. It should be slimmed down to be more to the point.
KD Too much prose, too complicated for someone who is new. This includes feedback from external people. Not "pleasant" to read.
OT If too big and hard to read, it's not easy to correct both at the same time.
KD Need a primer a dummy guide.
KG Good thoughts like DOV. Very complex subject. Need a primer a pleasant to read intro. Need DOV defined separately from GL text. Need a primer before public review but shouldn't do anything else.
LH We have a superset of what we need. DOV is useful. Can collapse and consolidate GLs and checkpoints (GLs 14 and 15). Move details of DOV elsewhere (perhaps to ET).
DH too much, should clarify what needs to stay and what needs to be separated into appendix or elsewhere. Need clear distinction between GLs and ExTech.
LH Lot of progress from first to second draft.
MS Editors must understand DOV to write a good spec.
DH DOV should not be part of spec, but of a higher level. It's architectural design.
Consensus let's proceed with DOV approach in the spec.
LH GL should just say how you're using DOV. ExTech should give examples.

GL 2.

We did not have enough time to discuss 3) GL 2.

October 9th, afternoon

Dimitris Dimitriadis

Demo of XSLT/XPath TS (Kirill)

[@@pointer to demo material to be made availabe]

SpecGL issues

Issue 87 DoV (revisited, spill-over from morning session)

Dom: Decision this morning to continue using the DoV and briefly explain the concept in SpecGL. Inside the spec or not? Two questions: Do we keep the structure? Do we keep the discussion about the relations?
Lofton: has to be introduced in GL
Dom: not incompatible. You can keep the definition in SpecGL and still have a separate more detailed document.
Mark: do not agree that minimizing the number of DoV increase the odds of interoperability
Lynne: would like to see some discussion of DoV in SpecGL, but also a separate document explaining the relations. Reasons: DoV unique and important concept, other WGs may want to capitalize on it annd use it.
Lofton: no disagreement, having a separate document does not limit normativity
Lofton.: the current discussion of DoV in SpecGL is minimal.
Dom: the concept is minimal, but we have 8 sections on discussion between DoV and modules, extensions etc.
WG agrees on cleaning up discussion of DoV currently in the document, in particular guidelines 3, 4, 7 (to divide between new document, other section in current document and Spec ExTech)

Issue 88 : SpecGL objective and value

SpecGL should be about the characteristics of good specifications and not about how to write them, which should be adressed in ExTech document.

Issue 89 : Is the SpecGL a spec?

Rephrased to
The SpecGL must conform to itself. Verified, Action item on Mark to verify if SpecGL conforms to itself.
Mark: should we say must, or should?
Conclusion: must
Lofton: we should be conformant at Last Call.
Dom: No, we must be conformant at CR.
Level A conformance for Last Call

Issue 101 : Should the GL documents use conformance levels instead of prioritized checkpoints?

Action item renewed to investigate (Olivier) [next week]

Issue 80 : RFC 2119 Keywords are used inconsistently in the SpecGL.

Upper-case all occurences
Issue 80 closed
Action items on Editors to implement this in their documents

Issue 81 : Definition of specification testability

Action item on Mark to rewrite sentence

Issue 82

Action item on Dom to change the title
Dom to implement

Issue 69: Should Spec Guidelines discourage "flavors of conformance"?

Recast "flavors of conformance" into "Dimensions of Variability".
Close the issue by replying to D. Connolly.
Action item on Lofton to reply to D. Connolly

Issue 61: Should standard terminology be required?

implicitly closed (done already).

Issue 73: Should SpecGL require atomicity of modules?

Leave as is, definition to be moved into the checkpoint and recommendation into ExTech

Issue 85: Should every test assertion be covered by a example?

Agree to drop checkpoint 1.4, agree to reformulate 1.3 by dropping the second sentence.

Issue 84 :Should examples and user scenarios be lower priorities?

WG believes that examples are helpful and that we should not presuppose that people reading the specificaiton will misinterpret it.
Action item on L. Rosenthal to respond to A. Rousskov.

Issue 86: Checkpoint 1.5 should be lowered to priority 3

Disagreement with issue formulation, reformulate checkpoint to better accomodate needs. (Not agreed with issue formulation. Reformulate ckpt to achieve intention of ckpt.
What is meant by interpretation section?
Rename: include examples for formal descriptions.
Add a clause for N/A for those specs that don't have formal descriptions (e.g., if you don't have formal description, this ckpt does not apply).
Leave as P2. What is a formal description syntactic descriptions, eg. IDL, BNF)

Issue 90: Remove guideline on deprecation (GL6)

WG disagrees with issue, keeps as is by consensus decision.

october 10th, morning


Test "group"


Dimitris reminds people that he made a proposition for a Test WG. The rationale behind it is that "People will need more help for test suite issues than specifications."

We may have to split the WG to allocate specific resources to reply to concerns people have.

He's requesting to have the mandate from the IG to write a proposal and target the membership to get the resources to start such a thing.

Olivier is concerned by the fact that it may be will destroy the resources of the WG.

Dimitris makes the points that even if the Test WG is starting, he will allocate his time on the current WG.

Dominique makes the point that it may not be necessary to start a specific WG. We are already starting to receive request from people.

Kirill says that there's no particular problem if the work is started from the very begining. We can help people to start the work.

Olivier sees it as a possibility to attract more people (or destroy the WG, see remark above)

Karl says that he's afraid to have resources stolen from this new WG. Will have no concerns if we were a big one. would prefer to make mandatory some resources in the process document at the start of a WG. Chair, Team contact AND QA person.

Lynne proposes to do a CFP inside this WG and if we have a bunch of new members so we can split.
Olivier agrees with Lynne. but asks if it's possible to have a CFP for a task force inside the WG, because the charter is already addressing the scope of TS.

Dimitris thinks that people will jump on a test group.

Round call for a resolution
We want more resources before to start the Task Force on Test.
AI: Dimitris is writing a informal charter for this task force. Deadline: in 2 weeks.
AI: Karl asks to Janet if it's possible and how to do it and write a Call For Participation (include Dimitris in the loop)

specGL issues Processing

Issue 91 : Why should discretionary items be handled consistently within an implementation (CK8.4)?

Is an implementation should say that it must be A or B different from sometimes A or sometimes B. We are trying to redefine what's a discretionnary behaviour. Discretionnary items are bad as large. two counter arguments if you define conditions on DI is not anymore discretionnary. Mark Skall advocates that a DI must be specified even if it's specified. Dominique says we should group DIs in a profile to lower the conformance.
Lynne precises that when she wrote it, it was intended that a DI is consistent if the behaviour has not changed.
AI: Lynne redrafts the CP 8.4, Lofton volunteers to review it to clarify the vocabulary.

Issue 94: Why identify unused DoV (ckpt 10.5)?

We can predict all behaviours and DoV in the future. Specs are things which arelive. We don't need to list all unused DoV. It's overkill.
Remove CK 10.5, add a CK that the list of the used DoV in one place and the conformance clause.

Issue 95: Is it practical to "provide specific wording of conformance claims"?

If you put a conformance icon on a page does that = all the verbage of a conformance claim? Can't provide all the possible wording of a claim, move specific wordings to examples. List the minimum things that need to be included in a claim. We will require that conformance claims contain this minimal list.
Resolution: Minimum things should be contained in the conformance claim. But specific wording will be moved in the ExTech.
AI: Lynne and Dom rewrite it.

Issue 96: Why require Implementation Conformance Statement?

Yes it's nice to have, specifications are not only about technology.
Coming back to Alex with examples.

Issue 97: Distinguish normative and informative text

Priority 1 agreed.
rewrite to priority 1.

Issue 100: "Non-normative" is undefined, and used in place of the defined "informative"


Issue 98: Should test assertions be included in a spec?

What would happen if the Test Assertion is slighty different from the text in the specification? The text is clearly the only normative version. So the text has precedence on the TA.
Discussion about the normativity of TA and TA as large.
The list of TA is mandatory, TA are normative except in case of conflict with the text of the specification which will take precedence on it. (testGL must be rewritten to reflect the resolution, How TA must be mapped to the spec is to be defined.)

Issue 99: Scope of definition of test assertion

Remove the definition and rely on the QA Glossary. and issue 19 reopen

Issue 15: Should QA Framework address the topic of valid conformance claims about W3C standards?

AI: Lofton reshapes the issue.

Issue 64 : Use of Keywords SHOULD, SHALL, etc in SpecGuide
Issue 39 : Use of Keywords SHOULD, SHALL, etc in OpsGuide

Left to the editors discretion.



october 10th, afternoon

Lofton Henderson

Issues processing

Issue #74, What is the scope of the minimality of level 1 in a leveled technology?
Issue #75 : What is the relationship of modules and levels?
Issue #76 : What is the relationship of levels and profiles?

Do we have enough knowledge to be making mandatory rules for how levels/ profiles/modules relate? Yesterday we decided that we should not try to impose mandatory rules, but rather give general requirements in GL for documenting the use of prof/levels/modules, and in ET we will discuss possible relationships and examples.

Issue #77 : Should SpecGL have any different requirements for older specifications?

Consensus. No, just write SpecGL for new specifications.

Issue #78 : Should Guideline 8 have a checkpoint proscribing implementation dependent features?

require documentation of interop impacts as priority 1; ban them at priority 2.
P1 to require documentation of interop impacts. Keep current wording of potential bad interop effects of impl dependent, in GL or in new CK.

Issue #13: Is inter-standard or multi-standard interoperability within the QA scope?

AI on Dimitris : draft wording (informative text) about not introducing inconsistencies when using/referencing other specs"

Valid XHTML 1.0!
Created Date: 2002-07-25 by Olivier Thereaux
Last modified $Date: 2011/12/16 02:57:07 $ by $Author: gerald $

Copyright © 2000-2003 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.