Minutes of QAWG/IG f2f meeting, 16-18 June 2003, Heraklion, Crete

Table of Contents

About the meeting

Read the logistic page and detailed agenda for this meeting.

This joint QA WG/IG meeting was held at FORTH in Heraklion, Crete. The QA WG expresses its many thanks to FORTH, Prof Markatos and Anna Doxastaki for hosting the meeting. Also thanks to Prof. Markatos and Anna for hosting a wonderful feast. Many thanks to Anna for handling the local arrangements and for making sure that we had everything we needed to conduct a successful meeting.


Action Items

Below is an excerpt from the QAWG Action Items Table withaction items given during the meeting (and still opened after the meeting). The actual Action Items Table has up-to-date information about current AIs.

Id Owner Action Item Deadline
AI-20031016-1 DD answer question in last call about relationship of QA specs and guideline to ISO 9000 2003-07-16
AI-20031016-2 Dom & Lynne estimate the dates for Spec GL before we leave Crete
AI-20031016-3 Daniel create outline of document presenting arguments for value of "QA now" 2003-07-16
AI-20031016-4 Daniel draft new WG and IG charters 2003-07-16
AI-20030617-1 MS to develop test assertions for Test GL 2003-09-01
AI-20030617-2 LH to ask Ian Jacobs for clarification regarding LC29.4 by email 2003-06-20
AI-20030617-3 DH ask Ian Jacobs why he doesn't want to use RFC keywords and in case there are good reasons go for the mapping 2003-06-24
AI-20030617-4 Mark rewrite SpecGL 8.1 2003-06-30
AI-20031018-1 Mark, Lynne look for availability in their Boulder Office for a F2F 2003-07-11
AI-20031018-2 Patrick look for availability in Boulder and California locations for a F2F 2003-08-01
AI-20031018-3 Karl to create the markup and the style for a complete CP 2003-06-25
AI-20031018-4 Lofton integrate the new markup/styling 2003-07-15
AI-20031018-5 Dom to integrate the new markup/styling 2003-08-01
AI-20031018-6 Dom to reply to Pat Hayes' email 2003-06-26
AI-20031018-7 Lofton to reply to David that the QA WG is interested in TCDL and will have reviews on the proposed TOC 2003-06-256
AI-20031018-8 Patrick to find a reviewer at Sun on the TCDL Proposal 2003-08-01
AI-20031018-9 Dimitris review the TCDL Proposal 2003-07-01
AI-20031018-10 Dimitris make a review of the Dom level 3 Core draft against the latest SpecGL 2003-07-24

June 16th, morning

Mark Skall

The Raw minutes for this part of the meeting are available in the QAWG mailing-list archives. This document summarizes them in a more readable form.

Certification at W3C?

Daniel reported about the latest Advisory Commitee Meeting (meeting with representatives of W3C membership). The topic of certification was addressed there.

The WG tried to identify a potential candidate for a certification experiment and thougt of SVG tiny as a good candidate.

The discussion moved to the general question of the interest of certification versus education, since they share the goal of better implementations. Better communication, more advertising, easier-to use, free tests may achieve more usage and passage of the tests, but does not preclude certification later.

Some expressed their impression that neither W3C specifications nor the test suites are mature enough to ensure a proper certification program.

re-chartering the QA IG

The group reviewed the present scope of the IG in order to prepare its rechartering.

Analysis as follows:

How the QA activity is moving forward (i.e. QA of QA)
This is ok. its what we did this morning.
Sharing experience in validation of Web content, documents and protocols
First to review QAWG deliverables
Is this really going on? Should we include it when we re-charter? DD suggests removing "first"
Discussing QA role in W3C standardization cycle (from early draft to Candidate Rec)
This is mostly discussed in the WG
Issue with external funding, partnering with commercial sector, IPR
Mostly WG matter
Coordination with external organizations
Mostly done from within WG
Use of formal QA practices (ISO9000, etc)
We dont do that for the moment. If we will, it will be WG.
How to address certification: branding, service, logo, metrics
Yes, we did that. (although mostly handled by Team/WG...) Very important; also done within W3C at large
Education issues: QA tutorials, documentation, etc.
Yes. Done in IG. The Team is currently giving more power/freedom to the IG in this field.
Evolution of the Activity into a more comprehensive domain to guarantee full compatibility between all W3C Recommendations and the unification of the various groups doing specification reviews (like WAI PF, I18N, TAG)
Mostly done as coordination within Team

June 16th, afternoon

Patrick Curran

The Raw minutes for this part of the meeting are available in the QAWG mailing-list archives. This document summarizes them in a more readable form.

Education / Outreach : cost of QA now versus later

Need to convince people that QA now is cheaper than later. The "cost" later is lack of interoperability, and the saving is in test development (share the burden).

We want to write a document on this topic, addressed to both monopolists (e.g "we only need to develop for this product") and non-monopolists. Daniel will draft an outline.

List policies / logistics review

Discussing policies (notably anti-spam mechanisms) for our lists, and the scope of our lists in general. No specific decision taken.

Status of the QA framework

The group discussed the status and plan for completion of the QA framework.

And then remains the question of what the documents will be in the end... Recommendation or Note. (No decision reached this time)

Rechartering the QA WG

The group reviewed the present scope of the WG in order to prepare its rechartering.

Analysis as follows:

maintainance of an up-to-date QAWG Web site (/QA/WG/)
timely minutes of teleconferences and face-to-face meetings
up-to-date matrix of specifications/QA activities
glossary of QA terms and taxonomy used in W3C
Improved editor guidelines for specification writers (aka pubrules++ ), including conformance section, list of testable statements, list of explicit discretionary behaviors, etc...
This is SpecGL - new charter should state that we will continue to work on/improve Examples & Tools
report on QA review of specification conducted
Rewrite as ongoing proactive reviews of other WGs' specs
Or - do we want to create a summary report?
Can we create a feedback loop? How to capture best practices? We are committing to ongoing reviews, and to working to improve ExTech docs
make it clear that these are connected
issue resolution related to appeal/vagueness/interpretation of specs
Delete this - the WGs must define their own processes
proposed changes to W3C Process Document and standardization cycle (from early draft to Candidate Rec, going to the AB, as the group responsible for the evolution of the W3C process)
Rephrase this to refer to OpsGL (parallel to the SpecGL rewrite)
report of review of QA tools conducted in the activity (WG or IG)
This is similar to the stuff in the TTF charter
Should we have a separate task force?
Think carefully about how to phrase and balance/prioritize to bring in new blood
a framework/process for developing, managing and running tests (data driven, platform independent, formal output language for results - e.g. EARL, etc
This is Test GL/ExTech
Promoting existing Notes on QA as W3C Notes and developing new ones
An IG activity?
An ongoing activity - no need to call this out explicitly as a deliverable

June 17th, morning

Kark Dubost

The Raw minutes for this part of the meeting are available in the QAWG mailing-list archives. This document summarizes them in a more readable form.

Charter and Test Materials for our own Guidelines

The QAWG is in the process of rechartering and we want to conform to the QA OPs Guidelines. We will comply to the triple-A conformance. Daniel will write a draft charter and we will fill the holes in it.

Later in the discussion the group tried to define what "Test materials" mean for the Spec and Ops Guidelines. Our Examples and Techniques? Do we have Test Materials? Is the Check Point on giving Test Material applicable?

The discussion moved to the question of whether the Techniques were sufficient Test Material.

If the techniques do not cover the Test needed for a particllar CheckPoint, we clause which says that you can explain the techniques you have used to test it. But what's happening when the technique used is wrong. Is the test bad? Is the CP bad? The discussion is trying to nail all details of this.

Question: What is TM? Resolution: a questionnaire asking how the individual requirements are met.

Spec GL Processing

Processing the Test Assertion spin-off issues..


Any specifications which want to be AA compliant have to include Test Assertions but at the same time it's a big burden. Test Assertions have a value for good specifications.

It allows to improve the quality of the spec itself by making people writing a better specification.

Should the CPs must be P2 or P3? What is the exact value of this CP and how much does it require to do it?

SpecGL is not to make better Test Suites but better Spec. Are the testable assertions useful for the clarity of the language used in the spec?


P2: No disagreement

Warning wording: No disagreement

Auto-generation: Agreed on the wording done by Lynne.

Exemption from TA: This is not applicable to specification written in a formal language that allows for automatic generation of test materials

June 17th, afternoon

Dimitris Dimitriadis

The Raw minutes for this part of the meeting are available in the QAWG mailing-list archives. This document summarizes them in a more readable form.

Spec GL Last Call issues for test assertions

issue 13


Issue closed by the following action item : Mark to develop test assertions fo Test GL, completed by September 1

issue 14


dom: providing means to have a link from the specification to a separate document

lynne: need to include in the explanation of the checkpoint that they can be provided either in the document or by reference.


Need to Clarify this in the terminology. The Editors will add something to this effect.

Test assertions and conformance requirements


pc: identifying and labelling is necessary

lh: 6 terms in use, the only one defined is "test assertion"

lr: someone needs to draft the definition of "conformance requirement"


"conformance requirement" = necessary or recommended optional conditions for conformance to the specification

Lofton will to ask Ian for clarification regarding LC29.4

Issue 65


Issue 67


lh: conformance requirements do not have to use 2119 keywords. testable statements is undefined.

dom: if there are good reasons, provide mapping between RFC keywords and the conformance requirements


Dom will ask Ian why he doesn't want to use RFC keywords and in case there are good reasons, will go for the mapping

issue 73.9


top definition goes into the main glossary.
We'll make our definitions consistent.

issue 75.9


issue 106


issues 102, 104, 105


Reorder the guidelines with the current numbers to be GL13, GL10+GL3, GL11+GL12

levels of conformance

agreement to add wording explaining that prioritites 1-3 are included to progressively go from good to better to best as far as qualitative assessment is concerned

June 18th, morning

Lynne Rosenthal

The Raw minutes for this part of the meeting are available in the QAWG mailing-list archives. This document summarizes them in a more readable form.

Schedule for processing SpecGL

Disposition of Comments: Aug 1st
Next Publication to WG : Sept 1st

TestGL discussion

The group went through the current TestGL draft, and reviewed the document structure and content. The Test Guildelines deal with analysis, structure, and management of the test suite.

The group discussed different wordings, changes of priorities, etc. see the raw minutes for the details.

Review of Comments

David Marston's comments on Test GL

These comments apply to a previous TestGL draft

DM-1.1 Agreed

DM1.2 Need to make sure we cover versioning. Should metadata tie back to SpecGL DoV? We talk about filtering, but should filtering on the DoV allowed by the specification be called out? Yes. Want to include or exclude tests based on DoVs. Need to include a test purpose or description.

GL4: talking about test case review. Can say something about where get test data from put this is anaylsis guideline.

CP5.4: should there be mention of storage of results? Covered in test results management. We don"t mention anything about coverage. Need to add something about coverage.

Sandra Martinez's comments on Test GL

GL4: concepts have been clarified.

CP4.3 Agreed. Add a sentence to clarify.

CP5.1 Need to clarify and define terms. Use some of the suggested words.

Peter Fawcett's comments on Spec GL

1. Addressed

2. Agreed

3. need to look at intro

4. Relates to coverage and strategy. Will say something about coverage.

5, 6, 7. Agreed

Processing Plan for Test GL

WG draft by late September
Need ExTech document prior to first call
First Last Call: ??

Review of Test GL'sIntroduction

Disagreement on whether a scope can be a set of requrements. Class of products also included interoperability. Intended audience, delete conformance. Remove or move last paragraph of 1.4. Sections 1.5, 1.6, 1.7 make sure that wording is consistent with other documents. Need to create use cases for TestGL.

Review of Test GL's Conformance and Definitions

The group agreed that the definitions needed work, and the Conformance Section needed to be reviewed.

June 18th, afternoon

Dominique Hazaël-Massieux

The Raw minutes for this part of the meeting are available in the QAWG mailing-list archives. This document summarizes them in a more readable form.

Plans for TestGL

Next F2F

The group studied these possible locations :

  1. Boulder (Colorado)
  2. California (Silicon Valley or LA)
  3. backup if needed: Boston

NIST people will check the availability of in their Boulder Office for a F2F, and Patrick in Boulder and California locations, for a meeting around mid-October (for last call planning of TestGL)

Discussion about Karl's re-styling proposal for the Guidelines

General agreement that karl's proposal for the style of our Guidelines is good. The proposal is thus adopted, and other categories ("examples", "related CP") will be added to Karl's final work.

RFC 2119 keywords styling

Dom will reply to Pat Hayes' email by June 26

TCDL Proposal from David Marston

This work is something considered as part of what the WG should work on, and the QA WG would welcome it as a submission.

It was decided that Lofton will send this reply to David, while Patrick will find reviewer at SUN for it, and Dimitris will review it too.

DOM Level 3 Core review

Dimitris will make a review of the Dom level 3 Core draft against the latest SpecGL.

AI processing

It was decided that the chairs should check before each telecon which action items are overdue so that the AI list doesn't fill up indefinitely

TTF Charter

As a preparation for the charter of the Test Task Force (TTF), there was a discussion on the prioritarization of the possible deliverables, based on easyness vs usefulness.

  1. Test suite management framework (ref. VBWG)
  2. Templates for WG for such common test activities as:
    • Test materials contribution & review process
    • Challenge process
  3. Test assertion specification markup
    • Requirements
    • Prototype
    • Deployment
  4. Binary serialized infoset comparator
  5. A Note: Test authoring principles
  6. A Note: Test documentation (?) principles
  7. Research & Note: TS & TM description (?) and quality assessment
  8. Reporting tools
    • or others
  9. TCDL -- generic Test Case Description Language (metadata for TM, TS, TCs)

Results of a quick poll as follows, for each participant:

Note regarding Test assertion spec markup: there are two different projects: authoring spec with marking up of test assertions vs identifying test assertions in an existing spec with specific metadata (annotating).

Meeting adjourned.

Valid XHTML 1.0!
Created Date: 2003-03-25 by Olivier Thereaux
Last modified $Date: 2004/01/07 10:13:13 $ by $Author: dom $

Copyright © 2000-2003 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark, document use and software licensing rules apply. Your interactions with this site are in accordance with our public and Member privacy statements.