Issues list history

QA WG Issues List

Last update: $Date: 2004/09/28 00:51:45 $

Issues regarding the documents produced by the QA Working Group should be reported to www-qa@w3.org (public archives).

Comments on this list should be sent to the www-qa@w3.org mailing list (Archives).

Summary List of Outstanding Issues

num Status Spec Topic Class Date Title
79 Resolved TestGuide TestGuide Substantive 2002-09-03 Should Test Guidelines address how to write a good test suite?
45 Resolved WG-process WG-process Substantive 2002-01-14 How should the QA WG interact with TAG?
18 Active All-Framework All-Framework Editorial 2002-01-02 Should the Framework documents be published as Note or as Working Draft?
23 Active TestMaterials TestMaterials Substantive 2001-12-28 Tests for MAY/SHOULD assertions.

Detailed List of Issues

num Spec Date Topic Class Status Raised By Owner
1 IssueList 2001-12-20 IssueList Editorial Closed Mark Skall
Title: How should QA Issues be managed and tracked?
Description: [email] This was raised and discussed at Brussels f2f, 11/2001, without clear resolution. Bugzilla was discussed at that meeting. Another proposal (Henderson & Gavrylyuk): Maintain them in a simple XML grammar that can be XSLT-transformed into tabular or other document format (SOAP and DOM do this). Alternative (Henderson): SVG has a simple method, that seems to be based on a simple plain text master document, http://www.w3.org/Graphics/SVG/Group/issues.txt, which has very little prose text and relies mainly on references to the mail archive. Other alternative, to use SourceForge (per DOM-TS) was proposed at the 2001-12-06 QAWG teleconference. Discussed and finally resolved at 2001-12-20 QAWG teleconference.
Proposal:
Resolution: Use simple XML/XSLT scheme, similar to SOAP and DOM.
2 IssueList 2001-12-20 WG-process Editorial Closed Lofton Henderson
Title: How should issues be identified or numbered?
Description: What do we want for an issue number scheme? Alternatives: no number, use a mnemonic name, such as "issue-number-scheme"; or, simple ordinal numbering 1, 2, 3, ...(SVG does this); or, year and number 2001-1, 2001-2, ...
Proposal:
Resolution: Have simple ordinal 'issue-num' plus meaningful 'id'.
3 IssueList 2001-12-20 IssueList Editorial Closed Lofton Henderson
Title: Should editorial and substantive issues be in same Issues List?
Description: Some discussion at Brussels face-to-face that we should separate them. Given that much of our initial work will take place in the context of producing the Framework documents, it might not always be easy to separate issues ("doc" versus "substance"). Classifying or categorizing will increase management complexity.
Proposal:
Resolution: Editorial and substantive issues in single Issues List.
4 Minutes 2001-12-20 ActionItems Editorial Closed Lofton Henderson
Title: How should Action Items (AI) be managed and tracked?
Description: How should Action Items (AI) be managed and tracked? We need to keep the open AI visible, and also to document the when and how of closing AIs. No firm proposals, but some options. Option: Karl Dubost is working on tracking open items in an in-design "Agenda" addition to a re-organized QA Web site (unclear whether this tracks all AI, including competion and closure). Option: closure and completion is tracked in minutes of f2f and teleconferences (original de-facto method). Option: both the open-AI list and documentation of individual closures and completions is tracked only in minutes. Option: something like the method of SVG (see the issue about "Issue tracking").
Proposal:
Resolution: Action Items document (HTML table) will be kept in the WG web space. Updated by email solicitation and response before telcons, only discussed in telcons if any AIs are problematic. Updated with additions after telcons and f2f meetings.
5 All-Framework 2001-12-20 All-Framework Editorial Closed Lofton Henderson
Title: How should the QA Framework documents be partitioned?
Description: How should the QA Framework documents be partitioned? Input to Brussels face-to-face was single document. It was partitioned at Brussels into: Intro & Process Guidelines; Operational Guidelines; Spec Guidelines; Test Materials Guidelines. Working on the first two pieces, Henderson & Gavrylyuk propose to move the Process Guidelines from "Intro" to "Operational" document. See qaframe-intro-1202.html.
Proposal:
Resolution: Introduction; Process&Operational Guidelines; Specification Guidelines; Test Materials Guidelines
6 All-Framework 2001-12-20 All-Framework Editorial Closed Lofton Henderson
Title: what should be the schedule for FPWD of the first two Framework parts
Description: What should be the schedule for First Public Working Draft (FPWD) of the first two parts? At Brussels face-to-face, the decision was: mid-December and early-January. Since the substantive Process guidelines have moved into the "Operational" part (now "Process & Operational Guidelines"), should we still put out FPWD "Intro" by itself? Is it substantial and interesting enough, and can it stand alone without at least one other part (notice proposed linkage between Intro and the other parts).
Proposal:
Resolution: Publish Introduction and "Process & Operational Guidelines" at the same time (target: Mid-January 2002).
7 All-Framework 2001-12-20 All-Framework Editorial Closed Lofton Henderson
Title: What should be the names of the four Framework parts
Description: Names for the (current four) QA Framework documents? See proposed names in qaframe-intro-1202.html, section 1.1.
Proposal:
Resolution: "QA Framework:" followed by one of the four "Introduction", "Process & Operational Guidelines", "Specification Guidelines", "Test Materials Guidelines"
8 All-Framework 2001-12-20 All-Framework Editorial Closed Lofton Henderson
Title: What format for the Framework documents?
Description: Agreement at Brussels face-to-face, Guidelines and verifiable checkpoints, similar to the WAI standards. "Examples & Techniques" documents associated with the guidelines documents, to handle per-WG and taxonomy-specific details, examples, and implementations (similar to WAI "Techniques").
Proposal:
Resolution: per Brussels agreement.
9 OpsGuide 2001-12-20 OpsGuide Substantive Closed Daniel Dardailler
Title: Where should test suites reside?
Description: Where should test suites reside? The draft Framework input to Brussels face-to-face has a Guideline that says they should ultimately live in the responsible W3C WG. In the 2-jan-2002 draft Proc&Ops document, see section 2.2.2.2. This has since been questioned, email from Lynne Rosenthal. It was discussed at 2001-01-17 telcon, with apparent consensus around some basic principles: W3C/WG should have a stake in the work and be an active participant/partner; it must be open and freely available; must have stable, secure repository; must have dependable maintenance and update plan; etc. The scope of this issue is limited to: free availability and ensured stability of at least one high quality test suite for a WGs work. Revised in Guideline 2 of Process & Operational Guidelines (01-feb- 2002, first published version). See related issue #44. Closed at 14-feb-2002 telcon.
Proposal: Daniel Dardailler proposed an interpretation of the checkpoint which seems agreeable to all.
Resolution: Per proposal.
10 SpecGuide 2001-12-20 SpecGuide Substantive Closed Daniel Dardailler
Title: How will QA WG perform spec reviews?
Description: When, how, and for what Working Group will QA perform a "spec review"? QA reviews of WG specs is one of our documented QAWG deliverables. The issue was mentioned at Brussels face-to-face. No proposals resulted. See also issue #36

Discussed and closed at Montreal face-to-face (6/2002): QAWG, in general, will not do spec reviews for the specs of the Working Groups (no resources); however, each QAWG member will do one spec review according to the Spec Guidelines. We will keep a matrix on /QA/Group/, with the assignments, and the results will be kept and linked from there. These reviews are principally for the benefit of QAWG, to help in refining and advancing the Spec Guidelines document. QAWG will encourage the WGs (through the Team contact and/or chair) to apply the Spec Guidelines and do their own reviews. QAWG believes that WG spec reviews should be required in the future (see issue #16).

Proposal:
Resolution: As described above: QAWG will do some reviews for SpecGuide development purposes; otherwise it is the responsibility of the WGs.
11 SpecGuide 2001-12-20 SpecGuide Substantive Closed Daniel Dardailler
Title: How should QA spec guidelines relate to pubrules?
Description: How will QA guidelines, especially the "Specification Guidelines", relate to publication and style rules (Comm team)? Raised at Brussels face-to-face, needs to be pursued and defined.

See issue #25. This is redundant with part of the scope of that issue.

Proposal:
Resolution: See issue #25.
12 All-Framework 2001-12-20 WG-process Substantive Closed David Marston Daniel Dardailler
Title: How will QAWG interact with WAI and I18N WGs?
Description: How will QAWG interact with WAI and I18N WGs, and how will QA guidelines interact with WAI and I18N guidelines? Raised in email and at at Brussels face-to-face. The Framework document input to Brussels face-to-face stipulates certain levels of WAI conformance in a WGs test materials. But the issue hasn't been carefully discussed yet. A further dimension, after defining the relationship, is: where and how will it be documented in the QA Framework and/or QA Web site resources? (Related issues: issue #25, issue #45.) See proposed way forward in DD email: current separate ad-hoc review systems are okay for now; and, we (QA) need to work out clear problem statement, what we want to do and why wrt horizontal technical reviews.

Discussed again at Montreal face-to-face (6/2002). For lack of any other proposals, closed as suggested, the WGs will use their "separate ad-hoc review systems." Interaction of QA guidelines with those of WAI and I18N are not addressed.

Proposal: Per DD email, as just summarized above.
Resolution: Separate ad-hoc review systems of the WGs. No specific defined relationship between respective guidelines.
13 TestGuide 2001-12-20 TestGuide Substantive Closed David Marston Peter Fawcett
Title: Is inter-standard or multi-standard interoperability within the QA scope?
Description: Is inter-standard or multi-standard interoperability within the QA scope? Issue raised in email thread (DM, 2001-11-07, 2nd pgph).

Discussed at Tokyo face-to-face, DD given assignment to: draft wording (informative text) about not introducing inconsistencies when using/referencing other specs.

Discussed at Seattle face-to-face, and reclassified from OpsGL issue to TestGL. It doesn't apply to OpsGL. It applies to SpecGL, and is adequately addressed by the normative-references checkpoint (CP10.2).

In addition to addressing multi-standard test suites, TestGL needs to consider whether this is a new point in the test materials taxonomy.

Discussed at Boston face-to-face. Clarification, the originator's original issue description was: "Also say something to position the idea of testing across multiple Recommendations, possibly from multiple WGs. This is needed for certain kinds of software but may be wishful at this time, so I'd say that the notion should be acknowledged but positioned as a long-range objective. Some of the goals in 1.1 around common test tools would be more compelling if they could be used to test the interoperation of multiple Recs."

Resolved: in order to keep the first version of TestGL simple and implementable, the first version of TestGL will not address the issue.

Proposal: Originator proposes that multi-standard test suites are in-scope, but are lower priority and/or should be long-term goals.
Resolution: In order to keep TestGL simple and implementable, the first version of TestGL will not address the issue.
14 n/a 2001-12-20 WG-process Substantive Closed Daniel Dardailler
Title: How should Education & Outreach be carried out?
Description: How should Education & Outreach be carried out, specifically what sort of presence should QA have at conferences? Issue raised at Brussels face-to-face. The desireability is recognized, but the realization is not decided.

Discussed at 30030317 telecon. Resolved that QAWG won't attempt to define a blueprint or master plan, but rather will continue its current ad hoc way of addressing E&O. That involves time for IG topics and E&O planning at the start of each face-to-face meeting, and teleconference time as needed.

Proposal:
Resolution: To be reviewed and planned at each face-to-face meeting, and at telecons as needed, and executed accordingly.
15 All-Framework 2001-12-20 All-Framework Substantive Closed Lofton Henderson
Title: Should QA Framework address the topic of valid conformance claims about W3C standards?
Description: [email] Should QA (e.g., Framework) address the topic of valid conformance claims, especially when W3C test materials are involved? It is fairly well accepted that W3C does not intend to get into the certification business. There is email thread and Brussels face-to-face discussion about related issues: accreditation of third party certifiers; use (or abuse) and claims of conformance related to W3C test materials; logos and branding, such as XHTML, WCAG1.0, etc (see for example issue #32). There are probably several closely related issues to be separated here.

Discussed at Tokyo face-to-face, and changed from SpecGuide to All-Framework classification (topic). Within SpecGL, the related question has been affirmatively answered, "Should SpecGL have requirements that specs detail how to make a conformance claim?"

Discussed at Seattle face-to-face. OpsGL checkpoint 6.4 addresses "open disclosure", by requiring WG to associate proper conformance disclaimers with any test materials, and checkpoint 4.6 requires WGs to consider and address branding (icon) policy. Several checkpoints of a conformance-claims guideline (GL11) adequately address requirements for conformance claims guidance in specifications, including recommended content and required minimal content. Beyond this -- e.g., policing or enforcing claims -- QAWG believes it is not the business of QA Framework (it is the business of certifying bodies).

Proposal:
Resolution: As just described (Seattle resolution).
16 OpsGuide 2001-12-20 OpsGuide Substantive Closed Lofton Henderson
Title: Should the W3C Process Document be modified to require QA deliverables and schedule of WGs?
Description: Should there be modifications to the W3C Process Document, to require that Activity statements and/or WG Charters address QA deliverables? There are at least two aspects: require that Activity statements and charters document QA deliverables for Activity/WG; and, require some QA deliverables synchronized with Rec-track phases (e.g., test suites/tools before CR-exit.) Discussed at Brussels face-to-face. General feeling at Brussels seemed to be that it was unnecessary. The QA Framework will require it. The Activity Statements and Charters get AC review and approval. This will suffice to ensure that QA deliverables are properly addressed.

Discussed again at Montreal face-to-face, and subsequent email proposal suggested that Ops Guidelines probably ought to be something like an addendum to Process Document. Subsequent email contribution fully explored the options for "how" to make GL conformance mandatory.

See related issue #71, on whether Guidelines (GL) conformance should be mandatory or not.

Proposal:
Resolution: For now, QAWG will not propose any changes.
17 OpsGuide 2001-12-20 OpsGuide Substantive Closed Lofton Henderson
Title: Where should conformance test materials reside after a WG disbands?
Description: Where should conformance test materials reside "long term"? After a WG disbands, who is responsible for managing test materials? Issue was mentioned at Brussels face-to-face. No clear resolution. Closed at 4-apr-2002 QAWG teleconference.
Proposal:
Resolution: See 2002-04-05 Operational Guidelines checkpoint 6.1
18 All-Framework 2002-01-02 All-Framework Editorial Active Lofton Henderson Daniel Dardailler
Title: Should the Framework documents be published as Note or as Working Draft?
Description: [email] Should the Framework documents be posted as Note or as Working Draft? Currently, the document-template icon says "Note" but the Status says "Working Draft. Ian Jacobs proposed, "as a Working Draft", using both "Working Draft" titles and css style, see his email proposal.

Revisited at the Montreal face-to-face, and extended per subsequent email proposal. Initial publication as a Working Draft is reaffirmed. Thereafter,

  1. Progress to Last Call (LC);
  2. Incorporate LC comments and produce Candidate Recommendation (CR) text;
  3. Implementation phase during CR -- WGs trial application;
  4. Integrate final changes and publish finally as W3C Note.

After Montreal f2f, issue was closed with resolution: "As WD, using WD titles and stylesheets. Then progress through LC, CR, and finally to Note". At Team Project Review (member-only) of QAWG, June 2002, some team members argued strongly that Framework docs should complete processing to Recommendation (see around 13:42:12 UTC). Therefore the issue is reopened, in this detail only: should Framework progress finally to "Note" or to "Recommendation"?

See related issue #71.

Proposal: Proposal-1: As WD, using WD titles and stylesheets. Then progress through LC, CR, and finally to Note; proposal-2: "...finally to Rec".
Resolution:
19 All-Framework 2002-01-02 All-Framework Editorial Closed Lofton Henderson Lofton Henderson
Title: Should each Framework document have a "Glossary" appendix?
Description: Should each Framework document have a "Glossary" appendix?. Currently, the documents say in their terminologies section that unusual QA terms will be defined when first used, will be collected into a per-document glossary appendix, and will ultimately migrate to the central QA Glossary. Resolved at 4-apr-2002 QAWG teleconference ("no").

The resolution was changed to "yes" (or more accurately, "maybe" -- case-by-case per GL document) at an editors meeting in July, where the editors of Intro, OpsGL/Extech, SpecGL, and TestGL agreed that a "Definitions" section that develops with and tracks the document is essential to the quality and clarity of at least some of the evolving GL documents (SpecGL and TestGL). These definitions sections can also have more elaborate functional or contextual definitions, as compared to the terse QA Glossary. So it is possible that the definitions will *not* migrate to the QA Glossary (although the terms should migrate with an appropriate generic definition).

Discussed at Tokyo face-to-face, and re-opened with request to: reverse the resolution that Framework documents may have definitions that differ from those in QA Glossary (by expanding on and supplementing them with functional or contextual information). Subsequent email discussion.

Discussed again at 2002-10-16 telecon. Agreement about the process and coordination issues -- yes, GL documents need a definition section as they progress. Definitions not already in QA Glossary may migrate to QA Glossary at some point when GL document has stabilized (LC? CR? ...). No unanimity yet about the QA Glossary "identicalness" issue: should the definitions remain in GL document when/if they migrate to the QA Glossary. Options: no; yes, but only a pointer; yes, extended (optionally).

Details of "yes, extended":

  • call it a "Definitions" section (avoid confusion with QA Glossary);
  • initial section verbiage explains relationship to QA Glossary;
  • if term is in QA Glossary, point to that first in its definition;
  • (optionally) supplement (terse QA Glossary) definition with exemplary material, explanation or extension in particular GL document context, etc

Agreed to start new QAIG email discussion thread.

Discussed at Seattle face-to-face. The basic scheme above was re-affirmed (and a glossary project of W3C-wide scope, using a similar model, was discussed).

Proposal:
Resolution: Yes, each Framework part may have a definitions section, to be decided as needed on a case-by-case basis, to supplement and not contradict any existing QA Glossary definition.
20 Introduction 2002-01-02 WG-process Editorial Closed Lofton Henderson
Title: Should the QA WG web page have a central bibliography index to notes and other documents?
Description: Should the QA WG web page have a central bibliography index to Notes and other documents? This is described in the 2001-01-02 draft of the "Framework: Introduction", but it is not on the WG action items list. Resolved at 4-apr-2002 QAWG teleconference.
Proposal:
Resolution: Yes, it is an action item to develop one.
21 All-Framework 2002-01-02 All-Framework Editorial Closed Lofton Henderson
Title: Where should WG-only drafts of the Framework documents reside?
Description: Where should WG-only drafts of the Framework documents reside? Currently they are being put into ../QA/WG, but their "This version" indicates locations like ../QA/2002/01/.
Proposal:
Resolution: latest : /QA/WG/qaframe-part(.html) redirects to dated : /QA/WG/YYYY/framework-YYYYMMDD/qaframe-part(.html)
22 Introduction 2002-01-02 Introduction Substantive Closed Dimitris Dimitriadis
Title: Should the scope of the Framework Introduction be expanded?
Description: [email] Should the scope of the Framework Introduction be expanded? Currently, the Intro scope covers: overview of QA activity and goals; survey of QA resources; description and roadmap of QA Framework document family; and, guide and primer to using the framework documents. It has been proposed that the scope be expanded to include:
  • Description of the way the W3C QA Working and Interest Groups work
  • Interdependencies with bodies/groups in the W3C and other organisations
  • List of deliverables of the QA
In email discussion, items #1 & #3 were rejected. Item #2, which relates to issue #12, issue #25, and issue #37, needs definition, but the location of such description has not been resolved. Further discussion in the email thread resolved that Chapter 2 -- Resource Survey -- of the 2-jan-2002 draft should be removed and the information put on the web site.
Proposal:
Resolution: The "Introduction" should introduce the Framework documents, not the QA Activity. No to #1 and #3. Remove "Resource Survey" to Web site. (Uncertain Re: #2)
23 TestMaterials 2001-12-28 TestMaterials Substantive Active Dimitris Dimitriadis Active
Title: Tests for MAY/SHOULD assertions.
Description: Should tests of MAY/SHOULD assertions be part of a conformance suite? They are useful, so we need to discuss these tests in the Framework documents, probably in Test Materials Guidelines and/or Spec Guidelines. Note that this issue applies to test suites related to W3C recommendations, not to the Framework documents themselves. issue #39 deals with normative language in the Framework documents themselves. The issue was discussed at the 2002-03-01 March QAWG face-to-face meeting, although the documentation of the discussion seems to refer more to issue #39. The sense of the discussion was "yes", test for SHOULD and MAY assertions in technical standards (Recommendations). The form of the tests and how they enter into the "conform/non-conform" verdict will depend on how the specification is written, specifically how it defines conformance related to SHOULD and MAY assertions.

Discussed at Boston face-to-face. The minutes are unclear of the exact resolution, but indicate that MS and DM have action items to (respectively) raise the corresponding issue in SpecGL and draft a resolution.

Proposal: Yes, method and verdict TBD by conformance statements of Recommendation.
Resolution:
24 WG-process 2001-12-28 WG-process Substantive Closed Kirill Gavrylyuk
Title: Appeal process when QA WG rejects a request from WG
Description: Do we need any appeal process? It is currently implied in section 2.6.1 of 2-jan-2002 Proc&Ops draft. See also Issue #52 and Issue #53. At 4-apr-2002 QAWG teleconference, it was decided to postpone this until discussion resumes on Issue #53.

Discussed at 30030317 telecon. Resolved that there won't be any formal appeal process. Reasons. Requests that are within the scope of external relations requests (see draft QA Process document) will be handled if resources are available. If there are no resources, that is essentially not subject to appeal.

Proposal:
Resolution: No formal appeals process will be defined.
25 All-Framework 2001-01-14 WG-process Substantive Closed David Marston Daniel Dardailler
Title: How should QA interact with W3C teams, notably the Comm team.
Description: [email] The QA Framework documents will include a part on "Specification Guidelines". The Comm team has normative pubrules, and (currently) non-normative "Style Manual". How will QA interact with Comm, and how will the respective specifications relate? A further dimension is: where in the QA Framework documents or other resources should this be documented? Related issues: issue #12; issue #45.

See proposed way forward in DD email In particular, note that (aside from documents), there is a second aspect of "how relate" -- how will QA use Comm to promote and publicize its work and products? This has not been addressed.

Discussed again at Montreal face-to-face (6/2002). Liaison re-affirmed as adequate solution. "Spec Guidelines" mentions pubrules and Style Manual, and their role for editors. Issue declared closed.

(Note. One aspect of the issue has been discussed on "Chairs", without conclusion. The proposal -- advocated by DD for example -- that pubrules, Style Manual, Spec Guidelines ought eventually to be combined into one document with normative and informative parts.)

Proposal: For now, establish liaison/point-of-contact with Comm (Dominique Hazaël-Massieux, most likely)
Resolution: Liaison to communicate significant events, such as publication of new Spec Guidelines versions, and request feedback.
26 OpsGuide 2001-12-28 OpsGuide Substantive Closed Lofton Henderson Kirill Gavrylyuk
Title: Conformance section
Description: This document should have a conformance clause. Uncertain what it should say now, as it is unclear what is the normative force of this document. At least it should probably address what is involved in (possibly voluntary) conformance to this document. Precisely what it might say is an issue that perhaps we can address later? Discussed in 2002-01-03 telcon, and resolved: similar to WCAG 1.0 clause.
Proposal:
Resolution: Similar to WCAG1.0, including three levels of conformance based on prioritized checkpoints (p1, p2, p3).
27 OpsGuide 2001-12-28 OpsGuide Editorial Closed Lofton Henderson Kirill Gavrylyuk
Title: Need test maintenance guidelines/checkpoints
Description: Need to distill guidelines/checkpoints from maintenence section. See 2.5.1-4 of 2-jan-2002 Proc&Ops draft.
Proposal:
Resolution: See Guideline 8 of Proc&Ops, first published version.
28 OpsGuide 2001-12-28 OpsGuide Substantive Closed Kirill Gavrylyuk Active
Title: Test maintenance task force
Description: Is it obligatory to have a task force for test materials maintenance? Should we use SHOULD or SHALL in the text? Working Group SHOULD/SHALL designate a task force or another entity to maintain the test suite, as well as maintenance procedures. See 2.5.1 of 2-jan-2002 Proc&Ops draft, and Guideline 8 of Proc&Ops, FPWD (1-feb-2002 first published WD). Closed at 4-apr-2002 QAWG teleconference.
Proposal:
Resolution: See resolution in text of 2002-04-05 Operational Guidelines, guideline 8.
29 OpsGuide 2001-12-28 OpsGuide Substantive Closed Lofton Henderson
Title: Should there be a checkpoint about publication of testing results?
Description: Do we need guidelines/checkpoints for publications of testing results? See 2.4.6 of 2-jan-2002 Proc&Ops, and checkpoint 6.7 of Proc&Ops FPWD (1-feb-2002, first published WD). Closed at 4-apr-2002 QAWG teleconference.
Proposal:
Resolution: Yes, see 2002-04-05 Operational Guidelines checkpoint 6.5
30 OpsGuide 2001-12-28 OpsGuide Editorial Closed Lofton Henderson
Title: Is test contribution review process a checkpoint or example?
Description: It is not clear whether we want review process to be a guideline or just an example. See 2.4.5.4 of 2-jan-2002 Proc&Ops draft.
Proposal: Both - a checkpoint and example(s), the latter to be in in "Ex&Tech" when it is written.
Resolution: Both. See checkpoint 6.5 of Proc&Ops FPWD (1-feb-2002 first published WD).
31 OpsGuide 2001-12-28 OpsGuide Editorial Closed Kirill Gavrylyuk
Title: Transfer of test material implies Rec phase is reached
Description: Text for transfer of test materials from outside implies that spec in question has already Recommendation status. Need to add wording/bifurcation for the other case. See 2.4.4 of 2-jan-2002 Proc&Ops draft.
Proposal:
Resolution: Moot. Issue disappeared in Proc&Ops FPWD (1-feb-2002 first published WD), see guideline 7.
32 OpsGuide 2001-12-28 OpsGuide Substantive Closed Lofton Henderson
Title: Guidelines/checkpoints for Branding
Description: Is Branding in scope of QA WG? What guidelines/checkpoints do we offer for Branding? Shall it be separate section or just an item under subsection "Scope"? This is contentious issue; also borders on "conformance claims", which was a big, hot www-qa argument.)

- WAI does this for content; CSS also and basic HTML validation.

- Controversial topic, as email thread and UAAG CR comments show.

- Taxonomy dependent: content (WCAG, CSS, etc) is less difficult/controversial; UAs will be contentious.

Discussed at Seattle face-to-face. Beyond the OpsGL checkpoint that requires WGs to address the topic of branding, it is considered out of scope for QAWG to try to specify further details or requirements.

Proposal:
Resolution: Detailed branding policy requirements are out-of-scope. Requiring that WGs address the issue is in-scope.
33 OpsGuide 2001-12-28 OpsGuide Substantive Closed Lofton Henderson Kirill Gavrylyuk
Title: Guideline for minimum level of WG commitment to QA
Description: Do we need a guideline/recommendation for minimum level of WG commitment to QA? See 2.2.1 of 2-jan-2002 draft. First published version of the Process & Operational Guidelines has de-facto proposal, "level-3". 2002-01-28 telcon decided to leave issue open till after first publication (2002-02-01). Resolved at 4-apr-2002 QAWG teleconference, resolution is implemented in 2002-04-05 Operational Guidelines document.
Proposal: As just stated, "level-3".
Resolution: Table wording revised, at-least-level-3 is P1, higher levels (subsuming level-3) are P2/P3.
34 OpsGuide 2001-12-27 OpsGuide Substantive Closed Daniel Dardailler
Title: Should QA bind entry/exit of spec milestones to QA deliverables?
Description: Should we provide any guidelines for exit criteria bound to spec milestones in the document? See 2.1 of 2-jan-2002 Proc&Ops draft.
Proposal: Significant test materials by CR-exit.
Resolution: See checkpoint 1.2 of Proc&Ops FPWD (1-feb-2002, first published WD).
35 WG-process 2001-12-27 WG-process Substantive Closed Daniel Dardailler
Title: Resources supplement to other WGs
Description: Issue raised at Brussels face-to-face, whether QA WG should provide resources to some WGs for building test materials. Lynne suggested involve universities, setup internship programs. See 2.6, 2.6.1, 2.6.4 of 2-jan-2002 Proc&Ops draft.

Discussed at 30030317 telecon. Resolved it is neither affordable nor desirable -- QAWG has neither the resources nor the domain expertise. QAWG does, however, intend to provide generally useful tools, toolkits, and frameworks that should be helpful to individual WGs that are building their own test materials.

Proposal:
Resolution: QAWG will not provide staff resources to WGs for direct participation in building test materials. QAWG does intend to provide generally useful tools, toolkits, and frameworks.
36 OpsGuide 2001-12-27 OpsGuide Substantive Closed Lofton Henderson
Title: Should QAWG initiate proactive reviews?
Description: It is still an open policy question, whether the QA Activity will pro-actively initiate reviews, and whether it has adequate resources to do so. The given wording in the Procs&Ops document may be soften a bit. If we want to do this proactive, should we also define some preliminary steps where by QAWG suggests, requests permission, informs WG? See 2.6.2 of 2-jan-2002 Proc&Ops draft. See related issue #10.

Discussed and closed at Montreal face-to-face (6/2002): QAWG, in general, will not do reviews for the QA aspects -- Operations, Specifications, Test Materials -- of the Working Groups (no resources); however, each QAWG member will do one review of each of the three types, according to the relevant Guidelines document. We will keep a matrix on /QA/Group/, with the assignments, and the results will be kept and linked from there. These reviews are principally for the benefit of QAWG, to help in refining and advancing the three Guidelines documents. QAWG will encourage the WGs (through the Team contact and/or chair) to apply the Guidelines documents and do their own reviews. QAWG believes that WG QA-guidelines reviews should be required in the future (see issue #16).

Proposal:
Resolution: As described above: QAWG will do some reviews for Guidelines documents development purposes; otherwise it is the responsibility of the WGs.
37 OpsGuide 2001-12-27 OpsGuide Substantive Closed Kirill Gavrylyuk Active
Title: Coordination of QA work with external entities
Description: Can an arbitrary WG request QAWG help in coordination between the WG and external entities? Examples where coordination might be desired:
  • joint QA-related activity;
  • materials transfer between external and WG;
  • "arms-length review" of WG products.
About "transfer", see 2.4.4 of 2-jan-2002 Proc&Ops draft. Closed at 4-apr-2002 QAWG teleconference.
Proposal: Yes.
Resolution: Yes, see 2002-04-05 Operational Guidelines, Chapter 3.
38 OpsGuide 2001-12-27 OpsGuide Substantive Closed Kirill Gavrylyuk
Title: Resolutions of external QA requests
Description: Should QAWG formally assist in resolution of external QA related questions that involves one or more W3C standards. See 2.6.5 of 2-jan-2002 Proc&Ops draft. Closed at 4-apr-2002 QAWG teleconference.
Proposal: Yes, QAWG should redirect them to the appropriate WG.
Resolution: Yes, see 2002-04-05 Operational Guidelines, Chapter 3.
39 OpsGuide 2001-12-27 OpsGuide Substantive Closed Lofton Henderson Lofton Henderson
Title: Use of Keywords SHOULD, SHALL, etc in OpsGuide.
Description: [email] Need to determine SHOULD, SHALL, MAY, etc keywords use throughout Proc&Ops document. Note. As of the Proc&Ops FPWD (1-feb-2002, first published WD), most of the normative language is replaced by Priority 1, Priority 2, Priority 3 checkpoints. See issue #55, regarding the latter. Note that this issue applies the Framework documents themselves, not to test suites related to W3C recommendations. issue #23 deals with test suites related to W3C recommendations.

Inter-related set of issues: #101, #80, #57, #39, #64

Discussed at Seattle face-to-face. All SHOULD and MAY usages were considered, and endorsed or changed.

Proposal: Editors to make initial choice, QAWG to do per-keyword review (prerequisite to marking issue "Closed").
Resolution: As reflected in OpsGL Last Call text (20030210).
40 OpsGuide 2001-12-27 OpsGuide Editorial Closed Lofton Henderson Kirill Gavrylyuk
Title: Test Materials home
Description: Currently (2-jan-02 draft) there is an overlap between 2.2.2, 2.5 and 2.4.5.5 (materials publication). Need to add at least pointers to the these 3 sections. (Note, there is a substantive issue on this topic as well, issue #9.) Closed at 4-apr-2002 QAWG teleconference.
Proposal:
Resolution: Moot, by virtue of text cleanup in subsequent versions.
41 OpsGuide 2001-12-27 OpsGuide Editorial Closed Lofton Henderson Kirill Gavrylyuk
Title: Guidelines/checkpoints are not numbered
Description: Guidelines and checkpoints are not yet numbered. Need to do before final version. (By adding to editors' XSLT stylesheet for transforming unformatted, unnumbered sections and guidelines/checkpoints.)
Proposal:
Resolution: "G 1. Statement of gdln..." "C 1.1. Statement of ckpt..."
42 Introduction 2002-01-08 n/a Editorial Closed Daniel Dardailler
Title: Where should the Taxonomy document be kept?
Description: Raised in QAWG teleconference. The Taxonomy document currently sits in the QA Web space, linked from several locations on QA web pages. It is currently a short (1-page) outline document with no descriptive text. Alternatives:
  • Leave it as it is, a linked resource in QA Web space
  • integrate it into "Framework: Introduction"
  • other?
Resolved at 4-apr-2002 QAWG teleconference. It is not normatively referenced from any Framework documents yet, maybe from "Test Guidelines" later. Stand-alone is okay for now.
Proposal:
Resolution: Leave it as a separate document.
43 All-Framework 2002-01-08 certification Editorial Closed Daniel Dardailler
Title: How should the Certification Note relate to the Framework(s)?
Description:

NIST (Lynne Rosenthal) submitted a document about Certification to the QA Activity. We need to work out what is the relationship between this note and the Framework's.

Discussed at Seattle face- to-face, and resolved that it does not relate to the QA Framework. It is a reference document to be kept in The W3C QA Library

Proposal:
Resolution: No special relationship to the QA Framework, but a valuable reference document for The W3C QA Library.
44 OpsGuide 2002-01-26 OpsGuide Substantive Closed Mark Skall
Title: Should W3C endorse externally produced test suites?
Description: Split off from issue #9 in 2001-01-17 teleconference. If W3C assumes control of a test suite, then in that case there is a quality assessment and (implicit) endorsement. If a W3C WG actively participates in a co-developed or externally developed test suite, there is also (implicit) endorsement in this case. These cases are covered by issue #9. In other cases, where W3C has no active participation, should W3C give endorsement to one or more externally developed test suites? Discussed and closed at 2002-03-01 March QAWG face-to-face meeting. See related issue #58
Proposal:
Resolution: No, do not endorse suites. But QA will link them from TheMatrix, provided that owner allows QA to publish the results of applying any appropriate QA Checklist (assessment) documents (e.g., future "Test Guidelines").
45 WG-process 2002-01-14 WG-process Substantive Resolved David Marston Daniel Dardailler
Title: How should the QA WG interact with TAG?
Description: [email] How will QAWG interact with TAG. Where and how will it be documented in the QA Framework and/or QA Web site resources? Similar issue (for the "horizontal groups" WAI and I18N) raised in email and at at Brussels face-to-face. Related issues: issue #25, issue #12. See proposed way forward in DD email.

Discussed again at Montreal face-to-face (6/2002). Issue put into hibernation ("postpone") until someone proposes some specific, actionable resolution.

Discussed at 30030317 telecon. Resolved that QAWG will designate a liaison (initially DH) to communicate QAWG concerns to TAG, and to be first point of contact within QAWG for TAG concerns. This will role will be documented on our Web site (as will as other roles like Lead Editors, owners of other documents and Web pages, etc). [KD takes action to make such documentation.]

Proposal: Establish liaison/point-of-contact, most likely with Ian Jacobs (TAG lead editor).
Resolution: QAWG liaison to TAG established. [Documentation on QAWG pages tbd.]
46 OpsGuide 2002-01-23 OpsGuide Substantive Closed Kirill Gavrylyuk
Title: Should "QAWG relationship to WGs" be normatively described with prioritized checkpoints?
Description: How the QA WG relates to other WGs -- expertise, requests, resources, reviews, etc -- is described in Chapter 3 of the 18-jan-2002 draft of "QA Framework: Process & Operational Guidelines". There are no guidelines or checkpoints. Is this how it should be? Raised originally in email by KG on editors list, discussed by DD/KG in email thread, apparent consensus. Note. Whether it is normative or informative is a separate issue -- currently there is normative language usage, e.g., "SHOULD" requirements.
Proposal: Leave it without checkpoints and explain that at beginning.
Resolution: Leave it without checkpoints and explain that at beginning.
47 OpsGuide 2002-01-23 OpsGuide Substantive Closed Kirill Gavrylyuk
Title: Should test materials be published in TR space?
Description: [email] Should test materials -- i.e., test suites and test tools -- be published in /TR/ space? Question has been raised as a reasonable option. WG discussed and resolved "no", for these reasons:
  1. Publishing Test Materials to TR space is a bad idea because of the dynamic nature of Test Materials. This dynamic nature takes two main forms. First there may be additional submissions to a test suite after the Rec has been published to TR space. Second there may be errata or a need for new cases that weren't anticipated when the Rec was finalized.
  2. Publishing Test Materials in the TR space dilutes the TR space. A related suggestion is to have Test Materials be published to their own web space. A few suggestions were put forward, primarily WG/QA or WG/Test.
  3. Keeping the Test Materials in a WG/Test space keeps the Test Materials under the control of and responsibility of the WG itself.
Proposal: No, test materials should not be published in TR space.
Resolution: No.
48 TestMaterials 2002-01-23 TestMaterials Substantive Closed Daniel Dardailler
Title: Should the planned Technical Guidelines be a guideline-checkpoint part of the Framework doc family?
Description: [email] The initial plan for the Framework document family specified a fourth part, consisting of a document pair, "Technical Guidelines" and "Technical Examples and Techniques". Three issues were raised:
  • "Technical" is bad choice of name
  • does this part really fit guidelines-checkpoints model, or is it a "test materials resource guide?
  • should it be part of the Framework document family?
The naming question has been raised before and a less confusing name seems like a popular idea. The gd-ck question is not as simple -- there clearly are some good practice principles (e.g., test case traceability) which seem appropriate as conformance requirements on test materials. Where would they be, other than "TM Guidelines"?
Proposal: 1.) Change "technical" to "Test materials"; 2.)...; 3.)...;
Resolution: Change name, keep guideline-checkpoints plus example-techniques parts of Framework family.
49 OpsGuide 2002-01-28 OpsGuide Substantive Closed Andrew Thackrah Lofton Henderson
Title: Should there be a global (W3C-standard) license for use and distribution of test materials?
Description: [email] There were originally two aspects to the issue:
  1. licenses under which external contributions are accepted by WG;
  2. licenses under which WG provides its test materials to the public;
Discussed at 2002-01-28 teleconference. The second aspect (#2) -- W3C redistribution and use license -- is addressed in checkpoint 6.3 of 2-February Process & Operational Guidelines (first published WD). It recommends, "use Document license if possible, Software license if necessary (i.e., no-modification document license won't work with the test materials). Some other license could be forced by external configurations. Agreed to split the checkpoint: "open and free availability of W3C distributed or endorsed test materials under whatever license is used"; and type of license. Further discussion at 14-feb-2002 telcon. Discussed again at the 2002-03-01 March QAWG face-to-face meeting. Aspect #2 closed at 4-apr-2002 QAWG teleconference, when Issue #59 (aspect #1) was split off as a separate issue.

See survey of current practice within W3C, generated subsequent to closure of the issue.

Reopened at Montreal face-to-face, with presentation of a proposed test materials license (KG, to be circulated). Further suggestions regarding accommodation of commercial test suites raised in email thread.

  1. proposal-1, "use Document license if possible, Software license if necessary";
  2. proposal-2, KG contribution (distributed paper form at Montreal).

Discussed at 2002-10-21 telecon, proposal-2 agreed by QAWG. KG to send electronic copy to LH immediately, LH to review with Joseph Reagle before publishing.

Discussions with Joseph Reagle have caused the issue to be reopened again. He is not convinced that a new license is necessary. The remaining point of contention is whether the lack of scope-of-use restrictions in the W3C TM publication policy and license will prevent some companies from contributing test materials to W3C. (Pending)

Discussed at Seattle face-to-face. A compromise was adopted, to try to address objections to introducing and maintaining a new W3C license. Use doc license if possible, software or W3C approved custom license if necessary; propose TM license as an example of license with scope of use restrictions. (There is still ongoing discussion between Kirill and Joseph.)

Note. Subsequent to completion of OpsGL Last Call text (20030210), new comments were received. The issue will likely be re-opened as a Last Call issue.

Issue summary presented and briefly discussed at Boston face-to-face (20030306).

Reopened during Last Call issues resolution, discussed and closed at 20030501 telecon.

Proposal: proposal-2, KG contribution. (First aspect is Issue #59.)
Resolution: If the Document License will work, it has attractive characteristics; if it won't work (non-modification and/or usability reasons), use the Software License; may use different ones for test documentation, test cases, and test software; consult with W3C Legal if neither Software nor Document works for you
50 OpsGuide 2002-01-28 OpsGuide Substantive Closed Andrew Thackrah Lofton Henderson
Title: How should W3C deal with potential legal liability arising from third-party use of test materials?
Description: [email] If third parties use W3C-supplied test materials to do certification or any sort of third-party testing of others' products, there is potential for W3C liability for improper statements or claims by the third party. Discussed at 2002-01-28 teleconference. Discussion whether checkpoint 3.3 of Process & Operational Guidelines (1-feb-2002, first published version) provides adequate liability protection. Discussion focussed on the accuracy of the statements in checkpoint 3.3. Some people thought that the statements need to include the context dependencies -- i.e., the conditionality (levels, profiles, etc) of conformance statements from subject Recommendations. Further elaboration in email from originator indicates two sub-issues in addition to the checkpoint 3.3 question:
  • need for W3C to have adequate "caveat emptor" in its licenses
  • conformance claims and implication of W3C approval
The 2nd sub-issue relates to Issue #15.

Discussed at Seattle face-to-face. Resolved that it is adequately handled by the existing W3C distribution licenses.

Proposal:
Resolution: It is adequately handled by the existing W3C distribution licenses.
51 SpecGuide 2002-01-29 SpecGuide Substantive Closed Karl Dubost Lynne Rosenthal
Title: How should Spec Guidelines deal with deprecated features and withdrawn features?
Description:

[email] Resolution of this issue needs to start with agreement on the definitions of "deprecate" and "withdraw". The original question was "What does conformance mean in the presence of deprecated features?" There seems to be some agreement that the conformance claim necessarily depends on how the spec is written to address deprecated and withdrawn features. That in turn depends on type of Rec/spec and the test materials taxonomy position. Peripheral questions to be answered include, what to do if the spec does not address the issue, i.e., what should be a default conformance interpretation. Some agreement that QAWG needs to define a menu/option approach to guide the WGs in writing specs.

It is addressed in Guideline 9 of FPWD Specification Guidelines (2002-05-15, first public draft). Discussed at 2002-10-28 QAWG telecon (@@DRAFT@@), declared closed (de facto) with resolution: handle with a deprecation-policy guideline (now GL9) and checkpoints, add definitions of deprecate and withdraw to SpecGL.

Proposal: See David Marston email proposal.
Resolution: Handle with a deprecation-policy guideline (now GL9) and checkpoints, add definitions of deprecate and withdraw to SpecGL.
52 OpsGuide 2002-01-29 OpsGuide Substantive Closed Lynne Rosenthal
Title: Does the QA Working Group need a "Process Document"?
Description: [email] In draft rewrite of Chapter 3 of the Process & Operational Guidelines (1-feb-2002, first published version), Lynne Rosenthal excluded much of the process-oriented information and recommended that this information be put in a separate document that she called, "Request of Assistance Process". Just as other WGs should have a TS-Process document (guideline 6), the QAWG should have a Process document. This document would describe the process for formal consultation, active reviews, establishing teams of reviewers, etc. Additionally, it is important that QAWG discuss what our process actually is - the Framework:P&O editors provided a starting place for discussion and the WG should discuss and decide how it will actually provide assistance. There are two Issues.....

1. Should we have a Process Document? (this issue)

2. What is the process for handling requests from the WGs? (Issue #53)

. How does someone formally request assistance, who is the point-person responsible to ensure that requests are processed, is there a task team appointed ad-hoc for each request, what is the timeframe for answering, etc. The issue was discussed and closed at the 2002-03-01 March QAWG face-to-face meeting.
Proposal:
Resolution: Yes, QAWG needs a process document.
53 WG-process 2002-01-29 WG-process Substantive Closed Lynne Rosenthal
Title: What is the process for handling requests from the WGs?
Description: [email] In draft rewrite of Chapter 3 of the Process & Operational Guidelines (1-feb-2002, first published version), Lynne Rosenthal recommendeded that information concerning intra-QAWG process for handling WG requests belonged not in Frm:P&O, but in a new, separate "Request of Assistance Process". See related related Issue #52 for more description. Specific request-response aspects to be decided include: how does someone formally request assistance? Who is the point-person responsible to ensure that requests are processed? Is there a task team appointed ad-hoc for each request? What is the timeframe for answering? Etc. The issue was discssed at the 2002-03-01 March QAWG face-to-face meeting; decided to postpone further discussion until strawman draft of QAWG process document is written and circulated.

Discussed at ???. Agreed that it should be reclassified from OpsGL issue, to WG-process issue (QAWG Process Document).

Discussed at 30030317 telecon. This is exactly the purpose of Section 2 of the QAWG Process Document (see draft QA Process document).

Proposal:
Resolution: Defined in Section 2 of the QAWG Process Document.
54 OpsGuide 2002-02-12 OpsGuide Editorial Closed Daniel Dardailler
Title: Several structure and content issues about Proc&Ops Guideline 6.
Description: [email] In the email thread, a several sub-issues are raised with Guideline 6 of Process & Operational Guidelines FPWD (1-feb-2002, first published WD):
  • it is too long - ought to be split or else checkpoints combined;
  • checkpoint 6.1 is not verifiable and shouldn't be there as a checkpoint;
  • checkpoint 6.2 looks like a duplicate of 6.5, or 6.5 + 6.2 - in any case, it looks like it's already covered by following checkpoints;
  • checkpoint 6.7 is similar to 3.2 and 3.3.
It was agreed to postpone resolution of the issue(s) until after FPWD. Discussed and resolved at the 2002-03-01 March QAWG face-to-face meeting.
Proposal:
Resolution: New Frm:Ops draft breaks it up, 6.1 becomes "Note", other bits are clarified.
55 OpsGuide 2002-02-14 OpsGuide Substantive Closed Lofton Henderson Kirill Gavrylyuk
Title: All of the checkpoint priorities in Proc&Ops should be confirmed.
Description: The checkpoint priorities in the Proc&Ops FPWD (1-feb-2002, first published WD) have not been explicitly discussed and endorsed by the QAWG. At the 14-feb-2002 telcon, QAWG split this out of issue #39, and agreed to discuss and confirm all priorities. Discussion and further processing at 2002-03-08 QAWG teleconference, covered checkpoints through 4.1, in the 2002-02-25 Operational Guidelines, editors draft. Processing finished and minuted at:
Proposal:
Resolution: Checkpoints added/modified/removed as documented in above-referenced telcon minutes.
56 All-Framework 2002-03-01 All-Framework Substantive Closed Mark Skall
Title: All guideline/checkpoint documents must have associated objective test criteria.
Description: The issue was raised at the 2002-03-01 March QAWG face-to-face meeting. Since QA guideline/checkpoint documents are standards to be applied to the process and content of W3C test building activities, they should have test suites just like any other W3C recommendation. As stated by the originator, this issue applies to all W3C guideline/checkpoint standards, not just those of QA activity. There was discussion about what constitutes a "test suite" for a Gd/Ck specification, without any conclusion or consensus. One alternative that was offered: a checklist, together with a "Techniques" document that defines suitable ways to satisfy each checkpoint.
Proposal:
Resolution: Agreed in principle, details TBD.
57 OpsGuide 2002-03-11 OpsGuide Substantive Closed Kirill Gavrylyuk Kirill Gavrylyuk
Title: Use of RFC2119 in Operational Guidelines
Description: [email] We aren't using words "MUST", "SHOULD", "MAY" as RFC2119 normative keywords. (Is this correct? See for example the usage of "must", "should", etc, in 2002-02-25 Operational Guidelines, editors draft.) How can we reflect this in the the Terminology section? Should we change the document back to use them? Alternatives: We can go either way:
  • a) Do not use RFC2119 and put a disclaimer in terminology section: The words "must", "must not", "required", "shall", "shall not", "should", "should not", "recommend", "may", and "optional" may be used throughout the document without any normative meaning defined in [RFC2119].
  • b) Keep using the keywords and specify (e.g., in the conformance clause) that keywords "MUST", "SHOULD", etc. in the checkpoint text and in the description text have normative meanings as prescribed in [RFC2119].
Resolved at 4-apr-2002 QAWG teleconference.

Inter-related set of issues: #101, #80, #57, #39, #64

Proposal:
Resolution: Alt b: RFC2119 keywords in checkpoint text and description text have the normative meanings for determining conformance within the checkpoint.
58 All-Framework 2002-03-01 All-Framework Substantive Closed Karl Dubost
Title: Should W3C rate test suites?
Description: After some discussion in [email] of what icon to use in The Matrix, for indicating the existence of test suites, another thread continued and broaded the discussion, which raised more fundamental issues: what is the value in the proposed QA guideline/checkpoint ratings of test suites? And, should QAWG pursue the rating of test materials with such rating tools? The issue was discussed and resolved at the 2002-03-01 March QAWG face-to-face meeting (see in those minutes, the discussion of issue #44). The resolution acknowledges:
  • the gdln/ckpt specs can't measure "goodness" or some detailed technical quality aspects of test materials;
  • they can measure conformance to some known, good-practice process and technical aspects;
  • this information can be useful, and can also encourage conformance to good practices;
  • the matrix should include reference to any test materials that agree to allow publishing of the gdln/ckpt results.

See related issue #44

.
Proposal:
Resolution: Yes, see above.
59 OpsGuide 2002-04-4 OpsGuide Substantive Closed Andrew Thackrah Lofton Henderson
Title: Should there be a global (W3C-standard) license for submitted test materials?
Description: [email] This was originally part of Issue #49, which (initially) dealt with the two topics:
  1. licenses under which external contributions are accepted by WG;
  2. licenses under which WG provides its test materials to the public;
Topic #2 is Issue #49.

Discussed and resolved at the 20021216 telecon, more or less as suggested in email proposal. Related to this resolution, it was agreed to revise OpsGL checkpoint 5.3 ("WG must publish an acceptable TM submission license") to be more consistent. New proposed CP5.3 text requires a submission license *policy*, not a draft license itself.

2003-01-03, reopened following email from Joseph Reagle, about new W3C-wide contribution license and policy. This does not necessarily change this issue's resolution, nor the related CP5.3. However, the existence of these materials and policies has not been considered previously in this issue's discussions, and the materials need to be looked at before final closure.

Discussed at Seattle face-to-face. Resolved to add a link in the discussion to W3C-wide contribution terms and form, which in turn has reference to W3C licenses.

Proposal: No. (See email proposal.)
Resolution: No.
60 OpsGuide 2002-04-15 OpsGuide Substantive Closed Lofton Henderson Lofton Henderson
Title: What are the QA commitments and responsibilities of existing Working Groups?
Description: [email] In the 2002-04-05 Operational Guidelines document, the checkpoints don't clearly address existing Working Groups.

The introductory section 1.3, "Navigating..", says: "This document is applicable to all Working Groups, including those that are being rechartered or already exist. Working Groups may already be doing some of these activities and should review the document and in so far as possible incorporate principles and guidelines into their work"

The first couple of guidelines -- QA responsibility, QA commitment, resource allocation, etc -- are all written for new groups. There is no mention of how an existing group should make its commitment, the TS responsibilities of a group that has published a Rec and has rechartered or is rechartering. For example:

  • in-progress towards Recommendation, but already chartered (e.g., XFORMS)?
  • done w/ a first Recommendation, but moving on to further work (e.g., SVG, XSLT, XML)?

In the first couple of Guidelines/Checkpoints, it is unclear what such a WG is recommended (required?) to do. Modify charter? Produce another sort of commitment document? Discuss and minute the commitment and resource allocation? Other? Proposal to fix the problem -- none yet, but one or more of the following options might be appropriate in Ops Guidelines:

  1. reword the guidelines and checkpoints, or add new ones (i.e., there would be "applicability" here -- some ckpts apply to new WGs and some to old WGs).
  2. add descriptive prose addressing "old groups"
  3. for new and old WGs, add criteria to Ops-Extech for pass/fail ("verdict criteria")
  4. Remove "In the charter" from statement of GDs and CKs, add treatment of new/old to conditional test requirements within CKs (and discuss in Extech).

Currently (2002-10-21), the resolution is closest to #2 (e.g., in 2002-05-15 OpsGL draft). "Responsible" proposes option #4.

Discussed at 2002-10-21 telecon, option #4 agreed by QAWG.

Resolution of this and resolution of issue #67 are inter-dependent.

Proposal: Option #4.
Resolution: Remove "In the charter" from statement of GDs and CKs, add treatment of new/old to conditional test requirements within CKs (and discuss in Extech).
61 SpecGuide 2002-04-22 SpecGuide Substantive Closed Mark Skall
Title: Should standard terminology be required?
Description: [email]Guideline 4 of the Specification Guidelines (20020515 publication) discusses what needs to conform by explicitly addressing an enumerated list of "classes of products". This issue addresses whether or not we should require specifications to categorize their classes of products as being one of the items within this enumerated list or whether specification developers are free to describe their classes of products in their own terms. In a larger sense, the issue also addresses whether to constrain specification developers to standard terminology to describe other things in the specification.

(Additional reference: DM paper at QAWG Workshop, April 2002.)

Discussed at Tokyo face-to-face, closed with resolution: per test assertion(s) in Checkpoint 2.1, use the SpecGL terminology for product classes, when your class is one of those defined in SpecGL GL2. Else use and define your own -- list is not exhaustive.

Proposal: Yes, require standard terminology by stating the requirement in a lower-priority checkpoint.
Resolution: Use the SpecGL terminology for product classes, when your class is one of those defined in SpecGL. Else use and define your own -- list is not exhaustive.
62 All-Framework 2002-05-24 All-Framework Editorial Closed Karl Dubost
Title: What convention should QA Guidelines documents use for conformance labels?
Description: [email] The current method of designating levels of conformance in the Guidelines documents of the QAWG Framework is: A, AA, AAA (good, better, best). This is similar to the WAI method of A, double-A, triple-A. It is thought that this might create confusion, and some other methods are suggested in the mail thread of this issue.

Discussed and closed at 2002-05-30 telcon.

Proposal:
Resolution: Leave as is -- A, AA, AAA.
63 SpecGuide 2002-05-28 SpecGuide Substantive Closed Lofton Henderson Lofton Henderson
Title: All of the checkpoint priorities in Spec Guidelines must be confirmed.
Description: The checkpoint priorities in the SpecGuide FPWD (15-may-2002, first published WD) have not been explicitly discussed and endorsed by the QAWG. Similar to issue #55 (which applies to OpsGuide), all checkpoint priorities must be discussed and confirmed. See related issue #64, about the use of normative keywords. Background -- the guidelines and checkpoints themselves were discussed and reviewed at QAWG teleconferences, see:
Proposal:
Resolution: Closed at 2002-08-14 telecon.
64 SpecGuide 2002-05-28 SpecGuide Editorial Closed Lofton Henderson Lynne Rosenthal
Title: Use of Keywords SHOULD, SHALL, etc in SpecGuide.
Description: [email]

The use of keywords SHOULD, SHALL, MAY, etc throughout the SpecGuide document must be confirmed. Document reference: SpecGuide FPWD (15-may-2002, first published WD). Most of the normative content in that specification version is represented by imperative language in Priority 1, Priority 2, Priority 3 checkpoints.

However if the resolution of OpsGL issue #57 is applied to SpecGL as well, then these keywords have the RFC2119 normative meanings when used within the text and description of checkpoints.

The scope of this present issue is limited to whether the correct normative keyword (MUST or SHOULD or ...) is being used in each place. See related issue #63, regarding the whether checkpoints have the right priorities.

Inter-related set of issues: #101, #80, #57, #39, #64

Discussed at Tokyo face-to-face. Conclusion: editors to make initial choice, QAWG to do per-keyword review. Although it was labelled as closed, the per-keyword review was *not* done. Subsequently decided the issue is not closed until per-keyword review is done. Another checkpoint criterion established: Every ckpt should have at least 1 normative aspect - each should have at least one MUST. At 2002-10-16 telecon, it was discussed and re-affirmed that the per-keyword review is needed (in particular, only to confirm all SHOULD occurrences?).

Discussed at Seattle face-to-face. The resolution for SpecGL effectively eliminates any normative keywords except for MUST, from the conformance requirements. (This may have been an explicit resolution, but it is not minuted.)

Proposal: Editors to make initial choice, QAWG to do per-keyword review (prerequisite to marking issue "Closed").
Resolution: As reflected in SpecGL Last Call text (20030210).
65 Glossary 2001-05-28 Glossary Editorial Closed Lofton Henderson Mark Skall
Title: Level of detail in QA Glossary definitions.
Description: [email] The issue was first raised in an email thread about new glossary items. The definitions in the current QA Glossary (2002-05-28) are fairly terse and dictionary- like. It was suggested in the thread that definitions of complex terms and concepts should be much richer, similar to the level of information for example in WAI's WCAG 1.0 Recommendation. Discussed at the 23-may-2002 telcon, without resolution.

Discussed at Montreal face-to-face (6/2002). No general agreement on level of detail. However, agreed that commentors not satisfied with detail should submit proposed wording for replacement definition, which will be accepted or treated as a QAWG issue for discussion.

Proposal:
Resolution:
66 Glossary 2001-05-28 Glossary Editorial Closed Karl Dubost Karl Dubost
Title: Should the QA Glossary be maintained as dated versions?
Description: The issue was raised in the 23-may-2002 telcon. Previously we decided (issue #19) that we won't have glossaries in individual Framework documents. Therefore the Framework documents (including those in /TR/) must refer to the central "QA Glossary". This suggests that dated Framework documents should have a stable reference.

Discussed and closed at 2002-05-30 telcon.

Proposal: Maintain dated versions in /QA/WG/YYYY/MM/ space, with dated names, with redirect from /QA/glossary.html.
Resolution: Yes (KD and MS to propose details).
67 All-Framework 2001-05-29 All-Framework Substantive Closed Lofton Henderson Lofton Henderson
Title: What should be the scope of the Framework "ExTech" parts?
Description: As originally envisioned, the Framework parts,

have a scope that includes not only real-world examples and case studies, but also specific enumeration of techniques that satisfy the checkpoints. In the latter sense, they contain something like alternative "verdict criteria" for each Checkpoint -- if the target under consideration does one of the techniques, then it satisfies the checkpoint. Presently, the Ops-ExTech is a collection of case studies that illustrate how several cases relate to the operational checkpoints.

The issue is: should the scope be limited to examples and case studies, or should the ExTech parts be developed per the original scope? Options could include:

  1. examples only;
  2. add techniques asap;
  3. examples for now, try to develop and add techniques later;
  4. etc.

Techniques will raise a number of additional issues: must (can) the enumeration be exhaustive? how precise (i.e., verifiable) can these be in diverse operational, specification, and test environments? how long would it take to develop exhaustive, verifiable techniques and do we have the resources? etc?

Background/Reference. See the Abstract of "HTML Techniques for Web Content Accessibility Guidelines 1.0". In the WAI model, the techniques are intended to "help authors of Web content who wish to claim conformance". It further disclaims that "while the techniques in this document should help people author HTML that conforms [to WCAG 1.0], these techniques are neither guarantees of conformance nor the only way an author might produce conforming content."

Discussed and closed at Montreal face-to-face (6/2002). Basically option #2 above. QAWG members who review OpsGL and SpecGL will make contributions. The Extech documents will include both per-checkpoint case study material, and per-checkpoint "virtual example" (techniques) material.

See related issue #56

Proposal:
Resolution: The Extech documents will include both per-checkpoint case study material, and per-checkpoint "virtual example" (techniques) material.
68 All-Framework 2001-05-29 All-Framework Substantive Closed Daniel Dardailler Lofton Henderson
Title: What should be the status of the "ExTech" parts, and how should they relate to their "Guidelines" parts?
Description: The issue was raised in the 23-may-2002 telcon. It relates to issue #67, and probably depends on that resolution. As originally planned, the three ExTech parts would be separate standlone Technical Reports (WD at least), in separate /TR/ directories from their Guidelines parts. It has been suggested that these could alternatively be:
  1. the 2nd part of two-part (Guidelines+Checkpoints) TR (instead of separate TR),
  2. informal, referenced, companion documents maintained in /QA/WG/ space instead of /TR/,
  3. other.

In all configurations (including current /TR/), the two parts are closely cross-linked. The "informal" option probably saves staff resources.

(Related issue not yet recorded -- "how to maintain" -- can an open-edit or open-contribute scheme be used in /QA/WG/?)

Discussed at Montreal face-to-face (6/2002), without resolution.

Discussed again at 20020918 telecon. Followup mail presented proposal for closure, which most closely resembles #2 above. A key point is that the ET parts have no normative content -- they are completely informative, showing how the checkpoints of the GL part "might be satisfied." By keeping them in /QA/WG/ for now, they can be changed and improved more frequently than the GL parts in /TR/. The GL parts in /TR/ would be linked to redirects in /QA/WG/ which are updated to point to the best current ET version. The ultimate disposition -- /TR/ or /QA/WG/ -- can be revisited when the GL parts (and the ET parts) stabilize more.

Discussed at Tokyo face-to-face, closed with resolution: as described in the previous paragraph, with some refinements.

  • links from older /TR/ versions should not break;
  • so links from /TR/ to /WG/ should be to *dated* redirects;
  • those can be updated to point to evolving ExTech versions, until next /TR/ publication;
  • when stabilized, ExTech can move to /TR/ as a Note (timing-- editor's discretion).
Proposal: As described above and in email proposal
Resolution: Extech remain in /QA/WG/ space, but linked from /TR/, and eventually (when "stabilized") be moved to /TR/ as W3C Note. Details above.
69 SpecGuide 2002-05-29 SpecGuide Substantive Closed Dan Connolly Lofton Henderson
Title: Should Spec Guidelines discourage "flavors of conformance"?
Description: [email] In the FPWD (2002-05-15) of Specification Guidelines, there is no language in "Guideline 3. Specify flavors of conformance." indicating that options/levels/flavors are bad. Originator proposes that there should be. It was discussed some in the email thread, without resolution.

Discussed at 2002-05-30 telcon.

Discussed at Montreal face-to-face (6/2002), without definitive resolution. However, there is general agreement that excessive variability hurts interoperability, and this will at least be reflected with warnings in new versions of SpecGuide (see for example August 2002 published SpecGuide WD).

The remaining problem is: is there a quantified or objectively verifiable variability constraint that should be placed on specifications? It was agreed to move forward by reorganizing SpecGuide according to DM email proposal. This in turn will be the basis for further input from WGs and continued discussion on the IG list. A specific "quantified" proposal for a checkpoint to limit to 3 dimensions was discussed but not accepted -- too inflexible to fit all the sorts of W3C specs.

See related issue #70

Discussed at Tokyo face-to-face. Summary follows.

  • It is not necessarily bad to have some variability (we have abandoned the terminology "flavors of conformance".)
  • Depends on the Use Case.
  • Not inherently bad, but can be implemented badly, or there can be too much or "unnecessary" variability.
  • DoV discussion handles some of this.
  • SpecGL warns about too much DoV and requires that specs explain of why they need a particular DoV.
  • QAWG doesn't think that we can quantify it.

Proposal: Originator proposes that minimizing (esp. zero) flavors should be a design goal for all W3C specs.
Resolution: Continue to warn about excessive variability, require justification of DoV used, don't attempt (in SpecGL) to further quantify it.
70 SpecGuide 2001-05-29 SpecGuide Substantive Closed Lofton Henderson Lofton Henderson
Title: Spec Guidelines needs a specific enumeration of "flavors of conformance".
Description: [email] The email thread indicates that there are widely differing understandings of what is meant by "flavors of conformance" in "Guideline 3. Specify flavors of conformance." of the FPWD (2002-05-15) version of Specification Guidelines. Before it can be argued whether flavors are good or bad, the statement of the guideline needs to be clarified. It needs a specific enumeration of all of the sorts of flavors to which it applies. So far, only "Strict Conformance", "Unconditional Conformance", and "Conditional Conformance" are mentioned.

Discussed at 2002-05-30 telcon.

See DM email proposal. This proposal was discussed Montreal face-to-face (6/2002). It was agreed to reorganize the SpecGuide document accordingly, in which case the "flavors" go away, a complete enumeration of "dimensions of variability" affecting conformance in specs is implemented, and "flavors of confomance" GL becomes a "conformance policy" GL.

See related issue #69
Proposal:
Resolution: As decided at Montreal (see above).
71 All-Framework 2001-06-13 All-Framework Substantive Closed Dominique Hazaël-Massieux
Title: Should WG conformance with QA Guidelines documents be mandatory?
Description: [email] Opened and discussed again at Montreal face-to-face, closed according to subsequent email proposal. QAWG consensus is that it should be mandatory for WGs to:
  • apply the Guidelines (GL) documents when the GL documents are in CR phase (see issue #18) -- CR being the "implementations" phase (for the GL in this case);
  • conform to the GL documents when they (GLs) finish CR and are published as W3C Notes

Whether or not these mandatory requirements are documented in the Process Document was not specifically addressed. The mechanism is still unresolved. Another email contribution fully explores options about "how" to make it mandatory. The post-Montreal email proposal suggested simply: an addendum to Process Document requiring Ops GL conformance; and amendment to pubrules requiring Spec GL conformance.

See related issue #16.

Proposal: See above.
Resolution: Yes, mandatory use and mandatory conformance, according to the above proposal.
72 SpecGuide 2002-07-10 SpecGuide Substantive Closed Lofton Henderson
Title: What is the definition of "normative use case"?
Description: [email] Opened at 2002-07-10 telecon. Two framework documents mention "normative use case":

There are two problems: "use case" means widely differing things to different people, ranging from careful prose description to rigorous coded scenario (SpecGL says "formally defined terms"); and "normative" implies that some testable requirements are placed on some target. The first problem (definition) begs for clarification and examples (see checkpoint 1.3!). And the second problem (normativeness) raises several questions. What is the target of such a requirement: an instance or implemenation of some class of product? or, the specification itself in which the use case is found? And finally what is the nature of the testable requirement?

Proposal: Proposal above. Plus proposals from LRAT
Resolution: Definitions of use case and user scenario adapted from UML. GL1 and CK1.1 reworded, made informative by default. See 2002-08-26 SpecGuide.
73 SpecGuide 2002-08-22 SpecGuide Substantive Closed Dominique Hazaël-Massieux
Title: Should SpecGL require atomicity of modules?
Description: [email]

A number of W3C specifications (e.g., SMIL 2.0, XHTML Modularization) impose constraints that modules be atomic (indivisible) in order to qualify for some conformance designations. It was proposed that this should be made into a checkpoint of Guideline 4 (GL4) of SpecGuide. It was counter-proposed that this was premature, and also that the current "clean design" wording should be removed.

Pending discussion and resolution of the issue, the 2002-08-26 SpecGuide reflects a middle option: GL4 describes the attributes of atomicity, present examples, and requires (in a checkpoint) that specifications that chose modularization must address the atomicity question.

Discussed at Tokyo face-to-face, closed with resolution: leave it as it is in checkpoint 4.3 (2002-08-26 SpecGL version) - do not require atomicity, but if modules are used, then the spec must address atomicity. Move atomicity definition from GL verbiage into checkpoint, move other discussion (recommendations) into ExTech.

Proposal:
Resolution: Don't require atomicity, but require that modularized specs address atomiticy.
74 SpecGuide 2002-08-22 SpecGuide Substantive Closed Lofton Henderson
Title: What is the scope of the minimality of level 1 in a leveled technology?
Description: [email]

In email threads such as the above, there is discussion that level 1 in a leveled technology is the core or minimum. Within the levels dimension of variability, that is agreed to be true. It has been suggested (see the email, see @@minutes? spoken in telecon but apparently not recorded or minuted@@) that level 1 should be minimal across all dimensions. I.e., profiles would not subset level 1, modules would not subdivide level 1.

There are counter-examples in W3C practice, and no examples have been put forward yet. Still there is the issue: in principle, is this a spec (or technology, more precisely) architectural principle that is agreed; and if so, should SpecGuide encourage or require it?

Related issues: issue #75 and issue #76.

Discussed at Tokyo face-to-face, closed with resolution: Do not try to define/mandate a "correct" relationships amongst DoV in SpecGL, discuss the possibilities in Spec Extech.

Proposal:
Resolution: Do not try to define/mandate a "correct" relationships amongst DoV in SpecGL, discuss the possibilities in Spec Extech.
75 SpecGuide 2002-08-22 SpecGuide Substantive Closed David Marston
Title: What is the relationship of modules and levels?
Description: [email] Also discussed in the 2002-07-24 telecon.

Modules and levels are two ways to divide a technology. When both are present in a specification, there is the question of whether one subdivides the other. I.e., levels subdivide modules, or do modules aggregate together to build levels? There are no examples of the former, although a clean design case can be postulated. It is unclear whether the latter is found in W3C specifications (a careful look is needed at the modularized specifications), although it can be postulated how such an architecture might evolve with "historical levels". It has been observed about the latter, that profiles would be a better choice than levels.

The issue that is raised is: should SpecGuide take a position on the goodness or badness of the two design possibilities. I.e., is one relationship better than the other and to be encouraged or required? (Other alternatives: no, neither is preferable; or, these two dimensions should not be used together.)

Related issues: issue #74 and issue #76.

Discussed at Tokyo face-to-face, closed with resolution: Do not try to define/mandate a "correct" relationships amongst DoV in SpecGL, discuss the possibilities in Spec Extech.

Proposal:
Resolution: Do not try to define/mandate a "correct" relationships amongst DoV in SpecGL, discuss the possibilities in Spec Extech.
76 SpecGuide 2002-08-22 SpecGuide Substantive Closed Lofton Henderson
Title: What is the relationship of levels and profiles?
Description: [email] (none)

This issue is raised, but not yet resolved, whether SpecGuide should promote requirements about how profiles and functional levels relate, i.e., should particular architectures be encouraged or discouraged, prescribed or proscribed?

The (current) 2002-08-26 SpecGuide describes some possibilities, without taking position. From Guideline 3, "A profile can require a particular level in its entirety. A profile can also be defined as a subset of a level or levels. Hierarchy is an essential attribute of levels. While it is not an essential attribute of profiles, nevertheless profiles can be hierarchical in the sense that Profile B can be defined as all of Profile A plus additional capabilities." From Checkpoint 3.5, "levels [...] could depend on profiles."

Related issues: issue #74 and issue #75.

Discussed at Tokyo face-to-face, closed with resolution: Do not try to define/mandate a "correct" relationships amongst DoV in SpecGL, discuss the possibilities in Spec Extech.

Proposal:
Resolution: Do not try to define/mandate a "correct" relationships amongst DoV in SpecGL, discuss the possibilities in Spec Extech.
77 SpecGuide 2002-08-22 SpecGuide Substantive Closed Lofton Henderson Lynne Rosenthal
Title: Should SpecGL have any different requirements for older specifications?
Description: [email]

First raised in QAWG telecon of 2002-08-07, the issue concerns whether or not SpecGuide should, under any circumstances, give special consideration to legacy specifications in any checkpoints -- specs that precede the publication of SpecGuide. The QAWG decided that the general answer is "no" -- SpecGuide is being written for new specifications. That notwithstanding, QAWG also believes that the wording of checkpoints should not arbitrarily or unnecessarily exclude legacy specifications which meet the checkpoint in spirit. (See last paragraph of section 1.2 of 2002-08-26 SpecGuide.)

The specific case in point here concerns a previous statement that existed in many of the dimension of variability (DoV) checkpoints, that required specifications to document whether or NOT they used a DoV. The telecon discussion shows problems with the negative disclaimer requirement, not only for legacy specifications but going forward if new DoV are added. The interim solution is a new priority-2 checkpoint, 10.5, which collects all negative disclaimers (so legacy specs could still be A-conforming), and the various DoV checkpoints (3.1, 4.1, 7.1, etc) have dropped the NOT requirements.

This legacy exemption needs to be discussed and endorsed, or else eliminated. If kept, then the style and structure of 3.1, 4.1, etc need to be looked at.

The issue is raised again in email thread about checkpoint 10.5 in the 20020826 published WD of SpecGL. Should documenting DoV for existing specs be in the SpecGL? Although useful, this information should be in the Examples & Techniques or perhaps elsewhere. Is a target specification required to address all dimensions of DoV, even those it doesn't use? There is no need to establish a dimension that is not being used - that violates 'minimal set' requirement.

Discussed at Tokyo face-to-face, closed with resolution: do not try to accommodate past work, i.e., do not try to structure the checkpoints so that legacy documents might pass easier. Goal is to make new specifications better. Change front matter -- to indicate that these guidelines target new specs, although existing specs can use the guidelines to see how they do. Remove CK10.5 and related stuff.

Proposal:
Resolution: Do not try to accommodate past work, focus is to make new specifications better.
78 SpecGuide 2002-08-22 SpecGuide Substantive Closed Lynne Rosenthal Lynne Rosenthal
Title: Should Guideline 8 have a checkpoint proscribing implementation dependent features?"
Description: [email] (see sub-thread at the end of message).

The 2002-08-26 SpecGuide discusses (in Guideline 8) discretionary items. There are flavors: discretionary choices (implementation choses from 2 or more well defined behaviors); optional features (impl choses whether or not to implement a well-defined feature); implementation-dependent features (what the implementation does is completely free and undefined).

It is suggested that the last one -- implementation-dependent features - is considerably worse for interoperability than the others and deserves a checkpoint of its own (either ban them at some priority level, or at least require that specifications address the interoperability impact of allowing them.)

Discussed at Tokyo face-to-face, closed with resolution: add a new checkpoint, or new MUST requirement within CK8.2, "MUST document interoperability impact of any implementation dependent features". A checkpoint to ban implementation dependent features at P2 or P3 was discussed inconclusively -- not accepted at this time.

Proposal:
Resolution: Add a new checkpoint, or new MUST requirement within CK8.2, "MUST document interoperability impact of any implementation dependent features".
79 TestGuide 2002-09-03 TestGuide Substantive Resolved Alex Rousskov
Title: Should Test Guidelines address how to write a good test suite?
Description: [email]

The 2002-07-01 TestGuide draft (1st discussion draft for QAWG telecon input) addresses:

  • what a good test suite should be
  • how to make a good test suite

Because the guidelines in the second category are often subjective and limit developer options, originator suggests moving them to a separate informational document if not completely eliminating them.

Discussed at , without clear resolution. However, it was decided that the next TestGL draft (Oct or Nov 2002) would at least disambiguate in each checkpoint, what are required results or characteristics in a TS (test suite) -- "deliverables" -- and what are process aspects. With that clarification, it will be possible to decide whether to keep the process-oriented bits, remove them, or migrate them to the TestET (Examples & Techniques) document, when the latter is written.

Discussed at Boston face-to-face. Agreed that TestGL should describe the results -- i.e., the test materials -- not the process of producing them. TestGL "Scope" should clarify this.

Proposal: Originator proposes removing "how to" guidelines from the TestGuide document.
Resolution: Agreed. TestGL will describe the test materials, not the process. TestGL "Scope" will clarify this [tbd].
80 SpecGuide 2002-09-24 SpecGuide Editorial Closed Alex Rousskov Lynne Rosenthal
Title: RFC 2119 Keywords are used inconsistently in the SpecGL.
Description: [email], [origin]

Reference: 20020826 Spec Guidelines draft (WD)

The document does not use MUST, SHOULD or MAY keywords as defined and recommended in RFC2119 -- upper case. This is how the words are presented in section 1.4, "Terminology", however there are numerous usages in SpecGL which are lower case, capitalized, etc. Alternatives:

  • upper-case all occurrences;
  • or, lower-case all occurrences (including in sec 1.4);
  • or, in section 1.4 state that usage in body of text is lower case, but still has RFC2119 meaning;

The related issue #64 and (OpsGL) issue #57 deal with the substantive aspects of RFC2119 usage.

Inter-related set of issues: #101, #80, #57, #39, #64

Discussed at Tokyo face-to-face, closed with resolution: use uppercase in all occurrences that have normative meaning. Note this in "Terminology" subsections. Editors Action Item: apply this to all Framework Guideline documents.

Proposal: Upper-case all occurrences.
Resolution: Upper-case all occurrences, where they are used with RFC2119 normative meaning. Note this in "Terminology" subsections.
81 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Definition of specification testability
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

There is confusion and disagreement as to the meaning of specification testability. The current definition states that there needs to be a finite cost-effective process that checks that "the requirement is met". Discussion regarding finite and testing behavioral specifications; what does 'check that requirements are met' mean. As pointed out in email thread, current definition was taken from an IEEE definition (@@reference?@@).

Discussed at Tokyo face-to-face, closed with resolution: Change "the requirement" to "all requirements", take out "finite cost effective", and split into two sentences: one defines single testable requirement, and the other uses that to defines testable specification.

Proposal: (Numerous conflicting proposals in email thread.)
Resolution: Change "the requirement" to "all requirements", take out "finite cost effective", and split into two sentences: one defines single testable requirement, and the other uses that to define testable specification.
82 SpecGuide 2002-09-24 SpecGuide Editorial Closed Alex Rousskov Lynne Rosenthal
Title: Re-title Guideline 1, replacing "use cases" with "scope".
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

"Scope" would better define the guideline objective and can be normative, whereas use cases are complementary and should not be normative. Does the current wording reflect the entire guideline and convey the appropriate message or will it be misconstrued? Scope is covered by checkpoint 1.1

Discussed at Tokyo face-to-face, closed with resolution: change Guideline 1 to "Define Scope."

Proposal: Originator suggests to rename GL1, "Define scope."
Resolution: Change Guideline 1 to "Define Scope."
83 SpecGuide 2002-09-24 SpecGuide Editorial Closed Alex Rousskov Lynne Rosenthal
Title: Clean up subjective and imprecise wording.
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Subjective qualifiers such as 'clearly' need to be removed. These are not testable terms. Note that there is a lurking issue here -- the phrasing of the checkpoint is merely a "title" for the checkpoint, and the normative requirements are carried in (RFC2119) language within the checkpoint description. This is the WAI model, and implied by the resolution of issue #57.

Additional email thread discussion.

Discussed at Tokyo face-to-face, closed with resolution: agree that subjective imprecise language should not be in the checkpoint titles, and must not be in any test assertion statements (requirements) within the checkpoints.

Proposal: Remove subjective, imprecise qualifiers from checkpoint titles, as well as from any normative language within the checkpoint.
Resolution: Remove subjective, imprecise qualifiers from checkpoint titles, as well as from any normative language within the checkpoint.
84 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Should examples and user scenarios be lower priorities?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

It was suggested that user scenarios and examples don't belong in a specification since the more non-normative scenarios/examples the more chance for conflict with a normative statement and eventual implementation bugs. Specifications should be precise and succinct and illustrations don't belong unless they are normative. Suggested lowering priorities to 3.

Discussed at Tokyo face-to-face, closed with resolution: disagree with originators proposal. Some rationale follows. Examples and user scenarios add to the understanding of a spec. Those reading the spec will not necessarily misinterpret. SpecGL shouldn't say that examples have to be embedded in the spec -- they can be provided outside the spec, but published concurrently and linked from Spec. Examples are understood to be informative. That can be emphasized under the ckpt related to make sure it is clear what is normative/informative (CK13.2).

Proposal: Originator suggests priority 3 (if the checkpoints 1.2 & 1.3 are kept).
Resolution: Keep checkpoints 1.2 and 1.3 at current priorities, investigate enhancing checkpoint 13.2.
85 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Should every test assertion be covered by a example?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Why is an example needed for each test assertion? Satisfying this requirement will make some specifications (e.g., protocol specifications) huge without added benefit. Not every test assertion needs an example, nor would it be helpful. How many examples are enough? Counter argument pointed out that one example could cover many test assertions.

Discussed at Tokyo face-to-face, closed with resolution: agree with originators proposal, remove checkpoint 1.4. Further discussion. An example covering every TA is neither needed nor desirable in SpecGL. What are positive benefits of an example for every TA? (Possible answer - clarifies the meaning.) But copious examples do not need to be in spec. Every feature should have an example, not every TA (need to define "feature"). Remove 1.4. Can SpecGL or Spec Extech provide informative guidance of how to determine how many examples? Remove "The more numerous the examples, the better." May want to 'save' some of the words in 1.4 for ExTech.

Proposal: Originator suggests eliminating the checkpoint (1.4).
Resolution: Remove checkpoint 1.4. Investigate possibility of some informative guidance (esp. in Spec Extech) for the "right number" of examples.
86 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Checkpoint 1.5 should be lowered to priority 3.
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Checkpoint 1.5: "Include an interpretation section (priority 2)". If somebody doesn't understand the formal descriptions, they should not rely on the specification to help them. A short informal introduction section is (@@ed error, remainder of statement lost@@).

Discussed at Tokyo face-to-face, closed with resolution: disagree with originators proposal. Additional issue discussion summary follows. (@@Ed note. Fix editorial truncation above.@@) Discussion wrestled with, "What is meant by interpretation section"? Resolution: re-title the checkpoint as, "Include examples for formal descriptions." Add a "not applicable" exemption for those specs that don't have formal descriptions (i.e.., if you don't have formal description, this ckpt does not apply). Leave as P2. Explain or enumerate what is a formal description - syntactic descriptions, eg. IDL, BNF, ... SpecGL lead editor (DHM) to work on improvement of statement of checkpoint.

Proposal: Originator proposed to lower ckpt 1.5 priority to 3.
Resolution: Re-title as "Include examples for formal descriptions", leave at P2, make other improvements as indicated.
87 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Should the SpecGL elaborate on DoV definition and discussion.
Description: [email], [origin]

Reference: 20020826 Spec Guidelines draft (WD)

Is the discussion on DoV excessive? Is there a benefit from talking about all the well-known forms of variability? It was proposed that the DoV discussion is not needed to define a good specification. Advising that less variability is good, may be enough. Prior WG discussions resulted in agreement to include a discussion on DoV, which includes more than just degrees of conformance, but other sorts of variations. The SpecGL should not mention DoV model in any normative way and should point to a DoV tutorial or white paper.

Numerous alternatives seem implied in the email thread. Some alternatives:

  1. eliminate DoV as organizing principle for SpecGL -- remove all discussion of DoV, eliminate most or all DoV-related guidelines and checkpoints;
  2. keep concept of DoV as organizing principle for SpecGL -- keep many/most of the DoV guidelines and checkpoints, but reduce discussion, and try to identify and eliminate some less useful checkpoints.
  3. leave as is, except possible tightening of prose;

Discussed at Tokyo face-to-face, closed with resolution: similar to #2 above, and

  • try to identify what needs to be moved into ExTech;
  • move DoV concept definition into a separate section (give it more visibility), or separate document for more detailed discussion/examples;
  • try to collapse and consolidate (some of the common checkpoints of) 3,4,7.

Proposal: Originator proposal is closest to alt. 1 above.
Resolution: Keep concept of DoV as organizing principle for SpecGL, simplify and clean up as indicated.
88 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: SpecGL objective and value.
Description: [email], [origin]

Reference: 20020826 Spec Guidelines draft (WD)

Should the SpecGL be a guide that applies to a given spec and not the process of writing specs. Is the SpecGL's purpose to help editors (and WGs) focus on key issues and make explicit choices? Is it an "instructions how to write a good spec"? Everything in the form of "the authors should consider or be aware of this or that" is an instruction. Suggested that 'useful and helpful instructions to spec editors should be outside the SpecGL scope -- apparently regardless of whether it is stated as process guidance or requirements on result (i.e., the specification).

In counter-argument, it is asserted that current SpecGL does not in fact address process (despite possible misleading implications of "imperative" voice in checkpoint statements). See additional comments from DM.

Discussed at Tokyo face-to-face, closed with resolution: it is our intention that SpecGL should be about the characteristics of good specifications and not how to write specs. SpecGL is about result of writing the spec, not the process.

Proposal: Originator suggests to remove material whose motivation is "useful and helpful guidance" to spec authors.
Resolution: SpecGL address the characteristics of good specifications, not how to write them.
89 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Should SpecGL conform to SpecGL?
Description: [email], [origin]

Reference: 20020826 Spec Guidelines draft (WD)

If the SpecGL is a spec, it should be about other specs (the subject is a spec), not about ways of writing other specs (the subject is the author of a spec or the writing process). The SpecGL should conform to itself.

Discussed at Tokyo face-to-face, closed with resolution:

  1. Rephrase issue from, "Is the SpecGL a spec?" to "Should SpecGL conform to SpecGL?"
  2. Answer: SpecGL must be self-conformant, A-conforming by Last Call.

Higher degree of self-conformance (AA or AAA) will be discussed and decided in the future.

Proposal: (Unclear, for this extracted issue.)
Resolution: SpecGL must be self-conformant, A-conforming by Last Call.
90 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Remove guideline on deprecation (GL6).
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

It is suggested that Guideline 6 is unnecessary, since it is already covered by other guidelines, specifically, the spec must define conformance. Disagreement with that assertion, in the email thread -- deprecated features have been inconsistently dealt with within the WGs. Was included as in the SpecGL due to a previous email thread.

Discussed at Tokyo face-to-face, closed with resolution: disagree with originator proposal. This was previously discussed via an email thread (@@reference?@@). GL6 was included as a result of consensus by the QAWG. No compelling new arguments to reverse that consensus.

Proposal: Originator proposes removing GL6.
Resolution: GL6 (about deprecation) will be kept.
91 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Why should discretionary items be handled consistently within an implementation (CK8.4)?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Originator: This shouldn't be required. Is browser X that is user-configured to do FOO the same implementation as browser X that is user-configured to do BAR? Can browser display cached documents differently from first-hand responses? Counter-argument. The idea is to lower the bad effect of discretionary items. Probably need to rewrite Checkpoint.

Additional clarifying email thread from the author indicates that the checkpoint as stated does not capture the intent, and that "items" in the checkpoint statement should be "choices" (one of three kinds of "items").

Discussed at Tokyo face-to-face, closed with resolution: replace "items" with "choices" in statement of checkpoint; and, implementations must do the same things in the same conditions. Further details of Tokyo discussion: Checkpoint addresses specifically discretionary choices - e.g., choice one of a,b,c - not discretionary items in general. Users should be able to count on getting the same thing under same conditions. If conditions have changed, then maybe you're not going to get same behavior - may not always be possible to have the same conditions - e.g., cache changes, network changes (recognize that requirement might be useless in some environments.) There is a possible inter-related issue which is not addressed by this resolution: consistent treatment of groups of discretionary choices. If "A or B" is a discretionary choice applicable to several different circumstances within the specification -- e.g., two possible responses to a collection of several different error conditions -- then should specifications treat these consistently as a group.

Proposal: Originator proposes that checkpoint 8.4 be removed.
Resolution: Keep CK8.4; replace "items" with "choices" in statement of checkpoint; and, specifications must state that given identical conditions, the effect of a discretionary choice is consistent within a single implementation.
92 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Remove guideline on Extensions (GL9).
Description: [email], [email2]

Reference: 20020826 Spec Guidelines draft (WD)

It was suggested that Guideline 9 be removed since if a spec mentions an extension, it MUST document compliant behavior just like any non-extension. Alternative suggested: a) define compliant behavior for every conformance object, b) and, define compliant behavior for every non-conforming object. May need to define what is meant by 'extension'. Extension is being addressed since it isn't always obvious to all spec writers what the consequences to interoperability are, nor some of the various ways to allow extensions.

Discussed at (2002-10-21 telecon. Resolution -- keep the guideline -- extensions are sufficiently harmful to interoperability that SpecGL should specifically address them. (There will be reworking of the wording of the GL and some of checkpoints.)

Proposal: Originator proposes that GL9 be removed.
Resolution: Keep GL9, with reworking of the wording of the GL and some of checkpoints.
93 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Why register extensions?
Description: [email], [email2]

Reference: 20020826 Spec Guidelines draft (WD)

Originator suggested that ckpt 9.5 is unnecessary and introduced too much overhead. Responders gave counter-examples: XSLT, CGM. Plus suggestions: Publication/registration of extensions important for some specs and less critical for others as long as there are mechanisms that can mitigate the extensions. The SpecGL should distinguish between published and registered. Currently SpecGL instructs authors how to make some extension work: "register your extension and you are covered". It should require that extension mechanisms maintain interoperability among parties that may not be aware of each extension definition. It should address the core of the problem and not require a specific solution.

Further discussed at 2002-10-21 telecon. Tentative resolution -- keep the checkpoint, but LR and MS to redraft the checkpoint for further discussion (possibly incorporating refinements discussed above and in telecon).

Discussed at Seattle face-to-face. The requirement to "register" is dropped, and a requirement to "publish" remains.

Proposal: Originator proposes that checkpoint 9.5 be removed.
Resolution: Drop "register", in favor of "publish".
94 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Why identify unused DoV (ckpt 10.5)?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

This is overkill. All that is necessary is to define what is in scope, everything else is out of scope. Why would you want authors to define non-scope and non-conformance. The conformance section is becoming too verbose.

Discussed at Tokyo face-to-face, closed with resolution: agree with proposal, remove checkpoint 10.5. Additional notes. There will also be impacts on checkpoints X.1 which have some verbiage about negative disclaimers and reference to CK10.5. Some additional discussion about a possible new checkpoint to require that all used DOV are documented, in one place in the conformance clause.

Proposal: Originator proposes that CK10.5 be removed.
Resolution: Remove checkpoint 10.5.
95 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Is it practical to "provide specific wording of conformance claims"?
Description: [email], [origin]

Reference: 20020826 Spec Guidelines draft (WD)

Ckpt 11.2 -- provide specific wording of conformance claim -- is questioned as to its practicality and verbosity. Counter-argument: What is required by this ckpt is routine and sensible for software components. Should this be required for content or other target objects?

Discussed at Tokyo face-to-face, closed with resolution: In checkpoint, list the minimum things that need to be included in a claim, and require that conformance claims contain this minimal list. Checkpoint can't provide and require all possible wording. Additional treatment and recommendations will be in ExTech. The other aspect/question of the issue: What about conformance icons? If you put a conformance icon on a page does that equal all the verbage of a conformance claim, either implicitly or explicitly (e.g., by link)? Resolution about icons: If there exists an icon to indicate conformance, then the explanation of what that icon represents (i.e., the conformance claim) must be available somewhere. The conformance claim would be written such that it would apply to the content (or whatever) that is using the icon. For example: The use of this icon X, indicates conformance of the content upon which this icon resides to Specification XYX, version 000, DDMMYYYY at conformance level P1." (Note. The later implies a new MUST or SHOULD assertion in the checkpoint.) Action item given to LR and DHM to rewrite checkpoint.

Proposal: None yet.
Resolution: In checkpoint, list the minimum things that need to be included in a claim, and require that conformance claims contain this minimal list; and, there must be an association (linkage) between any icon and corresponding conformance claim(s).
96 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Why require Implementation Conformance Statement?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Although nice to have, an ICS has nothing to do with the spec. Remove guideline (GL12). Counter-argument: the SpecGL are not just about the specification understood as a document but about the abstract specification as described by Al Gilman.

Discussed at Tokyo face-to-face, closed with resolution: disagree with originator proposal. Priority 3 (P3) equates roughly to "nice to have" (useful/beneficial), and the checkpoints of GL12 are P3. Some specs do have ICS. Examples: checklists of WAI and QA guidelines are ICS.

Further discussed at (2002-10-21 telecon, in email, and (2002-10-28 telecon (@@DRAFT@@), with the result that the two checkpoints (in GL12, 2002-10-25 SpecGL editors draft) will be refined somewhat: specs must have ICS *if applicable*; and specs must require ICS as part of conformance claim (either the spec's ICS, or another that contains at least as much information -- i.e., a superset of the spec's model ICS is permissible.)

Proposal: Originator proposes that GL12 be removed.
Resolution: Keep GL12 and keep its checkpoints at priority 3.
97 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Distinguish normative and informative text.
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

This (checkpoint 13.2) should be priority 1. A spec is no good if normative and informative statements are indistinguishable.

Discussed at Tokyo face-to-face, closed with resolution: agree with originator proposal. Change CK13.2 to priority 1.

Proposal: Raise checkpoint 13.2 to priority 1.
Resolution: Raise checkpoint 13.2 to priority 1.
98 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Should test assertions be included in a spec?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Test assertions should not be included since there should be a single normative definition and nothing else. Why not publish test assertions outside the spec? Counter-argument(s): By including test assertions in a spec, they are normative. Question(s): May need to clarify the rationale, as well as the meaning of "include". Can "include" encompass identification of test assertions in-line, by markup or other reference, so that they are not duplicated?

Discussed at Tokyo face-to-face, closed with resolution: it is is mandatory to provide test assertions (TA), and the provided TA are normative except in case of conflict with the text of the specification which shall take precedence. (TestGL must be rewritten to reflect the resolution). The resolution does not limit some inter-related details: are the test assertions internal to the specification (either separate list, or in-line markup); or are they a referenced external (normative) document. How the TA map to the specification -- i.e., the relationship between listed test assertions and bits of normative text in the specification -- was also discussed but is not addressed or limited by this resolution.

Proposal: Originator suggests that GL15 be removed or modified.
Resolution: It is is mandatory to provide test assertions (TA), and the provided TA are normative except in case of conflict with the text of the specification which shall take precedence.
99 SpecGuide 2002-09-24 SpecGuide Substantive Closed Alex Rousskov Lynne Rosenthal
Title: Scope of definition of test assertion.
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Does the definition preclude expressing a test assertion in a formal language without words and phrases rather than in sentences of a human language? It was suggested in the email thread to match the definition with the one used in the glossary. Counter-suggestion: see if the SpecGL definition can be fixed to cover the question/objection, because issue #19 (initially) resolved that in-document "Definitions" sections of the Framework documents could contain definitions that differ from but are compatible with the central QA Glossary.)

Discussed at Tokyo face-to-face (DRAFT!), tentatively closed with resolution: remove TA definition from SpecGL. Reopened per email request.

Discussion in 20021216 teleconference indicates that there is significant disagreement and/or misunderstanding about the definition of test assertion, or what comprises test assertions. Followup email enumerated, for example, the submitter's view of the test assertions that correspond to the SpecGL "to fulfill" criteria.

Discussed at Seattle face-to-face. Resolved that the definition should be in the central QA Glossary, not in the SpecGL Definitions section, and the definition in the QA Glossary should be the one that is in the verbiage of SpecGL Guideline 14.

Proposal: Several competing.
Resolution: As just described (Seattle resolution).
100 SpecGuide 2002-10-01 SpecGuide Editorial Closed Lynne Rosenthal Lynne Rosenthal
Title: "Non-normative" is undefined, and used in place of the defined "informative".
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

Within the SpecGL, Section 6 References has the additional qualifier, "(non-normative)". Although logic tells us that non-normative is the same as informative, is there a reason that we did not label this section "informative". The reason for this question, is that we don't define non-normative but define informative. Additionally, Checkpoint 13.2 states: Distinguish normative and informative text and Checkpoint 13.4 states: use consistent terminology.

Discussed at Tokyo face-to-face, closed with resolution: agree with originator proposal. Change label to "Informative".

Proposal: Replace non-normative with informative.
Resolution: Replace non-normative with informative.
101 All-Framework 2002-10-01 All-Framework Substantive Closed Lynne Rosenthal Lynne Rosenthal
Title: Should the GL documents use conformance levels instead of prioritized checkpoints?
Description: [email]

Reference: 20020826 Spec Guidelines draft (WD)

The WCAG WG, in their recently published Working Draft of Web Content Accessibility Guidelines 2.0 [WCAG20] has changed their conformance criteria - from using prioritized checkpoints to using Levels of satisfaction of checkpoints. (Minimum, Level 2, Level 3).

The QAWG model for its Guidelines documents was adapted from the original WAI models (WCAG, ATAG, and UAAG). As summarized in a recent email message the QAWG model is:

  • the statement of a checkpoint is just its "title" -- imperative voice but no normative implications;
  • one or more normative requirements are contained within the body of the checkpoint;
  • the individual normative requirements use RFC2119 keywords -- MUST, SHOULD, MAY, etc
  • the checkpoint as a whole is assigned priority 1 or 2 or 3, roughly meaning: essential/critical, important/desirable, useful/beneficial.
  • conformance degrees A, AA, AAA are defined respectively as all p1 ckpts, all p1+p2, all p1+p2+p3.

Inter-related set of issues: #101, #80, #57, #39, #64

Discussed at Tokyo face-to-face, without closure (too premature to decide). OT to investigate.

Discussed at Seattle face-to-face. The current QAWG model is considered to be a good match for QA Guidelines document requirements. Resolved that there doesn't seem to be an obvious reason to move to this new scheme.

(Additional reference: Requirements for WCAG 2.0 Checklists and Techniques

Proposal: (Investigate WCAG's rationale, including talk to WCAG.)
Resolution: No, it does not appear to be as suitable as the present model, for QA Framework application.
102 SpecGuide 2002-11-04 SpecGuide Substantive Closed Lofton Henderson Lynne Rosenthal
Title: Scope and content of the "conformance policy" guideline (GL3).
Description: [email], [email2], [email3]

Reference: Spec Guidelines 20021108 (WD)

This issue is comprised of a couple of sub-issues:

  1. should the issue scope be documented in the GL3 (formerly, this was GL6) verbiage as a catch-all and/or global guideline containing checkpoints that don't fit under another one of the DoV guidelines
  2. should we add a new CK3.5 as proposed in email

The issue was discussed at the 20021104 telecon, with no resolution other than to formulate it as an issue and deal with it after the 20021108 SpecGL publication.

Discussed at Seattle face-to-face. The issue was withdrawn (therefore closed for now) -- the currently queued changes to Guideline 3 are so extensive, that no one is sure whether the issue is still relevant or moot.

Proposal: Document GL scope as catch-all and/or global; add new CK3.5 similar to proposed.
Resolution: As just described (Seattle resolution).
103 SpecGuide 2002-11-04 SpecGuide Substantive Closed David Marston Lynne Rosenthal
Title: What are the fulfillment criteria for navigation requirements of CK13.4?
Description: [email]

Reference: Spec Guidelines 20021108 (WD)

Several previous checkpoints requiring TOC entries (to locate DoV and other conformance information) were combined into the single new CK13.4. As pointed out in email thread, the fulfillment criteria are not developed enough to prevent trivial satisfaction without meeting the intent of the checkpoint. It was suggested that there should be a MUST requirement, that the mechanism work in hardcopy versions of the specification.

The issue was discussed at the 20021104 telecon, without clear resolution other than to formulate it as an issue and deal with it after the 20021108 SpecGL publication.

Discussed and closed at the 20021216 telecon, the resolution being:

  • 10.1 removed.
  • 10.2 renamed 10.1 and made priority 1
  • 13.4 will be reworded with some new "ease-of-navigation" language from David. Left as priority 2 for now.

It was not resolved, how to handle the suggested easy-in-hardcopy requirement. A new issue, issue 108, was spun off to deal with that aspect.

Proposal: Add MUST requirement, that navigation mechanism work in hardcopy.
Resolution: As detailed above, with (#108) for the hardcopy question.
104 SpecGuide 2002-11-04 SpecGuide Substantive Closed Dominique Hazaël-Massieux Lynne Rosenthal
Title: Should CK7.4 (deprecation examples) be rewritten to cover both producers and consumers?
Description: [email]

Reference: Spec Guidelines 20021108 (WD)

The issue was posed during review of draft text for the 20021108 SpecGL publication, and was discussed at the 20021104 telecon.

The question was raised: does this checkpoint apply just to producers and not consumers (the wording implies "producers")? Some think it should also apply to consumers. If that is accepted, then there is a choice of adding a parallel checkpoint for consumers, or rewording CK7.4 so that it applies to both producers and consumers.

Discussed at Seattle face-to-face. Resolved that it applies only to producers.

Proposal: Reword CK7.4 so that it applies to both producers and consumers.
Resolution: No, producers only.
105 TestGuide 2002-12-11 TestGuide Substantive Closed Mark Skall Lynne Rosenthal
Title: Should the "fail" part of the conformance disclaimer be eliminated?
Description: [email]

Reference:

Note. Although indicated for Test Guidelines, the issue pertains to the other guidelines documents as well.

The second part of the conformance disclaimer states "Failing to achieve conformance degree of at least A-conforming does not mean that the subject specification is necessarily deficient to its intended purposes, nor does it mean that it is an unacceptable basis for the development of quality test materials."

Originator: "This undermines our attempt to get WGs, recs and test suites to adhere to at least Level A. In fact, it may be in direct contradiction to this goal."

Options:

  1. leave disclaimer as is;
  2. reword part 2 of the disclaimer;
  3. eliminate this part of the disclaimer

Discussed at Seattle face-to-face. Resolved to remove the fail part of the Conformance Disclaimer.

Proposal: Originator recommends option #3.
Resolution: Option #3, remove the fail part of the disclaimer.
106 SpecGuide 2002-12-11 SpecGuide Substantive Closed Lofton Henderson Lynne Rosenthal
Title: Should the "extensions" checkpoints 9.1 and 9.2 be repackaged?
Description: [email]

Reference: Spec Guidelines 20021108 (WD)

CP9.1 says, "Indicate if extensions are disallowed", and requires definition of the conditions under which they are disallowed. CP9.2 says, "Indicate if extensions are allowed," and enumerates details that must be supplied, including the conditions under which they are allowed. There are a couple problems: the packaging of the two is awkward; and, there is little distinction between CP9.1 and CP3.2 ("Identify strict conformance requirements"). A equivalent and cleaner packaging would be:

  • CP9.1: Indicate the conditions under which extensions are allowed and disallowed. (Subsumes "if allowed", "not allowed", and "under what conditions" parts of 9.1, 9.2).
  • CP9.2: If extensions are allowed, completely define their scope and constraints. (Rest of requirements of 9.2).

Another equivalent rewording of 9.1 would be, "Indicate the conditions under which conformance requirements are strict and not strict". This wording emphasizes that these checkpoints have a lot of redundancy with 3.2.

Discussed and closed at the 20021216 telecon, with resolution as proposed.

Proposal: Originator recommends repackaging 9.1, 9.2 as indicated. (No proposal about 3.2).
Resolution: As proposed.
107 TestGuide 2002-12-11 TestGuide Substantive Closed Lynne Rosenthal Kirill Gavrylyuk
Title: Differentiate between test materials for CR and those for Recs.
Description: [email]

Reference: Test Guidelines 20021205 (WD)

There is a topic that is not addressed in TestGL, and ought to be: in reality, test suites developed at the CR stage may be different than a test suite at the Rec stage. The objectives for these test suites are often very different - for example: for CR the objective may be to 'test the specification', whereas a test suite for a Rec targets interoperability of the implementations. Since Operational Guidelines advocates having tests for CR exit, this aspect must be considered. There should be discussion in the Introduction, and/or treatment in the checkpoints, to capture the idea that test suites for CR may be different, and all of the CPs in TestGL may not be applicable (e.g., CPs about maintenance of tests).

Discussed at Boston face-to-face. Resolved that TestGL needs a new checkpoint (LR to draft), that test materials clearly document their scope and intended purpose.

Proposal:
Resolution: Add a new checkpoint to TestGL, that test materials clearly document their scope and intended purpose.
108 SpecGuide 2002-12-26 SpecGuide Substantive Closed David Marston Lynne Rosenthal
Title: Should the fulfillment criteria for CP13.4's navigation requirements specifically include usability in hardcopy?
Description: [email]

Reference: Spec Guidelines 20021220 (WG discussion draft)

In closing issue 103, several changes were made to SpecGL to improve the easy findability of critical conformance information by the required navigation mechanisms. However, one aspect of the issue was not addressed: it is suggested that there ought to be a MUST requirement -- either a new MUST requirement in CP13.4 or a new checkpoint -- that the mechanism work in hardcopy versions of the specification. Others believe that this is a detail that should not be in SpecGL's fulfillment criteria (or test assertions), but rather is a detail that should be dealt with in Spec Examples & Techniques (SpecET).

Discussed at Seattle face-to-face. Resolved that it will not be in the requirements to satisfy the checkpoint, but will be in the discussion for the checkpoint.

Proposal: Add a MUST requirement, that navigation mechanisms work in hardcopy.
Resolution: No, not a requirement, but document it in the checkpoint's discussion as a useful and recommended attribute of the navigation mechanism.
109 SpecGuide 2002-12-26 SpecGuide Substantive Closed Lofton Henderson Lynne Rosenthal
Title: How do the "extensions" checkpoints (9.1 & 9.2) relate to strict conformance checkpoint (3.2)?
Description:

Reference: Spec Guidelines 20021220 (WG discussion draft)

Issue raised during 20021216 telecon.

CP9.1 and CP9.2 are phrased in terms of extensions, but given the definition of strict conformance in GL3, an equivalent wording of CP9.1 would be: "Indicate the conditions under which conformance requirements are strict and not strict". Worded that way, it is almost completely redundant with CP3.2. There are two sub-issues:

  1. is the definition of strict conformance in GL3 the correct one -- should it be limited (as now) to extensions, or should discretionary items be included in it as well?
  2. what is the relationship between CP3.2 and CP9.1/9.2?

Discussed at Seattle face-to-face. Multi-faceted resolution: remove checkpoint 3.2 and transfer it's strict-by-class concept to guideline 9 (or checkpoint 9.1?), migrate the definition of "strict conformance" to guideline 9, as well as some of the other verbiage of guideline 3 (2nd pgph).

Related issue: issue 106.

Proposal:
Resolution: As just described (Seattle resolution).
110 SpecGuide 2002-12-26 SpecGuide Substantive Closed Dominique Hazaël-Massieux Lynne Rosenthal
Title: It is unclear how to apply CP3.3, "distinguish requirements from product-specific extra features."
Description: [email]

Reference: Spec Guidelines 20021220 (WG discussion draft)

Issue raised during 20021216 telecon.

The fulfillment criteria state, "To full this checkpoint, a specification MUST state in its conformance section all facets of the requirements where the required features represent the maximum allowable capabilities."

While applying the specGL to the opsGL, originator couldn't determine what the checkpoint and fulfillment criteria means. QAWG agreed on the call that this CP definitely needs some clarification.

Discussed at Seattle face-to-face. Resolved to remove checkpoint 3.3.

Proposal:
Resolution: Remove checkpoint 3.3.
111 OpsGuide 2003-01-03 OpsGuide Substantive Closed Lofton Henderson
Title: Should OpsGL require that WGs have a general process document?
Description:

Reference: Ops Guidelines 20021220 (WG discussion draft)

At the reference above, checkpoint 4.3 (CP4.3) requires, "Produce the QA Process Document" (QAPD). The QAPD addresses multiple facets of the quality practices of the WG. In QAWG teleconference, the issue was raised: should OpsGL require, in addition to producing a QAPD, that the WGs also produce a general process document that addresses the non-QA facets of WG life and process.

  1. yes, add a new checkpoint;
  2. no, it is not QAWG's business;
  3. as #2, but add an informative note in descriptive verbiage of CP4.3, that a generic WG Process Document is useful and valuable, though outside of the scope of OpsGL.

Discussed at Seattle face- to-face. Option #2. It is not QAWG's business, and even giving (informative) advice on the topic should be considered out-of-scope for QA.

Proposal: Option #3.
Resolution: Option #2.
112 OpsGuide 2003-01-12 OpsGuide Substantive Closed \1 Lofton Henderson
Title: There are structural and testability problems in the QA-commitment table in OpsGL.
Description: [email]

Reference: Ops Guidelines 20021220 (WG discussion draft)

The problems are three-fold:

  1. that the columns (C) of the table were inextricably combined in satisfying a given level;
  2. that the levels (rows, R) were nested, and someone could in principle satisfy e.g. R3C1 without satisfying R1C1 (ignoring the "in addition to..." clause, of course);
  3. that the conformance requirements were fuzzy.

Discussed and resolved, per proposal, at 20030113 teleconference.

Proposal: Structural issues per email proposal, and testability issues per table re-write.
Resolution: As proposed.
113 SpecGuide 2003-01-02 SpecGuide Substantive Closed Alex Rousskov Lofton Henderson
Title: Collected editorial and substantive issues on SpecGL.
Description:

Reference: Spec Guidelines 20021220 (WG discussion draft)

Originator has submitted collected comments containing 28 editorial and substantive issues.

Discussed and resolved at Seattle face-to-face, 20030113 telecon, and 20030122 telecon.

The resolutions are summarized in a collected disposition of comments.

Proposal:
Resolution: See disposition of comments.

Table Legend

num
Issue number
Title
Short title/name of the issue
Spec
Document referred to in issue (Intro = Framework: Introduction; ProcOps = Framework: Process&Operational Guidelines; SpecGuide = Framework: Specification Guidelines)
Description
Short description of issue, possibly including link to origin of issue
Date
The date at which the issue was raised or initially logged.
Topic
Rough topic categorization, one of: ...tbd... [replace following XMLP stuff... env(elope), rpc, enc(oding), meta(issue), bind(ing), fault]
Class
Substantive or Editorial
Status
One of: Unassigned, Active, Closed, Postponed
Proposal
Current proposal for resolution of issue, possibly including link to further text
Resolution
Short description of resolution, possibly including link to a more elaborate description
Raised by
Person who raised the issue
Owner
QA WG Member responsible for the issue

Maintained by Lofton Henderson.