QA Framework (QAF) Candidate Recommendation Issues

Last update: $Date: 2005/06/22 21:25:14 $

Issues regarding the QA Framework (QAF) Candidate Recommendation (CR) documents and associated Notes produced by the QA Working Group (QAWG) should be reported to the QAWG per the instructions in the "Status..." sections of those documents: send mail to www-qa@w3.org (public archives).

Comments on this issues list should be sent to the www-qa@w3.org mailing list (Archives).

In this document,

Summary List of Outstanding Candidate Recommendation Issues

num Status Spec Topic Class Date Title
CR-32 Resolved QAF QAF Substantive 2004-02-11 QAF contravenes W3M resolution that Rec-track specs should not coerce the WGs.
CR-29 Reopened SpecGL SpecGL Substantive 2004-02-11 Detectable errors for instance data CoP, and error reporting requirements for associated processor (software) CoP.
CR-33 Active QAF QAF Substantive 2004-02-11 QAWG failed to formally address Last Call comment before requesting CR transition.
CR-34 Active QAF QAF Substantive 2004-02-11 QAWG failed to list open issues when advancing to CR.
CR-35 Active QAF QAF Substantive 2004-02-11 QAF is only a "test development framework", therefore it is misnamed.
CR-36 Active QAF QAF Substantive 2004-02-11 QAWG has failed to meet the quality commitments in its charter.
CR-37 Active QAF QAF Substantive 2004-02-11 Make QAF informative, and represent its key normative content with small changes to W3C Process Document.
CR-38 Active QAF QAF Substantive 2004-02-11 QAF violates RFC2119.
CR-39 Active QAF QAF Substantive 2004-02-11 All parts of QAF, including supporting informative & resource references, must advance synchronously.
CR-40 Active QAF QAF Substantive 2004-02-11 2nd Last Call needed, with stricter advancement criteria.
CR-41 Active QAF QAF Substantive 2004-02-11 QAWG & QAF have many breaches of W3C Process, pubrules, etc.
CR-42 Active SpecGL SpecGL Substantive 2004-02-15 Test Assertions -- definitions
CR-43 Active SpecGL SpecGL Substantive 2004-02-15 Test Assertions -- how and where?
CR-44 Active SpecGL SpecGL Substantive 2004-02-15 Test Assertions -- should SpecGL require them?
CR-45 Active SpecGL SpecGL Substantive 2004-02-15 SpecGL & SpecET inconsistencies about CoP and scope requirements.
CR-46 Active SpecGL SpecGL Substantive 2004-02-15 CP2.2 is hard to implement.
CR-47 Active SpecGL SpecGL Substantive 2004-02-15 Rationale of CP2.3 (spec. category) is unconvincing, and it is unclear how to satisfy the CP.
CR-48 Active SpecGL SpecGL Substantive 2004-02-15 Error in Conformance Requirements of CP3.2 (profile min. rqts.)?
CR-49 Active SpecGL SpecGL Substantive 2004-02-15 The statement of CP5.1 (discretionary items) is unrelated to the given Conformance Requirements.
CR-50 Active SpecGL SpecGL Substantive 2004-02-15 Redundant requirement in CP5.3 (number of disc. items/choices).
CR-51 Active SpecGL SpecGL Substantive 2004-02-15 Inconsistencies in CP5.4 (consistent handling of discretionary choices).
CR-52 Active SpecGL SpecGL Substantive 2004-02-15 Problems in CP5.5 (relationship of discretionary items with other DoV).
CR-53 Active SpecGL SpecGL Substantive 2004-02-15 Error in Conformance Requirements of CP6.5 (mitigate extensions)?
CR-54 Active SpecGL SpecGL Substantive 2004-02-15 CP8.1 (universal reqts for min functionality) is difficult to implement.
CR-55 Active SpecGL SpecGL Substantive 2004-02-15 Define "conformance concepts" in CP8.2.
CR-56 Active SpecGL SpecGL Substantive 2004-02-15 How do "conformance designations" (CP8.5) relate to "conformance concepts" (CP8.2)?
CR-57 Active SpecGL SpecGL Substantive 2004-02-15 The meaning and purpose of "conformance disclaimer" are unclear in CP9.2.
CR-58 Active SpecGL SpecGL Substantive 2004-02-15 Clarify what comprises "list" of Test Assertions.
CR-59 Active SpecGL SpecGL Substantive 2004-02-15 ...~20 substantive SpecGL issues from JC will start here...

Detailed List of Candidate Recommendation Issues

num Spec Date Topic Class Status Raised By Owner
CR-1 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: QAF is too big and too expensive.
Description:

Comment reference.

Three interrelated sub-issues can be identified here:
  1. Size & complexity: large number of interrelated documents in QAF is too confusing.
  2. Inconsistencies & incompleteness: amongst the Guidelines documents family and associated documents like QA Glossary.
  3. Too burdensome on WGs: requires detailed knowledge of whole QAF at WG-Charter time, and requires commitments which may be premature (and regretted later). (See also CR-20.)

Related CR issues: 20.

Discussed at 20040223 telecon. Discussed and resolved at TP2004 QA face-to-face. Resolved: redesign the QA Framework for simplicity, clarity, usefulness, and usability. There will be fewer documents, they will be smaller and simpler, less authoritarian, and well syncrhonized with each other. For details, see: f2f minutes, TP2004 presentation, QAIG announcement, and W3C Chairs announcement.

Proposal: [Originator] No specific proposals are associated directly to these high-level issues by originator.

[LH] The proposed revisions to the QAF would mitigate some or all of these concerns.

Resolution: Comments accepted. The QA Framework are being redesigned for simplicity, clarity, usefulness, and usability. There will be fewer documents, they will be smaller and simpler, less authoritarian, and well syncrhonized with each other. For details, see: f2f minutes, TP2004 presentation, QAIG announcement, and W3C Chairs announcement. For ongoing implementation of resolution, see QAF status on QAWG home page.
CR-2 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: Comprehensive conformance test materials versus usable and useful test materials.
Description:

Comment reference.

  • QAWG Charter: "main purpose [...] foster development of usable and useful test suites.
  • OpsGL Scope: "primary goal is to help [...] with the planning, development, deployment, and maintenance of conformance test materials [TM]."

WebOnt opted for the first, to good effect; and, believes that the thoroughness and extensive procedures implied by following QAF for the second would cost a lot, without commensurate return of value.

The scope of this issue might appear to pertain narrowly to TestGL. However, the specificity of "conformance test materials" appears in Intro and in OpsGL requirements as well (although that specificity is inconsistent -- there are some references just to "test materials".) Therefore the issue needs to be resolved consistently throughout synchronized QAF components.

Discussed at 20040223 telecon. Along with the comments of Issue CR-1, discussed and resolved at TP2004 QA face-to-face. Resolved: redesign the QA Framework for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, TestGL will be clear about its initial scope (conformance TM), and will be explicit that there are other kinds of useful test materials (TM) than its initial scope. The remaining QAF components will be synchronized consistently with TestGL.

Proposal: [Originator] (Implicit) QAF should recognize WebOnt "usable and useful" approach as satisfying TM needs of the WG.

[LH] While the proposed revisions to the QAF would not directly address these concerns, nevertheless this is an issue that would have to be worked out during development of the details of the proposal (if it were accepted).

Resolution: Agreed. The QA Framework is being redesigned for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, TestGL will be clear about its initial scope (Conformance test materials), and will be explicit that there are other kinds of useful test materials than its initial scope. The remaining QAF components will be synchronized consistently with TestGL.
CR-3 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Dominique Hazaël-Massieux
Title: Rec-track documents should not constrain the WGs or their specifications.
Description:

Comment reference.

It is inappropriate for QAWG, in the QAF documents, to mandate conformance of WGs to procedures (OpsGL) and conformance of WG specifications (SpecGL).

This is effectively the same issue as CR-32, which contains somewhat more detail.

Related CR issues: 3, 32

Proposal: [Originator] Don't mandate conformance of WG procedures and specifications within QAF.

[LH] Originator misunderstands what QAF actually says. QAF gives requirements that must be met for the conformance target -- a WG (OpsGL) or a specification (SpecGL) -- to conform to the stated guidelines. But QAF does not say that the objects must conform, which would effectively be defining W3C policy. This should be clarified and emphasized more in QAF.

Resolution: The comment misinterprets what QAF actually addresses. XyzGL (Xyz = Ops or Spec or Test) defines what an object (WG, spec, TM) must do to conform to XyzGL. It does not require that the object conform. This will be even clearer in the redesigned QAF. While QA believes that old-QAF did not violate W3C Process or policies, nevertheless please see Resolution of CR-1 -- in the new QAF, a more helpful, less authoritarian approach will de-emphasize strict testability of its presented advice, principles, good practices, etc.
CR-4 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: Conformance with QAF is not the only way to meet reasonable quality goals.
Description:

Comment reference.

Specifically, WebOnt believes that it accomplished its stated quality goals without conforming to the procedural requirements of OpsGL, and believes that its success is due to its freedom to choose the most appropriate approaches.

Discussed at 20040223 telecon. Along with the comments of Issue CR-1, discussed and resolved at TP2004 QA face-to-face. Resolved: redesign the QA Framework for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, OpsGL is being merged together with QAF-Introduction into a non-normative "QA Handbook". It presents collected experience and potentially useful advice for Chairs and staff contacts, while acknowledging that WGs may have good reason to do things differently.

Proposal: [Originator] No specific associated proposal.

[LH] The proposed revisions to the QAF would address this concern.

Resolution: Agreed. The QA Framework is being redesigned for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, OpsGL is being merged together with QAF-Introduction into a non-normative "QA Handbook" (QAH). QAH collects experience and potentially useful advice for Chairs and staff contacts, without presenting the content as normative procedural requirements.
CR-5 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: QAF should be a flexible, user-friendly set of resources, referenced but not mandated by W3C Process.
Description:

Comment reference.

Instead of a monolithic set of terse, conformance-driven documents, WebOnt would like to see QAF become a collection of useful and helpful tools, templates and guidelines. This issue concerns the general philosopy or nature of QAF. Specific details maybe found in related issues CR-6 through CR-14.

This CR issue group: 5 (parent), 6 through 14 (details).

Discussed at 20040223 telecon. Along with the comments of Issue CR-1, discussed and resolved at TP2004 QA face-to-face. Resolved: redesign the QA Framework for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, the old OpsGL+Intro merge and simplify to become a non-normative "QA Handbook". The Spec and Test components will be lighter-weight, with lots of examples, tools, templates, etc. The relationship of the new-QAF resources to W3C Process is unspecified (as it has always been), but will be explored when as the new-QAF becomes more mature.

Proposal: [Originator] Specific associated proposal are found in issues CR-6 through CR-14.

[LH] The proposed revisions to the QAF would address this concern.

Resolution: Agreed. The QA Framework is being redesigned for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, the old OpsGL+Intro merge and simplify to become a non-normative "QA Handbook". The Spec and Test components will be lighter-weight, with lots of examples, tools, templates, etc. The relationship of the new-QAF resources to W3C Process is unspecified (as it has always been), but will be explored when as the new-QAF becomes more mature.
CR-6 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: QAF-Intro needs more prominence as QAF starting point, and should maybe absorb OpsGL.
Description:

Comment reference.

Emphasize QAF-Intro as starting point into QAF. Consider removing OpsGL as separate component of QAF, and using Intro "Primer" to serve same role. (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Discussed at 20040223 telecon. Along with the comments of Issue CR-1, discussed and resolved at TP2004 QA face-to-face. Resolved: redesign the QA Framework for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, the old OpsGL+Intro merge and simplify to become a non-normative "QA Handbook".

Proposal: [Originator] As proposed in comment.

[LH] The proposed revisions to the QAF would address this concern.

Resolution: The QA Framework is being redesigned for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, the old OpsGL+Intro merge and simplify to become a non-normative "QA Handbook" (QAH), whose target audience is WG Chairs and staff contacts. Those topics of old-OpsGL which experience has shown to be most useful will survive as non-normative "Principles" and "Good Practice" in the new QAH. The disposition of the Primer parts of the old QAF-Intro is still under consideration.
CR-7 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: The use of MUST in conformance requirements is too strong.
Description:

Comment reference.

MUST ought to be weakened to SHOULD (at least), or MAY. Flexibility is needed by the WGs, especially in situations where timeliness is critical. (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Discussed at 20040223 telecon. Along with the comments of Issue CR-1, discussed and resolved at TP2004 QA face-to-face. Resolved: redesign the QA Framework for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, the language will be made less terse and authoritarian (a "good faith" conformance model). In those new-QAF components where RFC2119 language is retained, the choice of the various conformance keywords will be considered case-by-case.

Proposal: [Originator] As proposed in comment.
Resolution: Accepted in general. The QA Framework is being redesigned for simplicity, clarity, usefulness, and usability (see Resolution of CR-1). As part of the resolution, the language will be made less terse and authoritarian (a "good faith" conformance model). In those new-QAF components where RFC2119 language is retained, the choice of the various conformance keywords will be considered case-by-case, in the context of the overall conformance model.
CR-8 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: Put all QA-related terminology into QA Glossary.
Description:

Comment reference.

All QA-specific terms such as Commitment Levels, Priority Levels, etc should be in the QAF Glossary. (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Discussed at 20040223 telecon. As part of the discussion and resolution of the "QAF Future" topic at TP2004 QA face-to-face, a complete "QA Glossary" should result from the improved synchronization of the light-weight new-QAF components. (See also Resolution of CR-1).

Proposal: [Originator] As proposed in comment.
Resolution: Agreed. A comprehensive "QA Glossary" should be one of the synchronized new-QAF components. (See also Resolution of CR-1).
CR-9 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: All common boiler-plate sections in the QAF specs should be moved into Intro.
Description:

Comment reference.

Consider moving boiler plate sections to Intro. In any case move term definitions to a place prior to their use in base documents. (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Discussed at 20040223 telecon. As part of the discussion and resolution of the "QAF Future" topic at TP2004 QA face-to-face, streamlining of the QAF components is a goal. The new-QAF components should be shorter, more concise, and more focused. Opportunities to centralize or even eliminate boiler-plate parts will be looked at -- best approach will be determined during rewriting of the components. For example, Intro itself merges into a short and focused QA Handbook, so reduction or elimination might be the best option. (See also Resolution of CR-1).

Proposal: [Originator] As proposed in comment.
Resolution: Agreed in principle, although best specific approaches will be determined during rewriting of the new-QAF components. Streamlining of the QAF components is a goal. The new-QAF components should be shorter, more concise, and more focused. Opportunities to centralize or even eliminate boiler-plate parts will be looked at. Given that Intro itself merges into a short and focused QA Handbook, reduction or elimination of boiler plate might prove to be the best option. (See also Resolution of CR-1).
CR-10 QAF 2004-02-09 QAF Editorial Closed Jim Hendler Lofton Henderson
Title: Use consistent document abbreviations throughout the framework.
Description:

Comment reference.

Use consistent document abbreviations throughout the framework. [Ed note. Ask Originator for examples where this is not done.] (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Proposal: [Originator] As proposed in comment.
Resolution: Agreed. This has always been a principle, so exceptions to it have been mistakes due to lack of synchronization and coordination.
CR-11 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: Add conformance requirements to checklists.
Description:

Comment reference.

Add conformance requirements to checklists. (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

LH comment: this would almost turn the checklists into test materials, instead of (currently) results summaries.

See CR-16 -- this issue essentially duplicates that one.

See discussions of 20040218 telecon. For the old "QA Framework" (QAF), the QAWG had resolved to generate test materials. These would have satisfied Commenters need (while separately preserving Checklists for other purposes). The issue probably becomes moot with the redesign of the QAF -- see Resolution of CR-1 -- because the strict conformance model of QAF will be replaced with something more user friendly.

Proposal: [Originator] As proposed in comment.
Resolution: Adding the conformance requirements to checklists would essentially turn them into Test Materials for the associated guidelines documents (see 20040218 telecon). For the old "QA Framework" (QAF), the QAWG had resolved to generate such test materials. QAWG believes these would have satisfied Commenter's need (while separately preserving Checklists for other purposes). The issue probably becomes moot with the redesign of the QAF -- see Resolution of CR-1 -- because the strict conformance model of QAF will be replaced.
CR-12 QAF 2004-02-09 QAF Substantive Closed Jim Hendler Lofton Henderson
Title: Confirm consistency of overlapping information in QAF document families.
Description:

Comment reference.

While this should obviously be done for *all* QAF components, the specific instance pointed out by Originator is: Priority of checkpoints between OpsGL and OpsET (note problems with this in Guideline 6). (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Proposal: [Originator] As proposed in comment.
Resolution: Comment accepted. The QA Framework is being redesigned for simplicity, clarity, usefulness, and usability. There will be fewer documents, they will be smaller and simpler, and syncrhonization with each other is being prioritized. See also Resolution of CR-1.
CR-13 OpsGuide 2004-02-09 OpsGuide Editorial Closed Jim Hendler Lofton Henderson
Title: The normative version of OpsGL should be a single HTML file.
Description:

Comment reference.

Make the single HTML file version of QAF-OPS normative (if QAF-OPS stays an independent document). (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

See CR-18 -- this issue essentially duplicates that one.

Proposal: [Originator] As proposed in comment.
Resolution: Accepted. Although the issue probably becomes mostly moot with the QA Framework (QAF) redesign (see Resolution of CR-1). Smaller, simpler QAF components will be published as single HTML files.
CR-14 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: Operational Examples & Techniques contains no examples, only case studies.
Description:

Comment reference.

Either change the Operational Examples and Techniques to contain reusable examples or change its name to reflect its true content. Suggest "Operational Case Studies and Techniques." (This issue is a proposed implementation detail of CR-5.)

This CR issue group: 5 (parent), 6 through 14 (details).

Proposal: [Originator] As proposed in comment.
Resolution: The issue becomes moot with the QA Framework (QAF) redesign (see Resolution of CR-1). "Operational Examples & Techniques" (OpsET) will not be a component of the new QAF. Useful examples will be included directly in the "QA Handbook", and the associated templates (especially appropriately modified version of the QA Process Document template) will continue as a tightly referenced external appendix to QAH.
CR-15 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: Clarify what is meant by "New Working Groups" in Guideline 1.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] These guidelines should only apply to new work items begun after the QAF becomes a recommendation.
Resolution: Agreed. If the distinction survives in the new "QA Handbook" (QAH), it will be clarified. However the issue probably is moot because of the nature of the new QAH -- there won't be strict conformance requirements about old/new WGs, charters, commitments, etc. (See also Resolution of CR-1).
CR-16 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: Checklists should include conformance requirements.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

See CR-11 -- this issue essentially duplicates that one.

Proposal: [Originator] Include conformance requirements in checklists.
Resolution: Probably moot. See Resolution of CR-11.
CR-17 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: A, AA, AAA should be defined before their first use.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] Include explanations of A, AA, AAA before their first use.
Resolution: Moot. The issue goes away because old-QAF's strict conformance model and the three levels (A/AA/AAA) won't be used in the new QAF. The new QAF will address and discuss any conformance concepts before they are applied. (See also Resolution of CR-1).
CR-18 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: The single HTML file should be the normative version of OpsGL.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

See CR-13 -- this issue essentially duplicates that one.

Proposal: [Originator] Make the single HTML file the normative version of OpsGL.
Resolution: Accepted. See Resolution of CR-13.
CR-19 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: CP1.4 does not define "QA Deliverables & Milestones".
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal:
Resolution: Accepted (but partially moot). In the drafting of the "QA Handook" (QAH) of the redesigned QAF, attention is being given to more detailed and useful definitions of terms like "QA deliverables". The issue otherwise goes away because old-QAF's strict conformance model is abandoned, and "Operational Examples & Techniques" is not a component of the new QAF. (See also Resolution of CR-1).
CR-20 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: OpsGL should not require so much planning and assignment of resources prior to chartering a WG.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Related issues: CR-1 (3rd part).

Proposal: ...tbd...
Resolution: Agreed. The "QA Handook" (QAH) of the redesigned QAF will contain advice about early planning and commitment, from the general perspective of "earlier is better", but will also acknowledge that detailed commitments at charter time may not be appropriate or desirable. In addition, QAH replaces normative requirements with good-practice advice, examples, and how-to aids. (See also Resolution of CR-1).
CR-21 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: The Rationale of CP2.1 is inconsistent with the statement of the CP and its Conformance Requirements.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal:
Resolution: Moot. The "QA Handbook" (QAH) component of the redesigned QAF has completely reworked this material, and the comment no longer applies to anything in QAH. (See also Resolution of CR-1).
CR-22 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: The phrase "where-and-how-plan" is used in ConfReqs of CP2.2 but not defined anywhere.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal:
Resolution: Moot. The "QA Handbook" (QAH) component of the redesigned QAF has completely reworked this material, and the comment no longer applies to anything in QAH. (See also Resolution of CR-1).
CR-23 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: The scope of QA Moderator position is ill-defined and not distinguisable from "test development manager".
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] As proposed in comment.
Resolution: Agreed. To the extent that old-OpsGL concept of "QA Moderator" survives in the advice of the "QA Handbook" (QAH) of the redesigned QAF, we'll attempt to clarify what are the essential role(s) that ought to be addressed, and how they can be addressed. Whether or not they are broader than "test development manager" will be determined during development of QAH (and clarified). (See also Resolution of CR-1).
CR-24 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: CP5.1 is hard to distinguish from CP2.1 & CP2.2, and contains inconsistencies.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] As proposed in comment.
Resolution: Agreed. To the extent that these old-OpsGL concepts have survived in the advice of the "QA Handbook" (QAH) of the redesigned QAF, they have been consolidated and condensed. Also, much test development "process" material is being migrated to a non-normative process-advice chapter of new-TestGL (Test Guidelines), will reference and linking from QAH. (See also Resolution of CR-1).
CR-25 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: QAF-OPS and OPS-EXTECH docs out of synch.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] As proposed in comment.
Resolution: Moot. The issue goes away because "Operational Examples & Techniques" (OpsET) is not a component of the new QAF. The useful parts of OpsET and OpsGL have been simplified, consolidated, and condensed into the new "QA Handbook" (QAH). (See also Resolution of CR-1).
CR-26 OpsGuide 2004-02-09 OpsGuide Substantive Closed Jim Hendler Lofton Henderson
Title: CP6.2 and CP5.4 should be merged.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] As proposed in comment.
Resolution: Agreed. In the new "QA Handbook" (QAH) component of the redesigned QAF, licensing and branding issues have survived as topics worthy of advice and attention. They have been rolled together under one "Principle", as three short "Good Practice" advisories. (See also Resolution of CR-1).
CR-27 SpecGL 2004-02-11 SpecGL Editorial Closed Bjoern Hoehrmann Dominique Hazaël-Massieux
Title: Use of abbreviations in ConfReqs makes them much harder to read.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] Spell out all abbreviations in ConfReqs.
Resolution: Agreed in principle. In practice, it is probably mostly moot, due to the large change of formatting for "Specification Guidelines" (SpecGL). If it is applicable to any of the formatting constructs in the new lightweight SpecGL (and other QA Framework components), we will follow it.

In the just-published "SpecLite" (lightweight SpecGL), you will see that this has been followed in the Principles and the Good Practices, which most closely resemble the previous Conformance Requirements. It was not uniformly followed in the just-published QA Handbook, but will be applied to the next version.

CR-28 SpecGL 2004-02-11 SpecGL Editorial Closed Bjoern Hoehrmann Dominique Hazaël-Massieux
Title: The styling of checkpoints makes it difficult to recognize checkpoint headings.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Proposal: [Originator] Highlight checkpoint headings (see comment reference).
Resolution: The issue is mostly moot now, due to the large change of formatting that we applying for the new, lightweight Specification Guidelines (SpecGL). We are paying attention to readability and ease of finding key information in the new revision. See for example the just-published "SpecLite" (lightweight SpecGL).
CR-29 SpecGL 2004-02-11 SpecGL Substantive Reopened Bjoern Hoehrmann Dominique Hazaël-Massieux
Title: Detectable errors for instance data CoP, and error reporting requirements for associated processor (software) CoP.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now.)

Discussed at 20040503 telecon (see topic #5). In the SpecGL context, the question of interest may be: what are the different error handling mechanisms used across specifications? A Wiki topic was started on this. We will try to use the Wiki for generating additional examples, and make a catalog of techniques and practices (and use the QAWG list for issue discussion/resolution).

20040607, reopened in response to message from originator, that points out probable misreading of the issue.

Proposal: [Originator] "all W3C specifications defining a notion of instance data should be explicitly required to identify all programmatically reportable errors, make reporting these a requirement for a specific class of product, define how to identify such software and define how to identify instances which do not have reportable errors."
Resolution: [Originally closed by...] Agreed in principle that Specification Guidelines (SpecGL) should say something about this. Exactly what is appropriate for the SpecGL context is to be determined. A Wiki was started on this, to generate additional examples, and make a catalog of techniques and practices. We have put placeholders in the just-published "SpecLite" (lightweight SpecGL), to be fleshed out in a future revision. (Search on "error handling", and also see for example the placeholder section D.5.)

Currently reopened, per message from originator, that points out probable misreading of the issue.

CR-30 SpecGL 2004-02-11 SpecGL Editorial Closed Bjoern Hoehrmann Dominique Hazaël-Massieux
Title: SpecGL does not define "testable" or "measurable" but uses them in ConfReqs.
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now. Significant discussion thread starts there.)

Proposal: ...tbd...
Resolution: Agreed. Although formal "Conformance Requirements" sections are not a feature of the new, lightweight SpecGL, we will nevertheless try to we will try to ensure that all the terms we use in Principles and Good Practices are documented, since they are the equivalent of our old Conformance Requirements. An initial definition for "testable", for example, may be found in the just-published "SpecLite" (lightweight SpecGL) glossary section.
CR-31 SpecGL 2004-02-11 SpecGL Editorial Closed Bjoern Hoehrmann Dominique Hazaël-Massieux
Title: SpecGL does not define what is an "implementation".
Description:

Comment reference.

Specification reference.

(Only description is at "Comment reference" for now. Significant discussion thread starts there.)

Proposal: ...tbd...
Resolution: We will try to ensure that all the terms we use in Principles and Good Practices are clearly documented, since they are the equivalent of our old Conformance Requirements. An initial definition for "implementation", for example, may be found in the just-published "SpecLite" (lightweight SpecGL) glossary section.
CR-32 QAF 2004-02-11 QAF Substantive Resolved Jeremy Carroll Dominique Hazaël-Massieux
Title: QAF contravenes W3M resolution that Rec-track specs should not coerce the WGs.
Description:

Comment reference.

"Comment reference" states the nature of Originator's objection: he asserts that QAF violates a W3M resolution, which basically says that Rec-track specifications should address technology requirements, and may define conformance for implementations of the technologies, but should not place requirements on Working Groups themselves, that they must conform to some specified requirements.

In dialog on qa-chairs list, Karl Dubost and Dominique Hazaël-Massieux present counter-arguments and assert that in fact W3M pointed to QAF as a *correct* approach. As they point out, the MUST (and other RFC2119) usage in OpsGL describe what a WG must do to conform to OpsGL. Nowhere does it say that a WG must conform to OpsGL. The latter (according to previous QAWG discussions and resolutions) is a W3C policy issue.

Commenter refers to QAWG Issue 16 and Issue 71. While these issues express QAWG sentiment (from 2002) that some conformance to QAF should be required in the future, such (potential) policy is not embedded in the QAF specs themselves (or anywhere else, now).

Summary (and possible proposed resolution): the comment misinterprets what QAF actually addresses. XyzGL (Xyz = Ops or Spec or Test) defines what an object (WG, spec, TM) must do to conform to XyzGL. It does not require that the object conform. Queue an editorial issue to see if this could be made even clearer in QAF.

Related CR issues: 3, 32.

See also related detail and argumentation by author.

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Proposal: The comment misinterprets what QAF actually addresses. XyzGL (Xyz = Ops or Spec or Test) defines what an object (WG, spec, TM) must do to conform to XyzGL. It does not require that the object conform. Queue an editorial issue to see if this could be made even clearer in QAF.
Resolution: The comment misinterprets what QAF actually addresses. XyzGL (Xyz = Ops or Spec or Test) defines what an object (WG, spec, TM) must do to conform to XyzGL. It does not require that the object conform. This will be even clearer in the redesigned QAF. While old-QAF did not violate W3C Process or policies, nevertheless please see Resolution of CR-1 -- in the new QAF, a more helpful, less authoritarian approach will de-emphasize strict testability of its presented advice, principles, good practices, etc.
CR-33 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Dominique Hazaël-Massieux
Title: QAWG failed to formally address Last Call comment before requesting CR transition.
Description:

Comment reference.

Issue synopsis. "Comment reference" states the nature of Originator's objection: he objects that QAWG did not formally address Last Call comments of his before advancing to CR. He points to a long email message of July 2003, about the May 2003 TestGL Working Draft (WD), in which he had many criticisms. The main thrust of his message was about the waterfall model, and we did in fact make substantial revisions to TestGL in response.

"Comment reference" specifically highlights the QAWG failure to address his July 2003 comment about "QAF contravenes W3M policy" (newly restated in January 2004 as CR-32).

QAWG observations:

  1. The July 2003 email thread in which the alleged unaddress QAF Last Call (LC) comment occurred was about TestGL, which we considered to be in early WD, not in the LC set.
  2. Commenter never said until 6 months later (Jan 2004), that he wanted to raise a (OpsGL & SpecGL) CR-advancement objection based on this comment of his about a non-LC TestGL WD.
  3. QAWG, perhaps understandably, did not detect this TestGL WD comment as a formal objection to OpsGL and SpecGL CR.
  4. this objection relies critically on the related substantive/procedural issue about @@all-QAF-sync@@.

See also issue CR-34.

Related CR issues: CR-33, CR-34.

Proposal: ...tbd...
Resolution: ...tbd...
CR-34 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Dominique Hazaël-Massieux
Title: QAWG failed to list open issues when advancing to CR.
Description:

Comment reference.

Issue synopsis. "Comment reference" states the nature of Originator's objection: he asserts that two comments threads of his about (early-WD) TestGL were not listed as open issues when requesting CR advancement for OpsGL and SpecGL. The stated rationale for the relevance of his TestGL WD comments to LC/CR SpecGL and OpsGL is:

  • when advancing the two CR documents the QAWG gave a misleading impression that the related work of the whole QAF was stable;
  • and, splitting of the QAF into three recommendation track documents advancing at different speeds, was prejudicial against his timely comments on the Test Guidelines.

This issue is closely related to CR-33.

Related CR issues: CR-33, CR-34.

Proposal: [Originator] Return all documents to Last Call when the Test Guidelines are ready, after all comments on all documents have been formally addressed, and then advance them all in synchronization.
Resolution: ...tbd...
CR-35 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Lofton Henderson
Title: QAF is only a "test development framework", therefore it is misnamed.
Description:

Comment reference.

(Only description is at "Comment reference" for now.)

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Related issues: the author's (non-critical) preceding comment is similar and supportive to this one.

Proposal: [Originator] Rename QAF.
Resolution: ...tbd...
CR-36 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Dominique Hazaël-Massieux
Title: QAWG has failed to meet the quality commitments in its charter.
Description:

Comment reference.

(Only description is at "Comment reference" for now.)

Comment. The referenced QAWG charter is the new one, approved by AC in fall 2003 (@@exact date?@@).

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Proposal: [Originator] See comment reference.
Resolution: ...tbd...
CR-37 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Lofton Henderson
Title: Make QAF informative, and represent its key normative content with small changes to W3C Process Document.
Description:

Comment reference.

(Only description is at "Comment reference" for now.)

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Proposal: [Originator] Make QAF informative, and represent its key normative content with small changes to W3C Process Document.

[LH] The proposed revisions to the QAF would address this concern.

Resolution: ...tbd...
CR-38 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Dominique Hazaël-Massieux
Title: QAF violates RFC2119.
Description:

Comment reference.

The "Comment reference" is the starting place for the substantive complaint of Originator. The bullet list of references at "Comment reference" are examples of where the alleged violation of RFC2119 (Sec.6) occurs.

History: extensively discussed and resolved during Last Call, in LC-67.

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Proposal: [Originator] See "Comment reference".
Resolution: ...tbd...
CR-39 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Lofton Henderson
Title: All parts of QAF, including supporting informative & resource references, must advance synchronously.
Description:

Comment reference.

This issue incorporates by reference a list of sub-issues or comments, which appear to warrant individual treatment (QAWG may agree with some, disagree with others):

  1. SpecGL reference to SpecET (4.8) -- two objectionable aspects: CR text referencing "Editors draft" (according to SoTD of SpecET); and (veiled OpsGL/ET comment here?), CR text referencing something for which "no official publication".
  2. References to QA web pages and non-QAF resources (4.11) -- submitter asserts that informative references to things like QAWG/IG home pages, QA Library, Taxonomy, QA Glossary, are inappropriate from a CR document. Two aspects should be considered: resource is incomplete in a substantive way where it is intended to support the QAF GL document; and, resource is a living work-in-progress that may never be "finished".
  3. TAXONOMY is unintelligible (4.12)
  4. Ops & Spec should not advance faster than Test (4.14) -- this comment considers QAF to consist inseparably of Ops, Spec, and Test, therefore no component should have gone to LC or CR without all of them progressing.
  5. OpsGL makes Normative Ref to pre-Last Call in CR (5.9) -- similar to previous. However the title of this one asserts "Normative Reference". The case that these references are actually normative, as opposed to informative, is made in submitter's previous comment (5.8).
  6. OpsGL informative supporting material unfinished (5.11) -- too much supporting material like QAWG QAPD is unfinished, and therefore the move to LC and CR was premature.
  7. OpsGL informative supporting material unfinished (7.2) -- too much supporting material like SpecET is unfinished, and that makes SpecGL harder to understand.
  8. Ops & Spec should not be advanced without Test (10.1) -- Not possible to review OpsGL & SpecGL without TestGL/ET. specifically, it is asserted to be "an abuse of process".
  9. Cost Benefit Analysis (14.2) -- at this reference is actually an assertion that there are lower cost ways to achieve similar benefit, a variant on issue CR-37. Elsewhere, submitter requests completion of a CBA, and points out that QAWG said (LC-111.2) we would finish one.
  10. Perform adequate review before advancing documents (14.7) -- submitter characterizes the work as grossly unfinished, and then proceeds onward with some fairly intemperate assertions.

Note. In these sub-issues, numerous references to "inappropriate" or "circumvent pubrules & W3C Process". But specific references to violated rules are @@missing@@. Some research is needed here.

Note. Many of the comments in this list are repeated in issue CR-41, which alleges numerous specific breaches of W3C Process, Pubrules, etc.

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Proposal: [Originator] ...tbd...

[LH] The development plan which would necessarily be associated with the proposed revisions to the QAF would address many of these individual points -- either they would become moot, or would be approved, or QAWG would disapprove the need for synchronization. Point-by-point details TBD.

Resolution: ...tbd...
CR-40 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Lofton Henderson
Title: 2nd Last Call needed, with stricter advancement criteria.
Description:

Comment reference.

  1. The principal and critical assertion of this issue is found at "Comment Reference" -- the magnitude of needed changes to QAF will necessitate 2nd Last Call (all parts).
  2. A related comment by submitter asserts that QAWG's failure to meet its (fall 2003 re-charter) Charter commitment of AAA OpsGL conformance is by itself sufficient cause to have to return to Last Call.
  3. Concerning advancement, he suggests that QAWG AAA conformance "in all respects" should be part of PR entrance criteria. He reiterates in a closely related comment that QAWG must demonstrate that it meets its own committed (AAA) quality standards.

Note. In the author's collected list of formal issues, the "2nd Last Call" part of this one is specifically identified as one requiring a response.

Proposal: [Originator] ...tbd...

[LH] The proposed revisions would necessitate synchronization of the components at another Last Call.

Resolution: ...tbd...
CR-41 QAF 2004-02-11 QAF Substantive Active Jeremy Carroll Dominique Hazaël-Massieux
Title: QAWG & QAF have many breaches of W3C Process, pubrules, etc.
Description:

Comment reference.

Submitter alleges numerous specific violations of W3C Process, Pubrules, etc. He enumerates these: 4.8, 4.11, 5.8, 5.9, 5.11, 6.1, 6.2, 6.3, 6.4, 8.1, 8.2, 8.3, 8.4, 8.5, 11.2, 11.4, 13.1.

Originator identifies his comment 8.5, about QAWG's handling of Dan Connolly's late LC comment about RFC2119, as alone sufficient to support the present issue. There is @@some history@@ [links tbd] about the RFC2119 issue. The present objection is apparently not about the substance of the issue itself (LC- 67), but a procedural objection that QAWG didn't handle DC's comment properly (note that it was handled under old Process, pre-July-2003.)

The enumerated alleged specific violations are (mostly) given without a specific reference to a rule being violated. To respond to this issue, QAWG must either: research each point and prove "no discernable violation"; or, request Originator to provide a specific Process or Pubrules reference for each one.

Author has supplied these clarifications and references.

Note. Many of the comments in this list are repeated in issue CR-39, which is about the need to advance all QAF- related at equal levels of maturity.

Note. In the author's collected list of formal issues, this one is specifically identified as one requiring a response.

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-42 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Test Assertions -- definitions
Description:

Comment reference.

Specification reference.

This is one of a handful of related issues about Test Assertions that have been generated out of the mail thread starting at "Comment reference." This concerns definition of terms used in the discussion.

In the middle of the thread, it becomes clear that different people are using the terms Conformance Requirements and Test Assertion slightly differently.

  • [MS]"...Additionally, Andrew (or perhaps someone else) introduced the idea that the assertions are statement of fact about the implementation rather than requirements imposed (e.g., the spec contains rather than the spec MUST or the implementation produces a file that ... rather than the implementation MUST). This may be a subtle distinction but one that really makes a lot of sense (at least to me)."

That distinction was made in 16-December-2002 telecon, and is embedded in the definitions in SpecGL.

Precise definitions of the terms should be agreed -- either: 1.) reaffirm the existing definitions (and historical interpretation), and enhance them (e.g., with examples) to further clarify; or 2.) change them to agreed consensus definitions.

Related CR issues: 42, 43, 44.

Proposal: [Originator] Option 1, reaffirm and clarify.
Resolution: ...tbd...
CR-43 SpecGL 2004-02-15 SpecGL Substantive Active Mark Skall unassigned
Title: Test Assertions -- how and where?
Description:

Comment reference.

This is one of a handful of related issues about Test Assertions (TAs) that have been generated out of the mail thread starting at "Comment reference." This concerns how QAWG should handle the derivation of Test Assertions for the GL specifications. Two options are suggested by Originator [MS]:

  1. The TAs derived by QAWG and published with the GL documents as normative (external) parts/appendices may combine additional (informative) material with the normative material of the ConfReqs, to make the TAs more formally precise and testable than the ConfReqs.
  2. The TAs derived by QAWG must be based purely on other normative content, which means (now) exactly on the ConfReqs -- in other words, TAs and ConfReqs duplicate each other.

    MS suggested a 3rd option, which is a variant on (or way achieve) #2:

  3. Achieve #2 by starting with #1, and then augmenting the ConfReqs so that a straightforward transformation of ConfReqs leads to the TAs.

    PC's contributions in the thread suggest another variant of #2:

  4. Test Assertions and/or ConfReqs (terminology question arises here) should be formally precise and testable, embedded in the specification (and marked up), and there should be no second set of redundant normative content that is derived.

Note. This issue interacts with SpecGL Checkpoint 7.1, which creates strong pressure towards the form of testable statements that uses RFC2119 keywords -- the Test Assertion form of expression would require justification in a specification.

Related CR issues: 42, 43, 44.

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-44 SpecGL 2004-02-15 SpecGL Substantive Active Lynne Rosenthal unassigned
Title: Test Assertions -- should SpecGL require them?
Description:

Comment reference.

Specification reference.

This is one of a handful of related issues about Test Assertions that have been generated out of the mail thread starting at "Comment reference." This issue concerns whether or not SpecGL should require that specifications provide test assertions, i.e., should Guideline 10 and its two checkpoints be kept in SpecGL? Two questions are posed:

  1. what is the goal in requiring TAs in SpecGL - what are we trying to do?
  2. is this the only way to accomplish this goal?

Related CR issues: 42, 43, 44.

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-45 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: SpecGL & SpecET inconsistencies about CoP and scope requirements.
Description:

Comment reference.

Specification reference.

In CP1.1, SpecGL says that Class of Product (CoP) help define scope. SpecET, for CP1.2, says list-of-CoP help illustrate scope. These should be reconciled and made consistent.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-46 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: CP2.2 is hard to implement.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

It is unclear what is wanted for the implementation of CP2.2. On the one hand, there are the many detailed Conformance Requirements associated with all of the technical details of a specification. On the other hand, there are high-level conformance criteria or "schemes" that typically are applied to various CoP. I think we mean the relatively few criteria of the latter, rather than the (scores, hundred, thousands, ..) detailed conformance requirements. The examples of SpecET help some, but this should be disambiguated in the Techniques of SpecET (at least).

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-47 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Rationale of CP2.3 (spec. category) is unconvincing, and it is unclear how to satisfy the CP.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

This seems like a "weak" checkpoint:

  1. The Rationale is not convincing. Perhaps this is related to a second problem -- taking all of the material in the checkpoint, the explanatory material in Sec2.2, and the material in SpecET, it was still unclear how to satisfy the checkpoint.
  2. The Examples in SpecET are unrelated to the meaning of the checkpoint -- they deal with "field of application", not "specification category".

Possible solutions: delete the checkpoint; or, in SpecET include a table of all W3C specifications that are past Last Call (including Recs), and give their "standard" Specification Category, i.e., from the standardized list of SCs. (Successfully completing such a table would be convincing evidence that the CP is implementable!)

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-48 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Error in Conformance Requirements of CP3.2 (profile min. rqts.)?
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

In the Conformance Requirements of CP3.2, is "for each profile" wrong? It seems to make more sense that minimum requirements would apply across a group of things, i.e., across all profiles. I.e., this would be the intersection of the sets of requirements for all profiles. If "for each profile" is indeed intended, then how do "minimum requirements for Profile XYZ" differ from "the requirements for Profile XYZ"?

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-49 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: The statement of CP5.1 (discretionary items) is unrelated to the given Conformance Requirements.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

The statement of CP5.1 is about circumstances under which discretionary items are allowed. The Conformance Requirements are about explaining the Rationale and naming each item. Furthermore, in Rationale the "link target for each one" -- perhaps more useful than a name in automated environments -- is not required in the ConfReqs.

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-50 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Redundant requirement in CP5.3 (number of disc. items/choices).
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

The 2nd Conformance Requirement -- indicating any dependency of the number upon other DoV -- is apparently redundant with CP5.5 (relationship to other DoV).

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-51 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Inconsistencies in CP5.4 (consistent handling of discretionary choices).
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

There are several problems in the checkpoint:

  1. The Conformance Requirements are all about "terminology", and seem unrelated to parts of Rationale ("same results under same conditions"), and to the CP statement itself ("consistent handling").
  2. If this CP *is* about terminology, then it would appear to be redundant with CP7.3.
  3. The CP statement is about "choices", but Conformance Requirements are about "items" ("choice" is one of three types of "item").
  4. This CP has a @@long history@@ (tbd...get past links). After the @@Tokyo f2f@@ (Oct 2002), the substantive purpose of the CP was agreed to be -- collective specification for related choices. This resolved main purpose has completely disappeared from the CP, and only (IMO) trivial purposes remain.
Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-52 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Problems in CP5.5 (relationship of discretionary items with other DoV).
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

There are several problems in this checkpoint:

  1. The 2nd paragraph of Conformance Requirements (ConfReqs) is not about relationship to other DoV, it is about inter-relationship of choices within the Discretionary Items DoV.
  2. About the phrase "In other words," in the ConfReqs 2nd paragraph -- does that mean that the ConfReq preceding it is redundant with the ConfReq following it?
  3. The ConfReqs in the 2nd paragraph seems to belong to CP5.4 "consistent handling" (it looks a lot like the missing post-Tokyo part of CP5.4).
  4. The "Rationale" is not a rationale, and is entirely focused on Use Cases, which aren't mentioned in CP statement or ConfReqs.
Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-53 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Error in Conformance Requirements of CP6.5 (mitigate extensions)?
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

Is last sentence of Conformance Requirements true? Why wouldn't this CP apply, e.g., to extensions to an API standard?

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-54 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: CP8.1 (universal reqts for min functionality) is difficult to implement.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

There is ambiguity which make this checkpoint (IMO) unimplementable, namely: How do "universal requirements for minimum functionality" differ from CP2.2's "per-CoP conf. reqs."? I.e., what is the difference between the Conformance Requirements for a CoP and the "universal requirements for minimum functionality for [that] CoP"?

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-55 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Define "conformance concepts" in CP8.2.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

To be implementable, CP8.2 needs to define "conformance concepts", or give lots of guidance and/or examples to illustrate what the phrase means.

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-56 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: How do "conformance designations" (CP8.5) relate to "conformance concepts" (CP8.2)?
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

How does CP8.5, "define all conformance designations", relate to CP8.2, "define (special) conformance concepts"? Is one an aspect of the other?

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-57 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: The meaning and purpose of "conformance disclaimer" are unclear in CP9.2.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

OpsGL requires a conformance disclaimer for test suites -- passing the test suite does not imply conformance to the spec. But this is SpecGL. What is the disclaimer required by this CP? The intent and purpose of the required disclaimer is very mysterious.

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-58 SpecGL 2004-02-15 SpecGL Substantive Active Lofton Henderson unassigned
Title: Clarify what comprises "list" of Test Assertions.
Description:

Comment reference.

Specification reference.

This is one of several issues that have been uncovered during a close analysis of SpecGL, at "Comment reference."

CP10.1 Conformance Requirements are not specific enough to decide whether or not they are satisfied by some cases. For example, if TAs were marked-up & styled in-line in text, does that comprise a "list" and satisfy CP10.1?

Proposal: [Originator] ...tbd...
Resolution: ...tbd...
CR-59 SpecGL 2004-02-15 SpecGL Substantive Active Jeremy Carroll ...tbd... unassigned
Title: ...~20 substantive SpecGL issues from JC will start here...
Description:

Comment reference.

Specification reference.

...tbd...

Proposal: [Originator] ...tbd...
Resolution: ...tbd...

Table Legend

num
Candidate Recommendation issue number
Title
Short title/name of the issue
Spec
Document referred to in Candidate Recommendation issue (QAF = "QA Framework" document family, collectively; Intro = QA Framework: Introduction; OpsGuide & OpsGL = QA Framework: Operational Guidelines; SpecGuide & SpecGL = Framework: Specification Guidelines; TestGuide & TestGL = Framework: Test Guidelines)
Description
Short description of issue, possibly including link to origin of issue
Date
The date at which the issue was raised or initially logged.
Topic
Rough topic categorization, one of: ...tbd... [replace following XMLP stuff... env(elope), rpc, enc(oding), meta(issue), bind(ing), fault]
Class
Substantive or Editorial
Status
One of: Unassigned, Active, Closed, Postponed
Proposal
Current proposal for resolution of issue, possibly including link to further text
Resolution
Short description of resolution, possibly including link to a more elaborate description
Raised by
Person who raised the issue
Owner
QA WG Member responsible for the issue

Maintained by Lofton Henderson.