Template analysis for "QA Framework: Specification Guidelines"

Corresponding version of "QA Framework: Specification Guidelines":
http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/
Author:
Lofton Henderson
Date/version:
15 February 2004

Contents


Introduction

This document records the raw data for the SpecGL/SpecET analysis phase of the GL-templates project.

Analysis data

The usability of SpecGL is determined by these bits in combination:

  1. The clarity and completeness of SpecGL.
  2. The number and quality of the Examples in SpecET.
  3. The number and quality of the Techniques in SpecET.
  4. The provision of good templates as adjuncts to the Techniques of SpecET.

The SpecGL/ET analysis therefore took a look at 1, 2, and 3. #1 resulted in a collection of editorial and substantive comments (issues) about SpecGL.

#2 and #3 are necessarily somewhat subjective. I read the checkpoint in SpecGL, and then followed the link to SpecET. I tried to imagine how an average user (not a QAWG expert) would understand the SpecET content and what he/she would gain from it.

These data are contained in the following table, which contains all of the guidelines and checkpoints of SpecGL (the CPs are linked to the full SpecGL Candidate Recommendation document).

Also in the following table, there is a "Where" datum for each checkpoint. With two exceptions where SpecGL requires a particular spec location, I have considered and chosen a "good" location (or set of locations) for a CP's fulfillment. These are not unique or normatively exclusive of other possibilities, but ought to result in a clear and nicely arranged spec. This is the purpose of the templates ajdunct to SpecGL/ET -- give one good, easy way to do it, rather than explore all possibilities.

The "Templates potential" item is an assessment of the degree to which the checkpoint's implementability would be enhanced by having a template (presumably linked from SpecET). Note that the potential refers only to whether the particular checkpoint (or small collection of related CPs) has potential to benefit from a template. It does not consider what might be the highest priority template -- a super- or composite template that outlines a complete specification, reflecting (almost) all of SpecGL's CPs, and probably embedding a number of specific and focused templates. In other words, a AAA-compliant skeleton for a specification specification. Caveat. This column should be considered to be "working notes" from the analysis phase, rather than polished assessments or recommendations.

The priority of each checkpoint is also given -- a prioritized list of potential templates probably must factor in the checkpoints' priorities. (Note. The prioritized template list indicates a suggested importance and order for development of the templates, not whether or not they are useful.)

Issues about "template potential"

Caveat about "template potential": In this version, the information in that column of the table should be considered "working notes" (rather than a polished assessment) from the analysis phase. Actual recommendations will be made in the report for the project.

Key to Analysis Table

SpecGL analysis table

Guideline 1. Define Scope.

Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
1.1

Include a scope statement.  [P1]

 ok  ok  Intro(!)  mid (e.g., ISO material, partially done now in T)  
1.2

Illustrate scope.  [P2]

  • Question: ET/T says "list-of-CoP illustrates scope". Is this appropriate for T for CP1.2? (*CP1.1* says that CoP helps define scope,)
 0  unr  Intro,
external
 Need high (no E or T exists), potential low (because of wide variety and uniqueness).  S
1.3

Illustrate functional details.  [P2]

 plus  plus Body,
external
 Potential low. ET/E much more useful.  

Guideline 2. Identify what needs to conform and how.

Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
2.1

Identify classes of product.  [P1]

 plus  minus  Intro,
CC
 ET/E, or seems more useful (because of wide variety and uniqueness). Potential mid ... a choice list in tmpl, and/or a QT.  E
2.2

For each class of product, define the conformance requirements.  [P1]

  • Unclear in SpecGL how to satisfy -- doesn't distinguish detailed ConfReqs (many) from Conf. Criteria (few collective in CC). (I think latter is intended.)
  • ET/E helps, but ET/T should disambiguate more.
  • Would CC/PP be a good addition to ET/E?
 plus  minus  CC  ET/E seems more useful. Potential low (because of wide variety and uniqueness). [Concept & realization simple.]  S
2.3

Identify the applicable specification categories.  [P3]

  • CP is "weak": 1.) unconvincing rationale; 2.) from CP, sec2.2, and ET combined, it is still unclear how to satisfy
  • ET/E are unrelated to meaning of CP -- they deal with "field of application", not "specification category"
 ok,
unr
 0  Intro(!)  ET/E seems more useful (because of wide variety and uniqueness). Potential mid -- a choice list in tmpl, and/or QT .  E, S
2.4

If there are several classes of products, define their relationships and interaction with other dimensions of variability.  [P2]

 0  0  CC  Utility would be high. Potential mid (could be tricky to do -- matrix for all DoV?). Lots of ET/E needed.  E

Guideline 3. Address the use of profiles, modules and levels to divide the technology.

Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
3.1

Indicate whether or not the use of profiles is mandatory for conformance.  [P1]

 minus,
unr
 minus,
unr
 CC,
(or Sec)
 Potential low (it is a simple y/n statement). High need for ET/E & ET/T.  E
3.2

If profiles are chosen, define any minimal requirements.  [P2]

  • Is "for each profile" wrong (ConfReqs)? I thought the min rqts were across all profiles.If not, how do the "min rqts for a profile" differ from "the rqts of the profile"?
 minus  minus  CC,
(or Sec)
 Potential low (because of wide variety and uniqueness). High need for ET/E & ET/T.  S
3.3

If profiles are chosen, address rules for profiles.  [P2]

 minus  plus  CC,
(or Sec)
 Potential low. ET/E and ET/T are needed.  E
3.4

If modules are chosen, indicate any mandatory conditions or constraints on their usage.  [P1]

 minus  0  CC,
(or Sec)
 Potential low (because of wide variety and uniqueness). ET/E and ET/T needed.  
3.5

If profiles, modules or levels are used, define their relationships and interaction with other dimensions of variability.  [P2]

 minus  0  CC   Utility would be high. Potential mid (could be tricky to do). Lots of ET/E needed.  

Guideline 4. Identify the relation between deprecated features and conformance.

E
Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
4.1

Identify each deprecated feature.  [P1]

 plus  ok  Body
& CC
 Potential mid for 4.1, 4.2, 4.4, 4.5 together -- link table in CC plus sample language in Body.  
4.2

For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation.  [P1]

 plus  ok  CC
(for policy)
 Potential low? (because of wide variety and uniqueness). ET/E and ET/T are better. [Mid? A sample statement that could be placed at each DF? Markup?!]  
4.3

If deprecated features exist, define their relationships and interaction with other dimensions of variability.  [P2]

 0  0  CC   Utility would be high. Potential mid (could be tricky to do). Lots of ET/E needed.  
4.4

Include an explanation for the deprecation.  [P3]

 minus,
unr
 minus,
unr
 Body  See 4.1.  
4.5

Include examples to illustrate how to avoid using deprecated features.  [P3]

 ok  0  Body  See 4.1.  
4.6

Identify each obsolete feature.  [P3]

 0  0  CC  Potential & utility low-mid -- list in section of CC.  

Guideline 5. Define discretionary items.

E
Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
5.1

State the circumstances for when discretionary items are allowed.  [P1]

  • CP statement & ConfReqs seem unrelated.
 minus,
unr
 okay  CC &
Body
 deferred  E, S
5.2

For implementation dependent values or features, address the allowable differences between implementations.  [P1]

 okay,
unr (1st)
 0  Body  deferred  
5.3

Indicate any constraints on the number of choices/options that can be implemented for discretionary items.  [P2]

  • Is 2nd ConfReq redundant w/ CP5.5?
 minus,
unr
 minus
 Body  deferred  S, E
5.4

Promote consistent handling of discretionary choices.  [P2]

  • ConfReqs (all about "terminology") seem unrelated to parts of rationale (same results under same conditions) and CP statement.
  • CP statement is about "choices" but ConfReqs is about "items".
  • Substantive purpose post-Tokyo -- collective specification for related *choices* has been lost and only trivial purposes remain.
 0  0  ???  deferred  S
5.5

If discretionary items are used, define their relationships and interaction with other dimensions of variability.  [P2]

  • The ConfReqs 2nd pgph is not about other DoV, it is about inter-relationship of choices within the Discretionary Items DoV.
  • Redundant w/ CP7.3
  • The ConfReqs 2nd pgph seems to belong to CP5.4 (it is the missing post-Tokyo part?).
  • The "Rationale" is not a rationale, and is entirely focused on Use Cases, which aren't mentioned in CP statement or ConfReqs.
 0  0  CC   deferred

Utility would be high. Potential mid (could be tricky to do -- an all-DoV matrix?). Lots of ET/E needed.

 S

Guideline 6. Allow extensions or not

E
Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
6.1

Indicate if the specification is extensible, and if extensions are allowed, define their scope and constraints.  [P1]

 0  ok  CC  Potential low, due to high variability and uniqueness.  E
6.2

Prevent extensions from contradicting the specification.  [P1]

 0  0  CC  High -- include a prototype statement in T.  
6.3

Define a uniform mechanism to create an extension.  [P3]

 minus  minus  CC
(or Body?)
  Potential low, due to high variability and uniqueness. Good E's are needed.  E
6.4

Require that extensions be published.  [P3]

 0  minus  CC  High -- include a prototype statement in T.  E
6.5

Mitigate the impact of extensions on interoperability.  [P3]

  • Is last sentence of ConfReqs true? Why wouldn't this CP apply, e.g., to extensions to an API standard?
 0  0  CC  Potential low, due to high variability and uniqueness. Lotsa' E's are needed.  E, S
6.6

If extensions are allowed, define their relationships and interaction with other dimensions of variability.  [P2]

 0  0  CC   Utility would be high. Potential mid (could be tricky to do). Lots of ET/E needed.  

Guideline 7. Clearly identify conformance requirements.

Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
7.1

Use conformance key words.  [P1]

 minus  0  Body &
CC-subsec
 Potential high for model CC statement.  E
7.2

Distinguish normative and informative content.  [P1]

 minus  0  Body &
CC (sec?)
 Lotsa' E needed. T could provide statements for several alternate methods.  E
7.3

Use consistent terminology.  [P1]

 0  0  Body  No potential.  
7.4

Provide a fast way to find conformance information.  [P2]

 0  ok  CC ,
ToC, ???
 Potential high for ToC & CC.  E
7.5

Make normative reference to specifications on which the current specification depends.  [P1]

 ok  ok  Body &
Refs
 Potential mid -- low/Body, high/Refs. ET/T explanation better.  

Guideline 8. Document the conformance policy.

Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
8.1

Specify any universal requirements for minimum functionality.  [P2]

  • how does this differ from CP2.2 (per-CoP conf. reqs.)? What is the difference between the conf. reqs. for a CoP and the "univ reqts for min functionality for [that] CoP"?

 plus  0  Sec(!) ,
in CC
 deferred  S
8.2

If special conformance concepts are used, include a definition in the specification.  [P1]

  • need to define "conformance concepts", or give guidance and/or examples to illustrate what it means.
 ok  minus  CC  Low. Lotsa' E instead. (CC-sec? Glossary?)  S
8.3

Justify any usage of a dimension of variability.  [P1]

 0  0  CC  Potential low, due to high variability and uniqueness.  
8.4

Include a conformance section.  [P1]

 minus  0  Sec(!) ,
in CC
 Very high.  
8.5

Identify and define all conformance designations.  [P1]

  • how does this relate to CP8.2? Is one an aspect of the other?
 minus  0  CC  Mid-high (as conformance sub-sec). Hard to do?  S

Guideline 9. Specify how to make conformance claims.

E
Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
9.1

Provide specific wording of the claim.  [P3]

 ok  0  CC  Potential high  E
9.2

Provide a conformance disclaimer.  [P3]

  • confusion between a TS disclaimer & a "spec" disclaimer -- what is the latter?
 minus,
unr ???
 0  CC  deferred (hard to do because of unclear concepts).

(but potential would be high?)

 E
9.3

Impose no restrictions about who can make a claim or where claims can be published.  [P1]

 minus  0  CC  Potential high. Easy.  E
9.4

Provide an Implementation Conformance Statement proforma.  [P3]

 minus,
unr
 ok  CC,
external
 Potential low (because of wide variety and uniqueness). ET/E seems better.  E
9.5

Require the ICS be completed as part of the conformance claim.  [P3]

 0  0  CC  High (include wording in 9.1?).  

Guideline 10. Provide test assertions.

Nbr Checkpoint ET/E ET/T Where Template
potential
Cmmts
10.1

Provide test assertions.  [P2]

  • ConfReqs question: if TAs were marked-up & styled in-line in text, does that comprise a "list"?
 0  minus  Body,
or CC/ext
 (Whole different project.)  E, S
10.2

Provide a mapping between the specification and the test assertions list.  [P2]

 0  0  n/a,
or CC/ext
 (Ditto.)  E

Acknowledgements

This report was prepared with the support of National Institute of Standards and Technology, Gaithersburg, MD 20899-8970.


References

QAF-SPEC
QA Framework: Specification Guidelines, D. HazaŽl-Massieux, L. Rosenthal, L. Henderson, Eds., W3C Candidate Recommendation , available at http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/ .
SPEC-EXTECH
QA Framework: Specification Examples & Techniques, Karl Dubost, Lofton Henderson, Lynne Rosenthal, Eds., current companion to [QAF-SPEC], available at http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech-20030912 .
ED-COMMENTS
Email/text file containing SpecGL editorial comments, in email archives at http://lists.w3.org/Archives/Public/www-qa-wg/2003Dec/0042.html .