Template analysis for "QA Framework: Specification Guidelines"

Corresponding version of "QA Framework: Specification Guidelines":: http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/
Author:: Lofton Henderson
Date/version:: 17 December 2003

This document records the raw data for the SpecGL/SpecET analysis phase of the GL-templates project.

Analysis data

The SpecGL/ET analysis therefore took a look at 1, 2, and 3. #1 resulted in a collection of editorial and substantive comments (issues) about SpecGL.

#2 and #3 are necessarily somewhat subjective. I read the checkpoint in SpecGL, and then followed the link to SpecET. I tried to imagine how an average user (not a QAWG expert) would understand the SpecET content and what he/she would gain from it.

These data are contained in the following table, which contains all of the guidelines and checkpoints of SpecGL (the CPs are linked to the full SpecGL Candidate Recommendation document).

Also in the following table, there is a "Where" datum for each checkpoint. With two exceptions where SpecGL requires a particular spec location, I have considered and chosen a "good" location (or set of locations) for a CP's fulfillment. These are not unique or normatively exclusive of other possibilities, but ought to result in a clear and nicely arranged spec. This is the purpose of the templates ajdunct to SpecGL/ET -- give one good, easy way to do it, rather than explore all possibilities.

The "Templates potential" item is an assessment of the degree to which the checkpoint's implementability would be enhanced by having a template (presumably linked from SpecET). Note that the potential refers only to whether the particular checkpoint (or small collection of related CPs) has potential to benefit from a template. The assessment ignores what might be the highest priority template -- a super- or composite template that outlines a complete specification, reflecting (almost) all of SpecGL's CPs, and probably embedding a number of specific and focused templates. In other words, a AAA-compliant skeleton specification.

Note that the priority of each checkpoint is given -- a prioritized list of potential templates probably must factor in the checkpoints' priorities. (Note. The prioritized template list indicates a suggested importance and order for development of the templates, not whether or not they are useful.)

Key to Analysis Table

ET/E: Examples subsection of CP's corresponding content in SpecET.

ok -- seems like an adequate number and completeness of Examples.
minus -- ought to be more examples, or more thoroughly developed.
plus -- superlative number and quality of Examples.
unr -- the Examples seem unrelated to the meaning of the CP, or to misinterpret it.

ET/T: Techniques subsection of CP's corresponding content in SpecET.

Same entries as for Examples.

Where: Implied or suggested (or required) place in spec where CP should be addressed.

Intro -- Introduction or subsection of Intro.
"(!)" suffix (and bold) -- SpecGL requires this location.
external -- content could be in linked external document, referenced from location.
Body -- scattered throughout the body of the spec, in all the places where functionality is defined.
CC -- in the Conformance Clause section (or sub-section of CC).
Sec -- in a dedicated section of the spec (other than a CC subsection).
Refs -- "References" section of spec.
ToC -- Table of Contents

Template potential -- Potential for improvement by offering template(s).

Comments -- Editorial or Substantive comments about SpecGL or SpecET

E -- editorial comments were made during analysis (in separate document). These can be found at:
- http://lists.w3.org/Archives/Public/www-qa-wg/2003Dec/0042.html
S -- substantive comments were made during analysis (temporarily in the table as bullets following the statement of the CP, may be moved to separate document later).

Analysis table for potential templates

Guideline 1. Define Scope.
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts @tbd@
1.1	Include a scope statement. [P1]	ok	ok	Intro(!)
1.2	Illustrate scope. [P2] Question: ET/T says list-of-CoP" illustrates scope. Appropriate for T for CP2.1? (CP1.1 says that CoP helps define scope,)	0	unr	Intro, external		S
1.3	Illustrate functional details. [P2]	plus	plus	Body, external
Guideline 2. Identify what needs to conform and how.
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
2.1	Identify classes of product. [P1]	plus	minus	Intro, external		E
2.2	For each class of product, define the conformance requirements. [P1] Unclear in SpecGL how to satisfy -- doesn't distinguish detailed ConfReqs (many) from Conf. Criteria (few collective in CC). (I think latter is intended.) ET/E helps, but ET/T should disambiguate more. Would CC/PP be a good addition to ET/E?	plus	minus	CC		S
2.3	Identify the applicable specification categories. [P3] CP is "weak": 1.) unconvincing rationale; 2.) from CP, sec2.2, and ET combined, it is still unclear how to satisfy ET/E are unrelated to meaning of CP -- they deal with "field of application", not "specification category"	ok, unr	0	Intro(!)		E, S
2.4	If there are several classes of products, define their relationships and interaction with other dimensions of variability. [P2]	0	0	CC		E
Guideline 3. Address the use of profiles, modules and levels to divide the technology.
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
3.1	Indicate whether or not the use of profiles is mandatory for conformance. [P1]	minus, unr	minus, unr	CC, (or Sec)		E
3.2	If profiles are chosen, define any minimal requirements. [P2] Is "for each profile" wrong (ConfReqs)? I thought the min rqts were across all profiles.If not, how do the "min rqts for a profile" differ from "the rqts of the profile"?	minus	minus	CC, (or Sec)		S
3.3	If profiles are chosen, address rules for profiles. [P2]	minus	plus	CC, (or Sec)		E
3.4	If modules are chosen, indicate any mandatory conditions or constraints on their usage. [P1]	minus	0	CC, (or Sec)
3.5	If profiles, modules or levels are used, define their relationships and interaction with other dimensions of variability. [P2]	minus	0	CC
Guideline 4. Identify the relation between deprecated features and conformance.	E
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
4.1	Identify each deprecated feature. [P1]	plus	ok	Body & CC
4.2	For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation. [P1]	plus	ok	CC (for policy)
4.3	If deprecated features exist, define their relationships and interaction with other dimensions of variability. [P2]	0	0	CC
4.4	Include an explanation for the deprecation. [P3]	minus, unr	minus, unr	Body
4.5	Include examples to illustrate how to avoid using deprecated features. [P3]	ok	0	Body
4.6	Identify each obsolete feature. [P3]	0	0	CC
Guideline 5. Define discretionary items.	E
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
5.1	State the circumstances for when discretionary items are allowed. [P1] CP statement & ConfReqs seem unrelated.	minus, unr	okay	CC & Body		E, S
5.2	For implementation dependent values or features, address the allowable differences between implementations. [P1]	okay, unr (1st)	0	Body
5.3	Indicate any constraints on the number of choices/options that can be implemented for discretionary items. [P2] Is 2nd ConfReq redundant w/ CP5.5?	minus, unr	minus	Body		S, E
5.4	Promote consistent handling of discretionary choices. [P2] ConfReqs (all about "terminology") seem unrelated to parts of rationale (same results under same conditions) and CP statement. CP statement is about "choices" but ConfReqs is about "items". Substantive purpose post-Tokyo -- collective specification for related choices has been lost and only trivial purposes remain.	0	0	???		S
5.5	If discretionary items are used, define their relationships and interaction with other dimensions of variability. [P2] The ConfReqs 2nd pgph is not about other DoV, it is about inter-relationship of choices within the Discretionary Items DoV. The ConfReqs 2nd pgph seems to belong to CP5.4 (it is the missing post-Tokyo part?). The "Rationale" is not a rationale, and is entirely focused on Use Cases, which aren't mentioned in CP statement or ConfReqs.	0	0	CC		S
Guideline 6. Allow extensions or not	E
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
6.1	Indicate if the specification is extensible, and if extensions are allowed, define their scope and constraints. [P1]	0	ok	CC		E
6.2	Prevent extensions from contradicting the specification. [P1]	0	0	CC
6.3	Define a uniform mechanism to create an extension. [P3]	minus	minus	CC (or Body?)		E
6.4	Require that extensions be published. [P3]	0	minus	CC		E
6.5	Mitigate the impact of extensions on interoperability. [P3] Is last sentence of ConfReqs true? Why wouldn't this CP apply, e.g., to extensions to an API standard?	0	0	CC		E, S
6.6	If extensions are allowed, define their relationships and interaction with other dimensions of variability. [P2]	0	0	CC
Guideline 7. Clearly identify conformance requirements.
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
7.1	Use conformance key words. [P1]	minus	0	Body & CC-subsec		E
7.2	Distinguish normative and informative content. [P1]	minus	0	Body & CC (sec?)		E
7.3	Use consistent terminology. [P1]	0	0	Body
7.4	Provide a fast way to find conformance information. [P2]	0	ok	CC , ToC, ???		E
7.5	Make normative reference to specifications on which the current specification depends. [P1]	ok	ok	Body & Refs
Guideline 8. Document the conformance policy.
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
8.1	Specify any universal requirements for minimum functionality. [P2]	plus	0	Sec(!) , in CC
8.2	If special conformance concepts are used, include a definition in the specification. [P1]	ok	minus	CC
8.3	Justify any usage of a dimension of variability. [P1]	0	0	CC
8.4	Include a conformance section. [P1]	minus	0	Sec(!) , in CC
8.5	Identify and define all conformance designations. [P1]	minus	0	CC
Guideline 9. Specify how to make conformance claims.	E
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
9.1	Provide specific wording of the claim. [P3]	ok	0	CC		E
9.2	Provide a conformance disclaimer. [P3]	minus, unr ???	0	CC		E
9.3	Impose no restrictions about who can make a claim or where claims can be published. [P1]	minus	0	CC		E
9.4	Provide an Implementation Conformance Statement proforma. [P3]	minus, unr	ok	CC, external		E
9.5	Require the ICS be completed as part of the conformance claim. [P3]	0	0	CC
Guideline 10. Provide test assertions.
Nbr	Checkpoint	ET/E	ET/T	Where	Template potential	Cmmts
10.1	Provide test assertions. [P2] ConfReqs question: if TAs were marked-up & styled in-line in text, does that comprise a "list"?	0	minus	Body, or CC/ext		E, S
10.2	Provide a mapping between the specification and the test assertions list. [P2]	0	0	n/a, or CC/ext		E

Guideline 1. Define Scope.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts
@tbd@

1.1

Include a scope statement. [P1]

Intro(!)

1.2

Illustrate scope. [P2]

Question: ET/T says list-of-CoP" illustrates scope. Appropriate for T for CP2.1? (*CP1.1* says that CoP helps define scope,)

unr

Intro,
external

1.3

Illustrate functional details. [P2]

plus

Body,
external

Guideline 2. Identify what needs to conform and how.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

2.1

Identify classes of product. [P1]

plus

minus

Intro,
external

2.2

For each class of product, define the conformance requirements. [P1]

Unclear in SpecGL how to satisfy -- doesn't distinguish detailed ConfReqs (many) from Conf. Criteria (few collective in CC). (I think latter is intended.)
ET/E helps, but ET/T should disambiguate more.
Would CC/PP be a good addition to ET/E?

plus

minus

2.3

Identify the applicable specification categories. [P3]

CP is "weak": 1.) unconvincing rationale; 2.) from CP, sec2.2, and ET combined, it is still unclear how to satisfy
ET/E are unrelated to meaning of CP -- they deal with "field of application", not "specification category"

ok,
unr

Intro(!)

E, S

2.4

If there are several classes of products, define their relationships and interaction with other dimensions of variability. [P2]

Guideline 3. Address the use of profiles, modules and levels to divide the technology.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

3.1

Indicate whether or not the use of profiles is mandatory for conformance. [P1]

minus,
unr

CC,
(or Sec)

3.2

If profiles are chosen, define any minimal requirements. [P2]

Is "for each profile" wrong (ConfReqs)? I thought the min rqts were across all profiles.If not, how do the "min rqts for a profile" differ from "the rqts of the profile"?

minus

CC,
(or Sec)

3.3

If profiles are chosen, address rules for profiles. [P2]

minus

plus

CC,
(or Sec)

3.4

If modules are chosen, indicate any mandatory conditions or constraints on their usage. [P1]

minus

CC,
(or Sec)

3.5

If profiles, modules or levels are used, define their relationships and interaction with other dimensions of variability. [P2]

minus

Guideline 4. Identify the relation between deprecated features and conformance.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

4.1

Identify each deprecated feature. [P1]

plus

Body
& CC

4.2

For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation. [P1]

plus

CC
(for policy)

4.3

If deprecated features exist, define their relationships and interaction with other dimensions of variability. [P2]

4.4

Include an explanation for the deprecation. [P3]

minus,
unr

Body

4.5

Include examples to illustrate how to avoid using deprecated features. [P3]

Body

4.6

Identify each obsolete feature. [P3]

Guideline 5. Define discretionary items.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

5.1

State the circumstances for when discretionary items are allowed. [P1]

CP statement & ConfReqs seem unrelated.

minus,
unr

okay

CC &
Body

E, S

5.2

For implementation dependent values or features, address the allowable differences between implementations. [P1]

okay,
unr (1st)

Body

5.3

Indicate any constraints on the number of choices/options that can be implemented for discretionary items. [P2]

Is 2nd ConfReq redundant w/ CP5.5?

minus,
unr

minus

Body

S, E

5.4

Promote consistent handling of discretionary choices. [P2]

ConfReqs (all about "terminology") seem unrelated to parts of rationale (same results under same conditions) and CP statement.
CP statement is about "choices" but ConfReqs is about "items".
Substantive purpose post-Tokyo -- collective specification for related *choices* has been lost and only trivial purposes remain.

???

5.5

If discretionary items are used, define their relationships and interaction with other dimensions of variability. [P2]

The ConfReqs 2nd pgph is not about other DoV, it is about inter-relationship of choices within the Discretionary Items DoV.
The ConfReqs 2nd pgph seems to belong to CP5.4 (it is the missing post-Tokyo part?).
The "Rationale" is not a rationale, and is entirely focused on Use Cases, which aren't mentioned in CP statement or ConfReqs.

Guideline 6. Allow extensions or not

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

6.1

Indicate if the specification is extensible, and if extensions are allowed, define their scope and constraints. [P1]

6.2

Prevent extensions from contradicting the specification. [P1]

6.3

Define a uniform mechanism to create an extension. [P3]

minus

CC
(or Body?)

6.4

Require that extensions be published. [P3]

minus

6.5

Mitigate the impact of extensions on interoperability. [P3]

Is last sentence of ConfReqs true? Why wouldn't this CP apply, e.g., to extensions to an API standard?

E, S

6.6

If extensions are allowed, define their relationships and interaction with other dimensions of variability. [P2]

Guideline 7. Clearly identify conformance requirements.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

7.1

Use conformance key words. [P1]

minus

Body &
CC-subsec

7.2

Distinguish normative and informative content. [P1]

minus

Body &
CC (sec?)

7.3

Use consistent terminology. [P1]

Body

7.4

Provide a fast way to find conformance information. [P2]

CC ,
ToC, ???

7.5

Make normative reference to specifications on which the current specification depends. [P1]

Body &
Refs

Guideline 8. Document the conformance policy.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

8.1

Specify any universal requirements for minimum functionality. [P2]

plus

Sec(!) ,
in CC

8.2

If special conformance concepts are used, include a definition in the specification. [P1]

minus

8.3

Justify any usage of a dimension of variability. [P1]

8.4

Include a conformance section. [P1]

minus

Sec(!) ,
in CC

8.5

Identify and define all conformance designations. [P1]

minus

Guideline 9. Specify how to make conformance claims.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

9.1

Provide specific wording of the claim. [P3]

9.2

Provide a conformance disclaimer. [P3]

minus,
unr ???

9.3

Impose no restrictions about who can make a claim or where claims can be published. [P1]

minus

9.4

Provide an Implementation Conformance Statement proforma. [P3]

minus,
unr

CC,
external

9.5

Require the ICS be completed as part of the conformance claim. [P3]

Guideline 10. Provide test assertions.

Nbr

Checkpoint

ET/E

ET/T

Where

Template
potential

Cmmts

10.1

Provide test assertions. [P2]

ConfReqs question: if TAs were marked-up & styled in-line in text, does that comprise a "list"?

minus

Body,
or CC/ext

E, S

10.2

Provide a mapping between the specification and the test assertions list. [P2]

n/a,
or CC/ext

References

QAF-SPEC

QA Framework: Specification Guidelines, D. Hazaël-Massieux, L. Rosenthal, L. Henderson, Eds., W3C Candidate Recommendation , available at http://www.w3.org/TR/2003/CR-qaframe-spec-20031110/ .

SPEC-EXTECH

QA Framework: Specification Examples & Techniques, Karl Dubost, Lofton Henderson, Lynne Rosenthal, Eds., current companion to [QAF-SPEC], available at http://www.w3.org/QA/WG/2003/09/qaframe-spec-extech-20030912 .

ED-COMMENTS

Email/text file containing SpecGL editorial comments, in email archives at http://lists.w3.org/Archives/Public/www-qa-wg/2003Dec/0042.html .

ISSUES

...tbd... linked ext. file? or issues embedded inline?

Template analysis for "QA Framework: Specification Guidelines"

Introduction

Analysis data

Key to Analysis Table

Analysis table for potential templates

References