W3C

Test assertions for "QA Framework: Specification Guidelines"

This version:
http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/qaframe-spec-ta
This document is an appendix to:
http://www.w3.org/TR/2003/WD-qaframe-spec-20030912/
Latest version of "QA Framework: Specification Guidelines":
http://www.w3.org/TR/qaframe-spec/
Editors:
Mark Skall
Contributors:
See Acknowledgments in QA Framework: Specification Guidelines.

Test Assertions

Guideline 1. Define Scope.

Checkpoints

Checkpoint 1.1. [Priority 1]

The first section of the specification contains a clause entitled "scope" AND enumerates the subject matter of the specification.

Checkpoint 1.2.Illustrate scope[Priority 2]

The specification contains use cases to illustrate what is in scope OR the specification contains examples to illustrate what is in scope OR the specification contains use cases AND examples to illustrate what is in scope.

Checkpoint 1.3.Illustrate functional details[Priority 2]

For at least one functional requirement, concept or behaviour within the specification, the specification contains a corresponding specific example to clarify a complex concept, behavior or interaction.

Guideline 2. Identify what needs to conform and how.

Checkpoints

Checkpoint 2.1.Identify classes of product.[Priority 1]

The specification contains a list of classes of products pertaining to this specification AND the specification defines conformance requirements for each product listed. If the list is NOT a proper subset of the classes defined in this QA Framework Guideline THEN the specification defines and describes the list of classes of products.

Checkpoint 2.2.For each class of product, define the conformance requirements.[Priority 1]

For each class of product identified in checkpoint 2.1, the specification contains a definition of conformance for each one.

Checkpoint 2.3.Identify the applicable specification categories[Priority 3]

In its introductory section, the specification contains a list of specification categories pertaining to this specification. If the list is NOT a proper subset of the categories defined in this QA Framework Guideline THEN the specification defines and describes the list of specification categories.

Checkpoint 2.4.If there are several classes of products, define their relationships and interaction with other dimensions of variability.[Priority 2]

If the specification contains AT LEAST two classes of products THEN the specification contains a section describing the relations and interactions between these classes of products and other dimensions of variability. If the specification contains ONLY ONE class of product THEN this checkpoint is not applicable.

Guideline 3. Address the use of profiles, modules and levels to divide the technology.

Checkpoints

Checkpoint 3.1.Indicate whether or not the use of profiles is mandatory for conformance.[Priority 1]

If profiles are used by the specification THEN, for each identified class of product in the specification,he specification specifies EITHER that conformance is only defined within the context of prof the specification specifies EITHER that conformance is only defined within the context of profiles OR that conforming products may exist without respect to profiles AND describes any additional conditions associated with the profiles(s) for these class of products. If profiles are NOT used by this specification THEN this checkpoint is not applicable.

Checkpoint 3.2.If profiles are chosen, define any minimal requirements.[Priority 2]

For each identified class of product the specification specifies the minimum required features supported by each profile for that class. If profiles are not used THEN this checkpoint is not applicable.

Checkpoint 3.3.If profiles are chosen, address rules for profiles.[Priority 2]

If the specification allows derived profiles THEN the specification contains testable requirements for these derived profiles. If the specification does NOT allow derived profiles THEN this checkpoint is not applicable.

Checkpoint 3.4.If modules are chosen, indicate any mandatory conditions or constraints on their usage.[Priority 1]

If the specification supports modules THEN the specification contains a section documenting all mandatory conditions or constraints associated with the use of modules.

Checkpoint 3.5.If profiles, modules or levels are used, define their relationships and interaction with other dimensions of variability.[Priority 2]

If the specification allow profiles OR the specification allow modules OR the specification allows levels THEN the specification documents all relationships and interactions among the profiles, modules and/or levels used with any other dimension of variability. If profiles are not used AND modules are not used AND levels are not used THEN the checkpoint is not applicable.

Guideline 4. Identify the relation between deprecated features and conformance.

Checkpoints

Checkpoint 4.1.Identify each deprecated feature.[Priority 1]

If deprecated features exist THEN the specification contains a normative section that documents each deprecated feature OR the specification contains a normative section containing a list of links to where the features appear in the document. If deprecated features are NOT used THEN this checkpoint is not applicable.

Checkpoint 4.2.For each class of product, specify the degree of support required for each deprecated feature and the conformance consequences of the deprecation.[Priority 1]

If deprecated features exist THEN, for each class of product, the specification specifies the degree of support required for each deprecated feature AND specifies the conformance consequences of the deprecation.

Checkpoint 4.3.If deprecated features exist, define their relationships and interaction with other dimensions of variability.[Priority 2]

If deprecated features exist THEN, for each deprecated feature, the specification EITHER states that the deprecated feature is independent of all other dimensions of variability OR documents the relationship between the deprecated feature and each of the other DOV.

Checkpoint 4.4.Include an explanation for the deprecation.[Priority 3]

If deprecated features exist THEN, for each deprecated feature, the specification documents the feature AND includes a rationale for deprecation. If deprecated features do not exist THEN this checkpoint is not applicable,

Checkpoint 4.5.Include examples to illustrate how to avoid using deprecated features.[Priority 3]

If deprecated features exist THEN, for each deprecated feature, the specification provides an example of how an alternative approach done in place of the deprecated feature will produce the same objective. If deprecated features do not exist THEN this checkpoint is not applicable.

Checkpoint 4.6.Identify each obsolete feature.[Priority 3]

If obsolete features exist THEN, for each obsolete feature, the specification provides documentation on the feature. If obsolete features do not exist THEN this checkpoint is not applicable.

Guideline 5. Define discretionary items.

Checkpoints

Checkpoint 5.1.State the circumstances for when discretionary items are allowed[Priority 1]

If discretionary items exist THEN the specification labels each discretionary item as such AND indicates the rationale for each discretionary item. If discretionary items do not exist THEN this checkpoint is not applicable.

Checkpoint 5.2.For implementation dependent values or features, address the allowable differences between implementations[Priority 1]

If implementation dependent values or features exist THEN, for each implementation dependent value or feature, the specification describes any permitted variations or constraints for how the value or feature is realized by implementations.

Checkpoint 5.3.Indicate any constraints on the number of choices/options that can be implemented for discretionary choices[Priority 2]

If discretionary choices exist THEN, for each discretionary choice, the specification indicates whether zero, one or several of the choices/options are allowed to be implemented. If the allowable number is dependent on other dimensions of variability, THEN the specification states the dependencies. If discretionary choices do not exist THEN this checkpoint does not apply.

Checkpoint 5.4.Promote consistent handling of discretionary choices.[Priority 2]

If discretionary choices exist THEN the specification identifies policies for handling them.

Checkpoint 5.5.If discretionary items are used, define their relationships and interaction with other dimensions of variability.[Priority 2]

If discretionary items exist THEN the specification defines the relationship and interaction among discretionary items and all the other DOV.

Guideline 6. Allow extensions or not

Checkpoints

Checkpoint 6.1.Indicate if the specification is extensible, and if extensions are allowed, define their scope and constraints.[Priority 1]

The specification states whether or not extensions are allowed. If extensions are allowed THEN the specification states the conditions under which these extensions are allowed and disallowed AND the scope of the extensions AND their effect on conformance claims AND any limitations or restrictions on the use of the extension AND documents the rationale for allowing extensions by referencing use cases and/or project requirements.

Checkpoint 6.2.Prevent extensions from contradicting the specification.[Priority 1]

If extensions are allowed THEN the specification explicitly states that "extensions cannot negate or change support for required functionality". If extensions are not allowed THEN this checkpoint is not applicable.

Checkpoint 6.3.Define a uniform mechanism to create an extension[Priority 3]

If extensions are allowed THEN the specification provides a uniform way to define that extensibility is being invoked and provides syntax to be used to indicate the extension. If extensions are not allowed THEN this checkpoint is not applicable,

Checkpoint 6.4.Require that extensions be published.[Priority 3]

If extensions are allowed THEN the specification explicitly requires that the syntax AND the semantics of the extension be publicly documented.

Checkpoint 6.5.Require that implementations provide interoperable alternatives to extensions[Priority 3]

If extensions are allowed AND "producer of content" is one of the specification's classes of products THEN the conformance requirements within the specification contains a requirement that implementations provide a mode under which they produce only conforming content. ELSE the checkpoint is not applicable.

Checkpoint 6.6.If extensions are allowed, define their relationships and interaction with other dimensions of variability[Priority 2]

If extensions are allowed THEN the specification addresses and discusses the relations and interactions between extensions and all the other DoV.

Guideline 7. Clearly identify conformance requirements.

Checkpoints

Checkpoint 7.1.Use conformance key words.[Priority 1]

The specification identifies conformance requirements EITHER by using the RFC 2119 keywords to denote if the requirements are mandatory, recommended or optional OR (the specification defines an alternative way that conformance requirements are identified AND explicitly states why the RFC 2119 keywords were not used).

Checkpoint 7.2.Distinguish normative and informative content.[Priority 1]

The specification clearly distinguishes between normative (prescriptive) content and informative (informational) content. The specification distinguishes between normative and informative examples, illustrations and use cases as well as text.

Checkpoint 7.3.Use consistent terminology.[Priority 1]

If two or more provisions in the specification are identical THEN the specification uses identical wording to express them AND if two or more provisions in the specification are analogous THEN the specification uses analogous wording to express them.

Checkpoint 7.4.Provide a fast way to find conformance information[Priority 2]

The specification contains one or more navigation mechanisms that allows the reader to locate by direct access, all conformance-related information that is relevant to the specification. The mechanism specified locates the conformance section AND unambiguous statements about those DoV that the specification employs, from among the seven defined in this specification AND requirements for conformance claims.

Checkpoint 7.5.Make normative reference to specifications on which the current specification depends[Priority 1]

If the specification depends on other specifications THEN the specification contains normative references to those other specifications AND describes the relationship between the specifications and any identified conformance implications.

Guideline 8. Document the conformance policy.

Checkpoints

Checkpoint 8.1. Specify any universal requirements for minimum functionality. [Priority 2]

The specification contains a normative section enumerating the minimal requirements that apply across conforming products of a class. ELSE there are no requirements for minimum functionality.

Checkpoint 8.2.If special conformance concepts are used, include a definition in the specification.[Priority 1]

The specification contains a definition of new conformance concepts. ELSE the specification does not use any conformance concepts not defined in SpecGL.

Checkpoint 8.3.Justify any usage of a dimension of variability[Priority 1]

For each DOV the specification uses, the specification justifies the reason for its usage. ELSE no DOVs are used by the specification.

Checkpoint 8.4.Include a conformance section.[Priority 1]

The specification contains a conformance section AND the conformance section documents the specification's conformance policy.

Checkpoint 8.5.Identify and define all conformance designations.[Priority 1]

If a single, monolithic (strict) conformance definition is not used THEN the specification identifies and defines all conformance designations (e.g., degrees, levels or categories of conformance).

Guideline 9. Specify how to make conformance claims.

Checkpoints

Checkpoint 9.1.Provide specific wording of the claim.[Priority 3]

The specification contains specific wording relating to an implementation's claim of conformance. The specification includes, in this wording for the claim, the specification name AND the version, the conformance level satisfied AND information about the subject that which is claiming conformance AND the date of the claim.

Checkpoint 9.2.Provide a conformance disclaimer.[Priority 3]

The specification contains a conformance disclaimer that sends the message that a claim of conformanmance is no guarantee that the claimant is 100% conforming with the specification.

Checkpoint 9.3.Impose no restrictions about who can make a claim or where claims can be published.[Priority 1]

The specification DOES NOT contain any wording that restricts who can make a claim AND the specification DOES NOT contain any wording about where claims can be published.

Checkpoint 9.4.Provide an Implementation Conformance Statement pro forma.[Priority 3]

The specification contains an Implementation Conformance Statement OR the specification contains a discussion of why the ICS is not needed.

Checkpoint 9.5.Require the ICS be completed as part of the conformance claim.[Priority 3]

If the specification contains an ICS THEN the ICS is referenced within the specification's conformance claim wording.

Guideline 10. Provide test assertions.

Checkpoints

Checkpoint 10.1.Provide test assertions[Priority 2]

The specification contains a normative list of test assertions OR the specification references a separate document containing the test assertions. The test assertions cover all the requirements in the specification.

Checkpoint 10.2.Provide a mapping between the specification and the test assertions list[Priority 2]

For each test assertion, the specification maps that assertion to the specific requirement(s) in the specification tested by the assertion.