QA Framework: Specification Guidelines

Guideline 1. Include a conformance clause

A conformance clause is a part or collection of parts of a specification that defines the requirements, criteria, or conditions to be satisfied by an implementation or application in order to claim conformance. Typically the conformance clause is a high-level description of what is required of implementations and applications. The objective of the conformance clause and its related conformance statements is to provide clear and unambiguous statements, so that the reader knows what is required in order to claim conformance and what is optional.

Checkpoints

Checkpoint 1.1. Use conformance key words (RFC2119) consistently and appropriately [Priority 1]

The key words denote whether or not requirements are mandatory, optional, or suggested. Using these keywords helps to identify the testable statements in a specification.

Checkpoint 1.2. Create a separate conformance section [Priority 2]

Having the conformance clause exist as a separate section within the specification makes it is clearly identifiable, allowing a reader to find all conformance provisions from a single starting point.

Checkpoint 1.3. Generate a conformance clause entry in the table of contents [Priority 2]

Checkpoint 1.4. Label normative and informative sections [Priority 2]

Normative statements are the prescriptive parts of the specification whereas informative statements are for informational purposes and assist in the understanding or use of the specification. It is important that the reader be able to distinguish between normative and informative statements in order to know what is required to claim conformance and what is optional.

Checkpoint 1.5. Make normative reference to specifications on which the current specification depends. [Priority 1]

Often a specification is dependent on other specifications or portions of specifications. For example requiring the class of product called user agent, to be consistent with the XML 1.0 Recommendation and support CSS1. The inclusion or reference to related specifications can affect conformance of the current specification. To ensure clarity and understanding of these implications, provide a description of the relationship between the specifications and any conformance implications. Linking from the prose to the reference is described in the Style Manual [@@http://www.w3.org/2001/06/manual/#linking-within]

Guideline 2. Specify all flavors of conformance

A look at various W3C Technical Reports shows that the term 'conformance' is often qualified resulting in more than one type of conformance. The XML Recommendation [@@96 initial draft http://www.w3.org/TR/WD-xml-961114.html] was probably the first to do this by differentiating conforming XML processors into two classes: validating and non-validating. It is important to convey an understanding of what is meant by conformance and if there are various flavors, what is meant by each flavor. Examples of flavors include: unconditional conformance, conditional conformance [@@UAAG conform. clause] and strict conformance. Strict conformance is defined as conformance of an implementation that employs only the requirements of the specification and no more (e.g. no extensions). Flavors of conformance is different than conformance claims, conformance of a type of implementation (e.g., user agent, authoring tools) ......

Checkpoint 2.1. Define all flavors of conformance that are applicable to the specification [Priority 1]

Guideline 3. Identify what needs to conform and how

The conformance clause identifies the "class of products" (i.e., object of the claim) that will be developed, where "class of product" is an implementation, application, service and/or protocol (e.g., content, XML processor, user agent, authoring tool). In addition to identifying what conforms (i.e., class of products), is describing how conformance can be met. This would be a description of the conformance requirements, conditions and/or criteria for each class of product.

Checkpoints

Checkpoint 3.1. Identify all classes of product [Priority 1]

For example, within the SMIL 2.0 Language Profile, there are 2 classes of products: documents and basic user agents and within MathML there are output-compliant,authoring tools and input-compliant, rendering/reading tools.

Checkpoint 3.2. For each class of product, define the conformance requirements [Priority 1]

The conformance requirements indicate the conditions to be satisfied in order to claim conformance. In addition to specifying the requirements for claiming conformance, it may be appropriate to specify that which is not a requirement. It is likely that these conformance definitions will reference normative text within the specification or in other related specifications.

Checkpoint 3.3. Indicate minimal support requirements. [Priority 3]

Describe the minimum set of elements, attributes, etc., that are required to claim conformance for a specific module and/or class of product. This information is often presented in tabluar form (e.g., SMIL 2.0 section 2.4, table 5). If applicable, indicate that there are no minimum support requirements. This is helpful in providing the reader a full understanding of the conformance requirements, especially when some of the classes of products have minimal requirements whereas others do not (e.g., WebCGM??)

Guideline 4. Use modulization or profiling to group requirements

From the SMIL Recommendations - Modularization is an approach in which markup functionality is specified as a set of modules that contain semantically-related XML elements, attributes and attribute values. Profiling is a method for defining subsets of a specification by identifying the functionality, parameters, options, and/or implementation requirements necessary to satisfy the requirements of a particular community of users. In SMIL, it is the creation of an XML-based language through combining these modules, in order to provide the functionality required by a particular application. . Caution should be exercised - experience has shown that having too many modules/profiles/levels can inhibit interoperability as well as add to confusion in the marketplace.

Checkpoints

Checkpoint 4.1. Choose one or more of the following methods for grouping or dividing up the specification: a) modulization, b) profiling, c) none of the above [Priority 1]

It is possible for a specification to have modules and profiles. If choice (c) is selected, it should be clear to the reader that the specification does not contain modules, pr profiles. One method to ensure this clarity is to explicity state that none of these are supported.

Checkpoint 4.2. For a, and/or b, indicate whether its use is mandatory [Priority 1]

If there are additional conditions associated wtih a praticular module/profile, they should also be described. For example: any restrictions or constraints on the number or types of modules that can be implemented, if one and only one module/profile can be implemented at a time, or if specific modules can not be used in combination.

Checkpoint 4.3. For a, b, and/or c, indicate the conditions for claiming conformance [Priority 2]

Consider whether or not a claim of conformance to a praticular module/profile can include functionality or features of another module/profile. For example, can implementations that purport to conform to a specific module include functionality defined in another module?

Guideline 5. Specify how to make a conformance claims

A specification may differentiate conformance claims by designating different degrees of conformance in order to apply and group requirements according to modules, profiles, or priorities. When a conformance claim is linked to functionality, impact and/or incremental degrees of implementation, the term 'conformance level' is often used to indicate the varying degrees of conformance. Levels are often used to group functionality into nested subsets, ranging from minimal or core functionality to full or complete functionality. Typically, Level 1 is the minimal or core of the specification. Level 2 includes all of level 1 and also additional functionality. This nesting continues until level n, which consists of the entire specification.

Checkpoints

Checkpoint 5.1. Identify and define all conformance levels [Priority 1]

The naming convention used to label the conformance levels can provide useful information, such as imply incremental importance or difficulty. The WAI Guidelines and the QA Framework Guidelines use a 3-level conformance format (A, AA, AAA) to convey priorities based on a checkpoints impact. Other specifications use functionality to differentiate between conformance levels - the lowest level of conformance is defined by a core set or minimally supported features, the next level of conformance would include the core plus additionaly features, etc. until the entire specification was covered.

Checkpoint 5.2. Provide specific wording of the claim [Priority 3]

Checkpoint 5.3. Impose no restrictions about who can make a claim or where claims can be published [Priority 1]

Claimants (or relevant assuring parties) are solely responsible for the validity of their claims, keeping claims up to date, and proper use of the conformance icons. As of the publication of this document, W3C does not act as an assuring party, but it may do so in the future, or it may establish recommendations for assuring parties. Claimants are expected to modify or retract a claim if it may be demonstrated that the claim is not valid.

Guideline 6. Define discretionary behaviors

Discretionary items may be warranted because of environmental conditions (e.g., hardware limitations or software configuration, ro external systems), localtiy (e.g., time zone or language), optional choices providing flexibility of implementation, dependence on other specifications, etc. One type of discretionary item is implementation dependent values. Implementation dependent values are used when it is not possible to define the behavior or values of a function. Implementation dependent means that an implementation may determine the effect, rather than having the effect mandated by the specification. Details in a specification may deliberately be omitted (i.e., not specified), so as to provide freedom to adapt implementations to different environments and different requirements. In general this is not a recommended practice.

Checkpoints

Checkpoint 6.1. Explicity state the cases and conditions where discretion is allowed and/or expected. [Priority 2]

Checkpoint 6.2. Indicate implementation dependencies and where applicable address allowable differences between implementations. [Priority 1]

Examples of allowable differences that should be addressed include:

• implementation dependent ranges, data, minimum or maximum values,
• environmental resources (e.g., memory or disk limitations),
• enviornmental values (i.e., language and local settings).

Checkpoint 6.3. Describe alternative approaches and the conditions under which an implementation is considered to be conforming [Priority 1]

Specifications may describe several different ways to accomplish its operation (e.g., a choice of file formats, protocols, or encodings). In such a case, enumerate the approaches and specify if there are limitations on the number of approaches or combination of approaches that can be implemented. Some possible ways to define conformance when allowing alternative approaches include mandating that an implementation:

• implement only one approach,
• implement every approach,
• be allowed to implement none of the approaches.

Note that if the specification doesn't include the different approaches, this becomes an implemnetation detail irrelevant to conformance.

Checkpoint 6.4. Include a statement regarding consistent handling of a discretionary item within an implementation. [Priority 2]

The effect of a discretionary item should be consistent within a single implementation. For example, a brower's rendering of a XSL-FO should be the same for every invocation regardless of the document instance.

Guideline 7. Clarify the relation between depricated features and conformance

After the initial publication of a specification, specification developers may be considering the deprecation of a feature (i.e., element or attribute) defined in the specification. A deprecated feature is a feature whose use is discouraged becasue it has been outdated by newer constructs or is no longer viable. Deprecated features may become obsolete and no longer defined in future versions of the specification. Deprecated features warn implementers that the feature was a bad idea and it may be withdrawn in the future. Specification developers need to consider the affect of deprecation on all the classes of products that implement the specification (e.g., authoring tools, user agents) as well as the conformance consequences on each class of product. For the purpose of backward compatibility, it may be necessary to specify diffenent requirements for the support of deprecated features for each class of product. For example, authoring tools would not use the feature, but user agents continue to support it.

Checkpoint 7.1. Identify and clearly indicate each deprecated feature [Priority 1]

Checkpoint 7.2. For each class of product, specify the level of support required for each deprecated feature and the conformance consequences of the deprecation. [Priority 1]

Define what it means for a feature to be deprecated and how this affects conformance. For example, in MathML 1.x [@@http://www.w3.org/TR/MathML2/chapter7.html#interf_deprec], MathML-output-compliant, authoring tools may not generate MathML markup containing deprecated features; whereas MathML-input-compliant, rendering/reading tools must support deprecated features.

Checkpoint 7.3. Include an explanation for the deprecation. [Priority 3]

Providing the rationale for deprecating a feature is helpful in understanding the motiviation for the deprecation, the impact and consequences on current and future implementations, and perhaps insight into its eventually disappearance from the specification.

Checkpoint 7.4. Include examples to illustrate how to avoid using deprecated features [Priority 3]

Examples are helpful in providing alternatives or better ways to get the same results. By showing what can be done in place of the deprecated feature will help to get implementors to discontinue use of the deprecated feature.

Guideline 8. Allow extensions or NOT!

An extension to a specification is a mechanism to incorporate functionality beyond what is defined in the specification. Allowing extensions affects how conformance is defined as well as what conformance claims can be made. Care should be exerciesed in determining the extent to which extensions are allowed or not allowed. Since extensions can seriously compromise interoperability, specification writers should carefully consider whther extensions should be allowed.

Extensions may be private (often vendor specific) or may be public (a full description of the extension is public). Private extensions are usually truly private, i.e., valid for a specific implementation or are only known by prior agreement between implementations. Public extensions are extensions in which the syntax, semantics, identifiers, etc. are defined and published allowing anyone to implement the extended functionality.

Specification allow extensions for various reasons. Extensions allow implementers to include features that are in demand by their customers. Also, extensions, often times define new features that may migrate into future versions of the specifications. However, the use of extensions can have a severe negative impact on interoperability. Some methods for enabling extension have less impact on interoperability than other methods. For example, a specification that allows private extensions (e.g., proprietary) is more likely to impede interoperablity that a specification that requires extensions to be registered.

Checkpoints

Checkpoint 8.1. If extensions are disallowed, explicitly state it. [Priority 3]

If extensions are not allowed, then make sure its clear to the reader that not only are extensions not allowed, but the circumstances under which that are not allowed. The implementations of the specificaiton precisely implement the specification. This is strict conformance. Strict conformance is often imposed on applications or content (e.g., a software program or document instance). This prohibition of extensions could be applied to a specific profile or module, rather than to the entire specification.

Checkpoint 8.2. If extensions are allowed, explicity state it. [Priority 1]

State the conditions under which extensions are allowed, the applicability of the extensions, their affect on conformance claims, and any limitations or restrictions on the use of the extension.

Checkpoint 8.3. If extensions are allowed, make it clear that the extensions do not negate support for required functionality. [Priority 1]

An extension doesn't change the fact that an implementation needs to support all required functionality in the specifications exactly as specified; nor does it cause the non-conformance of functionality defined in the specification. The specification can include statements such as:

• Each implementation shall fully support all required functionality of the specification exactly as specified.
• Extensions shall not re-define semantics for existing functions,
• The use of extensions shall not contradict nor cause the non-conformance of functionality defined in the specification.

Checkpoint 8.4. If extensions are allowed, use a standard mechansim to define the extension [Priority 3]

One mechanism to allow extensions within a specification is to provide a standard way of defining the extension or a 'standard way of being non-standard'. This helps to ensure predictable handling of extensions, that is, its recognition as such and the appropriate actions (i.e., to ignore or to implement). The nature of the extension dictates the method for defining the extension. It may be possible to define a generic function or mechanism that indicates external functionality. This external function may take the form of an escape or control character or be an identify, which whenever invoked indicates an extension follows. Another method, especially when extending a list of numeric parameters is to use a scheme where positive values represent standardized values and negative values are reserved for private extensions.

Checkpoint 8.5. If extensions are allowed, register or publish them [Priority 3]

Reigstration is a procedure that allows extensions to be acknowledged and made available to the public. Registration provides for a degree of rigor and technical review for any proposed extension. Typically the WG would be responsible for processing the registration of an extension, thus ensuring adequate quality of a proposed extnsion and a technical description sufficient to be uniformly implementable. Often registered extensions may migrate into a later version of the specification.

Checkpoint 8.6. If extensions are allowed, require that implementations include a way to operate without the extension. [Priority 3]

If an implementation contains extensions, require it to have a mode under which the implemenation can be directed to produce only conformant files (documents) or to operate in a strictly conformant manner.

Guideline 9. Publish an Implementation Conformance Statement proforma

An Implementation Conformance Statement (ICS) or questionnaire is useful in clarifying and declaring optional functionality and discretionary behavior and values. The results of the ICS can be used to identify the subset of test cases from a conformance test suite that are applicable to the implementation to be tested. This will allow the implementation to be tested for conformance against only the relevant requirements. The ICS is also helpful in describing the expected interoperabliity to be achieved with other implementations or applications of the specifications.

Checkpoints

Checkpoint 9.1. Include an Implementation Conformance Statement proforma as part of the specification. [Priority 3]

If an ICS is included as part of the specification, indicate whether it is a normative or informative part of the specification.

Checkpoint 9.2. Require the ICS be completed as part of the conformance claim. [Priority 3]

An ICS provides specific information about the implementation and can be helpful in substantiating the conformance claim.

Guideline 10. Include test assertions

Some specifications include test assertions as part of the specification, and more should. A test assertion is a statement of behavior, action or condition that can be measured or tested. It is derived from the specification's requirements and bridges the gap between the narrative of the specification and the test cases. Each test assertion is an independent, complete, testable statement for requirements in the specification. Each test assertion results in one or more test cases. Multiple test assertions can bu combined to form a test case, in this case one tests multiple facets of a paricular behaviour. Including test assertions as part of the specification facilitates and promotes the development of test materials. Specific benefits include:

Checkpoints

Checkpoint 10.1. Detailed control over what parts of a specification an implementation supports, grouped by behaviour, technical aspect (method/interface) or othre conceptual grouping [Priority 3]

Checkpoint 10.2. Detailed reporting when testing an implementation using the test [Priority 3]

Checkpoint 10.3. Tag testable statements [Priority 3]

Guideline 11. Use granular grammars to author the specification

Using a W3C endorsed grammar language (DTD or XML Schema), one has control over the information conveyed in the specification. This allows for and facilitates more detailed reporting of test results as well as specification coverage.

Checkpoint 11.1. Use W3C endorsed grammar [needs to be further formalize, DD to look into what DTD/Schemas exist and are presently used] [Priority 3]

Checkpoint 11.2. Specify intended behaviour in the specification using markup [Priority 3]

Checkpoint 11.3. Supply prose description of intended behaviour together with each test assertion [Priority 3]