Postscript on Certification
This postscript summarises the presentation given by Andrew Thackrah on 4 April 2001 at the W3C QA workshop at NIST, Gaithersburg, MD.
1. Introduction
This presentation is a discussion of the process of certification. The Open Group's process for the certification of WAP clients is used as a model since the required testing strategy is XML-based content testing. This has obvious relevance to W3C testing needs. We have to consider whether the proposed QA activity should include any kind of certification process.
This presentation is not an attempt to 'sell' certification. It is intended to be an objective discussion of the issues that must be considered in any certification program. Even if a W3C QA activity advises against the use of certification programs it is likely that some of the methods involved will be of use in helping to promote the adoption of new technologies.
Where I say "vendor" in this document I refer to creators of products that implement a particular technology whether or not it is commercial. Similarly "Buyer" means any group that buys, adopts or otherwise acquires a product or `solution'.
Nore that the W3C already has branding systems in place such as the HTML validator which earns a conforming page the right to use a logo. This is a simple form of automated self-certification.
2. The Open Group
The Open Group is a computer industry consortium of over 300 members, mostly vendors and buyers of information
technology and software. The company is a merger of OSF and X/Open and is run as a not-for-profit organisation. The group is concerned with the promotion and adoption of open systems and standards. The Testing and Certification group is responsible for the operation of certification programs and the development of conformance test suites and other supporting tools.3. What is certification?
Certification is the business of evaluating a product against a standard. A product that meets certain requirements based on the standard may be awarded a certificate of conformance.
Two important things to note about this definition are that a reference standard against which the product is evaluated need not be a low level technical standard and that the product need not be fully conformant in order to be awarded a certificate. This is discussed below.
A product standard can be a high level document that references several normative specifications and provides numerous options and configuration details. This is preferable to directly referencing a technical specification in some cases as it maps more closely on to a product manager's and a buyer's idea of a product. Since the certification process is aimed at buyers and adopters it is worth developing a product standard in their language. A standard of this kind might be produced by an industry consortium representing a number of different vendors.
A certificate is usually accompanied by some kind of brand which acts as a trade mark. A certified product earns the right to carry this brand.
4. Why bother with certification?
Certification is one of the ways in which we can provide an assurance of quality. There are numerous methods available for QA which cover the entire range of activities surrounding technical development from specification, through standardisation to development and marketing. Some methods are applicable to particular activities, for example standards tracks with formal specification and review procedures suit specification. Software engineering processes such as versioning and source control suit the development of products (implementations of the specification). Certification is particularly relevant to the final stage - that of product release and marketing.
Certification then is an assurance that we give to buyers and end users. The principal goal of a certification program is to promote confidence in a technology. A range of certified products is intended to encourage buyers to adopt the technology. The methods of certification should help, not hinder the production of conforming technologies. The point made previously about the nature of a product standard is important here as this can form part of the collateral with which a product is marketed. Thus the product standand must be pitched at a level that is meaningful to buyers and users while still retaining the necessary technical rigour to act as a framework for developers.
5. Relationships and roles
The fact that certification is aimed at end users does not mean that the process precludes the involvement of standards authors and developers. Typically a certification agency works with an industry consortium that has been formed to promote an emerging technology. The consortium may create a brand around a set of conformance standards that are based on technical specifications issued by a standards body. Note that the specifications may be authored by representatives of vendor companies and that the same companies are likely to be represented in the industry consortium.
The certification agency must set-up a network of contacts from the various groups to handle interpretations and other problems.
The certification agency is in direct contact with the vendors who submit candidate products for certification and possibly also the cosumers who procure a branded product (this depends on who owns and promotes the brand).
The bottom line is that the certification agency decides whether a vendor's product satisifies the requirements of the industry consortium or not and provides arbitration in case of disagreement or extenuating circumstances.
6. How is it done? A sample certification process
The certification process described here is a simplified model of the process used for branding of 'phones and other mobile devices with the "WAP" logo (Wireless Application Protocol) which is a trademark of the WAP Forum.
ICS - Implementation Conformance Statement
A product developer approaches the certification authority with a request to submit a product for certification. They are asked to fill out an Implementation Conformance Statement (ICS). This statement declares all of the features (mandatory and optional) that they claim for the product. The reference for this is the product standard which lists all requirements. Sometimes the product standard is accompanied with (or replaced by) a Static Conformance Requirements document (SCR). The ICS is essentially a checklist version of the SCR in which the product vendor checks each supported function.
Testing
The candidate system must now be submitted to a range of tests. This requires a test suite and a site at which to run the tests. The exact sub-set of tests required is determined by the ICS. The test suite goes through a QA process of its own in order to gain approval for use as a certification tool (not trivial!). The testing may take place in a trusted 3rd party lab or be directly hosted by the certification agent. Testing could even be carried about by the party that has submittted the product. The end result of this stage is an official test report. Regardless of the details of the test campaign this test report must be signed-off as a binding document and delivered to the certification agent for evaluation.
Reference pool
A pool of reference systems is made available for the testing program. The candidate product must be tested against several references (3 in the case of WAP certification). The reference systems are submitted by members of the vendor-led consortium that is driving the certification program. It is worth noting that these reference systems are not "reference implementations" in the traditional sense. There is no baseline product in the certification program. Rather, these reference systems provide a controlled environment (sometimes called a `sandbox') in which to conduct the test campaign. This is particularly important for distributed web-based testing as there can be several agents mediating between client and server (proxies, caches, gateways and other portals are the main culprits). Since these intermediate agents are not part of the testing campaign it is important to prevent them from biasing the results so the the reference pool provides a combination of `clean' systems with which to conduct the testing.
Forcing the candidate product to test against more than one such system is part of the strategy for ensuring interoperabilty as as well as conformance.
Evaluation
The submitted test report is compared against a conformance standard. The decision of whether to award a certificate is based on this evaluation. Certification need not be a black and white issue. In the real world there are many conditions that can affect the certification process. A well designed process should have means to deal with bugs found in the test suite during testing, disagreement over interpretation of normative specifications and the ability to grant temporary or permanent waivers for a variety of reasons. This kind of flexibility is seen as an important feature in the eyes of product developers: overly rigid or simplistic certification procedures are not well regarded and this can deter submission for certification and ultimately hinder the (conformant) adoption of the technology.
Appeals
A developer submitting a product for certification must be allowed to appeal before, during and after the certification request (see evaluation above). The process requires a clear line of adjudication. There may be several groups involved in the arbitration and adjudication process. For example a test suite maintenance authority may be established to deal with test suite issues, representatives from a standards body may deal with specification interpretations and so on. Note that a product standard may refer to normative specifications that are outside of the control of the consortium of certification agency. In this case it is essential that the process has a way of dealing with problems that arise from errors in these specifcations.
The certification agency should aim to provide a single point of contact between all referees and the client (the developer with candidate product).
Remember that different groups operate on different timescales. A product developer may have much shorter release timescale than that of a change request process to a well established 3rd party specification. The ability to handle these different timescales, to provide satisfactory temporary solutions and to track the status of problem resolutions over potentially long (1 year+) timescales is necessary.
7. Respect - creating trust in a brand
If the end product of certification is the right to use a brand on a product then steps must be taken to ensure that the brand carries the right connotations for potential buyers. A brand will earn or lose respect through the performance of the products that bear it. To get this right, the QA process of certification requires internal QA processes itself. Certain measures can be taken to proctect the vendors from embarrasment or early disclosure of confidential information. It is useful for the certification agency to prepare a policy of vendor neutrality and confidentiality surrounding test campaigns and certificate awards.
During the creation of a brand thought must be given to its promotion and once in use, thought must be given to the protection of the brand and the claims that it makes. A weak or abused brand damages the reputation of the related technology. Promotion and protection of a brand may be the role of the industry consortium that created it.
8. Summary
I have described some of the functions and roles that are involved in a certification process. Certification is a QA process that is aimed at buyers with the intention of encouraging adoption of a technology by creating a badge of trust (a brand) for certified implementations of the technology.
The brand can represent a conformance to a standard which may be a low level specification or a higher-level product standard.
Once created, a brand (and the buyers that trust it) should be protected from abuse. To ease problems a comprehensive problem resolution process should be defined - this requires an understanding of the various roles in the "specify - standardize - develop - certify - deliver" chain.
Certification is not a simple yes-no process. It must be sufficiently flexible to reflect the grey areas of the real world.