Editor: Daniel Dardailler, W3C.
Date of original release: April 2003.
Note: as of June 2007: W3C hasn't started a Certification activity.
This document is a study of the whereabouts, pros and cons of a W3C Certification activity. No decision has yet been made with regard to starting such an activity. This document still provides a basis for discussing options.
Note that even though the term activity is used in this document, it doesn't really carry the same formalism attached to the word Activity in the W3C process document.
This new activity would be closely integrated with the existing W3C QA activity and aims to create and manage a certification program for potentially: Web content, Web applications, and Web developers.
The basic premises behind this program is that:
The benefit foreseen by certification is that this will create a market need for better recognition of quality of Web resources, quality sought by the user community, and that it will therefore improve the quality of the Web and the demand of good products by users.
The risks on the other hand are that we spend precious resources on such an initiative, for no good result, and worse, that we end up alienating part of the industry or the Web community for wanting to play what could be seen as a new police or a commercial role for the W3C.
A certification is an acknowledgement by a certifying authority (the organism issuing a certificate or branding) that a set of criteria has been met.
It is different from conformance, which defines an ability to meet a set of requirements defined by a specification.
Certification in itself makes no claim about a given specification, and its relationship to a specification is transitive in nature (e.g. a product is certified to pass a set of SVG tests, which themselves defines - or not - a level of conformance to the SVG specs).
It is an expression of trust and therefore a form of liability, while a conformance claim is more of a technical assertion.
Whether or not the tests are run by the party seeking certification, by W3C, or by a third party acting on its behalf, the novel idea behind this program is that W3C would engage its name and brand for things that it hasn't directly produced (e.g. software products on the market, Web development agencies).
It is therefore very important that a good control over this new W3C liability be recognized and properly organized within the W3C and the QA activity, involving the membership and the Web community. Given that the cost of the branding program itself, i.e. the MarComm around it, wil probably be very resource intensive, it is also very important not to underestimate them in any future planning.
The Quality Assurance (QA) Activity at W3C has a dual focus: to solidify and extend current quality practices, and to educate by sharing our understanding of coordination, certification, funding, and tracking of the quality of products and services related to W3C technologies. The mission of the QA Team is to improve the quality of W3C specification implementation in the field.
The QA activity is mainly focused toward the inreach, that is, the quality of the W3C own working groups processes and deliverables (e.g. specification).
Certification, on the other hand, is partly about judging the quality of other people's work.
The closest to something resembling today that at W3C is the submission process whereby W3C express its opinion on the technical work of its members doen outside W3C. This is a limited comparison as it concerns only specifications (not products or services) and W3C, when accepting a submission, only asserts that the work is good enough for becoming a W3C note.
But this constitutes nevertheless some kind of approval stamp we're giving to external work that we may want to draw experience from (in terms of liability and public reaction, press coverage, etc).
The QA activity proposal mentions certification as a topic of the QA Interest Group but it is specifically marked as currentlyout-of-scope in the QA IG charter and as something that needs more thinking and shared understanding.
So if we move forward with that, we do have to ask our membership, since it's not in any existing activity scope.
However, in recent W3C meetings (TP March 2003) of the QA chairs, staff, but also the AB, and some WGs, it appeared that a successful certification program could have the following benefits:
It was also recognized that there are associated risks and difficulties attached to such a new endeavor:
But the potentially high payoff calls for exploring this further and propose a plan to the W3C members at some point, once we have reached internal consensus on such a plan.
We need to study both the different classes of objects being certified and the different methods for certifying them.
On the objects of certification.
The QA activity has released a taxonomy of test materials which presents various categories of test materials.
Using that list as a starting point, we see three kinds of potential certification activities:
Note: certification of documentation, like HTML books. could fall into services, if we consider that the author/editor of the doc is being certified. Or it could fall onto products, e.g. a book is a product. Or we could have products splitted into programs (products above) and docs.
On the certification methods.
At some point, someone has to run a series of tests over something (the objects above).
At some other point, the W3C name is attached to this something as a guarantee that the tests were successful.
Between those two moments, W3C has to make sure its brand is not misused or abused.
In decreasing order of "assurance" and "cost for W3C", the tests could be run:
For each of these categories of objects and methods, we need to examine what a particular certification program would mean (what are the nature of the tests being saught and the exact objects of the certification, e.g. SVG or WCAG) and could look like (difficulties, existing test framework, etc).
See the decision matrix below.
The relationship between certification and testing is of course very strong, as the first cannot usually be obtained without the availability of the second, so it is important to know the status oftest development at W3C and availibility in each category.
One of the main goals of the QA activity is to have each W3C WG develops and being responsible for test materials that support their technology.
In order to achieve that goal, the W3C QA activity is developing a framework of operational, specification and tests development guidelines for promoting and facilitating these WG quality practices.
The QA Matrix lists the current achievement of WGs in terms of availibility of validation tools and test suites in their area of work.
It is a fair statement to say that we're far from achieving this goal today (of having each WG producing tests of quality), and many W3C groups do not have any test for checking either the content or the products implementing their specification.
This in itself is an indication that whatever we start now will not be comprehensive across W3C activities but will have to work incrementally over time.
This also means that we may want to wait another year to get better test materials available from the WG to embark into certification.
On the other hand, the QA Matrix shows some promising result that makes it possible to start such a program given the availibility of good validation and test suites programs in some W3C activities.
Let's now examine each potential certification activities in turn.
The tests used in that category are usually called validators or evaluation programs.
The objects of the certification are Web pages or Web sites.
We already have several test tools for various W3C specifications, like the HTML validator or the XML Schema validator.
W3C even provides logos for HTML, CSS and WAI WCAG compliance, but this is not a certification approach as it is clearly indicated that content providers are solely responsible for the use of our logos (it can't even be called self-certification, but more self-evaluation).
The real difficulty in that category lies not so much in the development or the running of the tests but in the amount of potential things to be tested: the Web itself!
If we start a certification program for Web content, one may wonder how this can scale, or more to the point, how does the return on investment compare to other forms of certification (e.g. authoring tools)?
Maybe the self-evaluation approach we've taken so far is the best one we can hope to achieve, even it it is sometimes abused (Web pages carry our logo without being valid or accessible at the level they claim, often for versioning reasons).
A related case of Web content certification program is the notion of gallery of Accessible Web Content that the WAI EO group has been working on. This effort already shows the range of difficulty associated with Web content related claims per se: need for a human liaison responsible for the content, need for tracking the site changes (from valid to not valid or the opposite), need for a selection process, etc.
The tests used in that category are usually called test suites, TS for short.
The objects of the certification are Web applications like user agents (e.g. browsers), authoring tools (e.g. HTML editors), or web servers.
There is existing market of such certification already for platform integration (e.g. Certified for Windows) and one can imagine that a W3C Certified for the Open Web (or some such attractive name) program would appeal to the product vendors as well.
The issues there lie in the developement of objective and performant test suites for, in particuliar, authoring tools, which cover one of the most important aspect of the certification chain. Indeed, if the tools that are used to generate Web content improves, Web Content itself improves widely.
Some of our most important specification (e.g. HTML) are also difficult to test as their semantics are difficult to define in the specs and are as a result sometimes ill-defined (and therefore we do not have good test for them).
Each sub-category of tools mentioned above has its own set of specific difficulties, both technical and logistics, that each deserve a plan description.
The tests used in that category are usually called exams, or training sessions, and are both technical and process oriented. They are used to recognize a human or a human structure qualification, not a technology like the other two above.
The objects of the certification are people (Web developers), or organizations (e.g. Web design agencies).
We do not have test support for that today at W3C, but we have a some expertise in judging the production made by Web designers wrt to applying our standards, and we have some good base materials (slides, tutorials).
There is a large existing market of Certified Engineers for various products and systems, and we could create a Certified W3C Designers program that follow the same model used today for proprietary systems.
We would have to create several sets of educational and checking materials and a framework for passing the tests (e.g. exams) or/and certifying the quality of the internal process used in relationg to Web design.
Another approach would be the development of a Good Web Design Practices document that people would agree to comply with (process wise) and that we (or a third party on our behalf) could audit on-demand, for a fee, much like the ISO9000 way of doing thing.
This is potentially as area that can bring a lot of visibility to W3C and the importance of standard development work.
Here's a matrix that gives some insight on the pros (+) and cons (-) of each classes of objects and methods and for each couple, what we should pay attention to.
|
Web content
+ that's the real thing! - too many pages to check + can be automatized for things like markup validation |
Software
+ good lever effect for Web content - still lots of programs - manual testing often required |
Providers
+ good level effect as well - lots of people! + good promotion for W3C brand |
||||
| W3C staff (potentially with help from Web
community)
+ high quality testing + low external coordination costs - high running test costs + no (re)checking since done by us |
. interesting concept of gallery
. do something special for W3C members' pages for visibility? . enforce a bit more our markup logo licence, e.g. use automatic checking to correct abuse |
. Some WGs (ATAG, SMIL, SVG, etc) have produced self-assess tables
- publication of results is a liability, even legally forbiden sometimes |
. we could gather our curricula, tutorials, and have more
developped.
. staff could do training for trainers (maximized level) |
||||
| Third party
+ more professional approach - require more trust and financial relationship - high startup cost but better long-term business + limited checking |
. watch EuroAccessibility program for WCAG certification, which is a kind of third party certif for content | . would need to be trained to our (various) test harness
. could do the (spot) checking in the case of self-certification |
. could run courses and exams
. could check if production of certified providers is ok |
||||
| Companies
- limited trust + checking necessary + low cost of running test for W3C |
. continue to promote self-evaluation with no formal checking (e.g. valid HTML logo) | . need for easily checkable test result (e.g. EARL)
. need to handle versioning, multiple technologies (HTML, SVG, SMIL, etc) |
. people could pledge allegeance to a W3C Good practices chart. |
If it is organized as a new W3C Activity, we could create an IG for overall discussion and one WG per certification program: one for products, one for services, etc.
The initial duration could be 2 years.
We could start with a workshop to gather more detailed input from the certification industry, the Web industry, and the Web community in general (e.g. Open Source).
As for deliverables, the NIST white paper on Certification gives some very good input as what they should be, see http://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html#Products
At least one more person would be needed to run this thing on the W3C QA staff side (difficult to do with just volunteers/Members - need a dedicated W3C or some contracted resources).
Eventually, this may become self-financed as product vendors pay for certification stamp.
@@ Include here a report from the Business Scenario Workshop on Certification held in Washington DC on October 25th 2003 (OpenGroup conference)
We already had discussions with some of these groups, and we need more.
NIST, TheOpenGroup, UCC, Veritest (running the MS Windows certified program), our members doing QA/Certif, Open Source community, ETSI, ISO9000, ANSI, etc.
See this very good paper submitted by NIST to W3C in 2002: http://www.w3.org/QA/2002/01/Note-qa-certif-20020102.html
And this document as well, on certification, testing and branding at ConsortiumInfo.org.
Created: April 4th 2003 - by danield