From: Rob Lanphier RealNetworks' position on the W3C Quality Assurance Activity Rob Lanphier -- Program Manager, Open Standards Curtis Reynolds -- Software Test Lead, Media Systems Client Technology March 16, 2001 The W3C was originally formed to create a process where standards keep pace with the industry. The original fear was that in the escalation of the "browser wars" that de facto standards driven by short-term needs of vendors trying to lock up the market would replace deep architectural thinking on behalf of the web community. Thanks in no small part to the efforts of the W3C, the web is still a thriving environment where innovation builds on a sound architectural foundation, and the world looks to the W3C to guide the coordinated development of one web. Now there's a growing perception that the W3C is getting out in front of itself. The effort to get out in front of the vendors may have been a little too successful; there are many W3C Recommendations that have yet to be fully implemented by anyone, including the vendors that participated in the process of creating them. One could say the W3C is a Formula One racecar with faulty brakes. If one is to have confidence in traveling at high speed, one must also be confident that the brakes work. Still, our understanding is that the W3C may be tackling the new Quality Assurance activity with some trepidation. There have been vendors who have expressed skepticism that the W3C could approach this with the proper focus and determination to make this successful. They point to the many other things that the W3C needs to do as a consortium, and conclude that this is a problem best left for others more focused on the problem. Furthermore, the W3C doesn't want to be seen as "grading" members. However, the choice not to act is a choice in and of itself. Working groups have little guidance on how strongly to word requirements. Several W3C Recommendations have requirements that are difficult or impossible to meet, since those skilled at the art of verification are not involved at the early stages. The W3C has lots of experience at putting out recommendations, but not a lot at holding vendors to those recommendations, or creating recommendations that vendors feel compelled to comply with completely. One can hold out for other groups to form around solving these problems. However, in doing a minimal job at this through conformance logo programs and asserting common law trademarks on W3C Recommendation names, the W3C has staked out the conformance territory as its own. Furthermore, due to the limited visibility to the inner workings that non-members have, the only people that may be aware that the opportunity exists are individuals within the W3C Membership. The W3C should not stake out a position in the middle of road on this; either pursue this activity with determination and purpose, or clear the road and actively ensure that some other organization is in the position to succeed at this activity. Certainly, the W3C risks alienating some members. The current set of vendors joined without having such strict interoperability requirements, and thus, being "graded" may scare them away. However, it also may draw in new vendors. Those vendors might be more willing to tackle supporting a standard if they knew this certification and quality assurance help would be available. == Creating Economic Interest == The speed at which the W3C is pumping out recommendations creates a problem where it's not clear whether or not it's even possible to interoperate on recommendations until it's too late to change the specification. What's worse, anyone can claim compliance, and anyone can propose new features that make standards more complex, but there's no countervailing economic interest to make the standards less complex. There's no consequence for creating specifications that are impossible to adhere to. Working group participants are often glib about adding new features, and cynical about the process of actually achieving interoperability. Such attitudes are toxic to the standards building process, but are pervasive in the consortium. Much more than mere policy is needed here, rather a built-up infrastructure and interests to act as an effective opposing force to the proliferation of half-baked ideas and overly complex mechanisms. One can and should appeal to software vendors' community spirit in an effort to make sure that W3C Recommendations are useful documents they fully intend to follow to the letter. However, a market economy can be a powerful tool when business interests and community interests can be aligned. By giving W3C Recommendations "teeth", i.e. by creating economic consequences for parting with recommended practice, the W3C can align the needs of the community with business mandates. The W3C needs to make sure that the market recognizes properly certified implementations. It needs to be part of the specification process. Whether or not the W3C actually does the heavy lifting is another story; the consortium just needs to make sure that it occurs. == Building Experience in the W3C == The W3C needs to build expertise in creating tested and proven interoperabilty. A W3C Certification Mark would provide a focal point that's necessary to get everyone talking the same language. Rather than having vendors claim support for a W3C Recommendation, with varying claims on what that means, customers could come to understand that a W3C Certification Mark is the only sure way of knowing that meaningful interoperability is possible. At the W3C Plenary Meeting in February, 2001, an idea was put forward that part of the process could be to review specifications after a period of time, and allow for subtraction of features. It could be that after 18 months, a certifier review may turn up things that are badly designed, either by showing that things are not being implemented correctly, or by showing that they just aren't being implemented. The fact that a lot of "standards implementations" aren't really spec compliant in some way reflects poorly on the W3C. Many typical web designers will refuse to try using certain standards because the implementation is poor. The vendors involved spend a lot of time getting the standard out but then not a lot of time supporting it -- thus creating a situation where the specifications exist, but they aren't getting read. == Recommendations == In light of all these considerations, here's what we would like to propose: * QA Review Group that reviews specifications in the same spirit as the WAI and i18n groups. This group would review specifications for testability and completeness, and would submit guidelines for specification writers that could become part of the mandatory process for W3C Recommendations. The first work item for this working group would be a watertight definition of the exit criteria for Candidate Recommendation. Specifically, different definitions are needed for modules intended for incorporation in many specifications, and profiles which are intended to define document types. * Certification program - The W3C should have a standard process whereby potential certification authorities can apply to the W3C for approval to certify implementations of any given W3C Recommendation. An appeals process would need to ensure that the W3C has final appellate authority in disputes over compliance. A process whereby the W3C can revoke specific certification stamps as well as certification authorities would also need to be in place. Certification stamps should never be permanent, only for fixed durations (e.g. one year). * A built-in safeguard in the recommendation process whereby long-term experience with implementations of specifications is taken into account. There may need to be something stronger than "W3C Recommendation" which is a "W3C Certified Recommendation (CertRec)" that occurs no *sooner* than 18 months (give or take a few months) after the W3C Recommendation is approved. CertRecs could pull features from the preceding W3C Rec that don't have two certified implementations. This would hopefully have the following effects: - Vendors participating in the process that hope to get certified would be more diligent in making sure that only those features that make sense to implement would make it into the specification - More vendors would be motivated to get involved, and get involved sooner, to ensure that poorly designed features don't make it into the specifications - Vendors would be motivated to make sure they get certified, so that their pet features have higher odds of making it into the CertRec With these measures, we hope that the W3C can continue to improve the process for building one web, and making sure that the one web we have has the best possible design.