HTML5 Testing

One month ago, I wrote an article about our recent progress on testing. Among several items, I mentioned "The HTML test suite only contains 97 approved tests for the moment so don't draw too many conclusion from the result table. The number of tests needs to increase significantly if we want to test HTML5 properly". 135 tests later, it seems that people are trying to draw conclusions from the tests or from the results, including whether one browser or another is better.

An increase of 135 tests isn't meaningful. It's way far from making the results significant in fact. We'll need several dozens of thousands of tests to make those results indicative. Still, while working on the HTML5 specification, there is an advantage to figure out where we have interoperability issues already.

Sure enough, the test report page was misleading but like other items, that's work in progress. We'd like people who care about HTML5 interoperability to come and help us make it better. We simply fixed the page once we received input on it. Feel free to see the discussion on public-html-testsuite if you're wondering what's happening. You'll even see in the minutes of the calls who has been participating and who has not.

The report is still bogus by the way and the percentages are incorrect. I might get to fix that once I'm done with TPAC 2010 but I'm hoping someone else will be faster than me.

Since I've got your attention, let me repeat it again:

We need all the help we can get to make the test suite relevant and informative. Unless the community starts helping W3C, we won't be able to properly test HTML5.

The Web browsers of tomorrow are being developed and tested today, so don't wait and help us make the Web a better place!