How do we test a Web browser? (one year after)
Author(s) and publish date
- By:
-
-
Philippe le Hegaret
-
- Published:
Almost one year ago, I wrote about browser testing in How do we test a Web browser?. Since then, the situation improved a bit but we're still far from reaching an appropriate comfort level.
There is now a new mercurial server and youl'll find tests related to HTML5 or Web Applications. All of those tests are automatically mirrored on test.w3.org. The Mobile tests are also linked from there. The CSS and MathML test suites are still in a separate space. The SVG Group and Internationalization effort are in a transition phase.
A few of our Groups have documented how to contribute tests:
The Internationalization effort has been reporting on test results for while, such as for language declarations.
The HTML Working Group, working on HTML5, also started to publish ongoing test results. Thanks to a few contributions, we started to test a some features, including video and canvas:
| Feature | Internet Explorer 9 Platform Preview 4 | Chrome 5.0.375.125 | Firefox 4 Beta 2 | Opera 10.60 | Safari 5.0.7533.16 |
|---|---|---|---|---|---|
| attributes | 100% | 100% | 100% | 100% | 100% |
| audio | 100% | 100% | 100% | 100% | 100% |
| canvas | 85% | 86.67% | 76.67% | 85% | 88.33% |
| foreigncontent | 100% | 10% | 100% | 10% | 10% |
| video | 84.62% | 69.23% | 61.54% | 61.54% | 61.54% |
| xhtml5 | 100% | 22.22% | 66.67% | 100% | 33.33% |
The HTML test suite only contains 97 approved tests for the moment so don't draw too many conclusion from the result table. The number of tests needs to increase significantly if we want to test HTML5 properly. Around 900 tests are waiting to be approved within the task force but we're lacking participants. Help in identifying more test sets and submitting them to the group would also be appreciated.
The Web browsers of tomorrow are being developed and tested today, so don't wait and help us make the Web a better place! So pick your favorite HTML, CSS, SVG, MathML, or API feature, write as many tests as you can on it, and submit those tests to us. Before sending bug reports to various browser vendors, that's your best chance to get your favorite feature properly implemented and the browser developers will even thank you.
Why are you comparing unstable versions (IE and Firefox) with stable releases (Chrome, Opera, Safari)? There's a some new HTML5 releated features in both Chrome 7 and Opera 10.70.
Hi Monty,
Given the low level of resources we have in the HTML testing group, I take what I can get in terms of test results at the moment. Microsoft and Opera are actually the only two browser vendors active in the effort, and Opera didn't judge necessary to update the results so far. They're both active at updating the test harness, which is more important than updating test results for the moment. Join the effort and you could help us maintaining more comprehensive results.
Philippe, it isn't helpful to post results of running a very small number of not at all representative tests (most of which were written by Microsoft), especially as they aren't very high quality. We are expecting hundreds of thousands of tests to be submitted, and I hope they will be more useful than the ones we have seen so far.
Hi Ms2ger,
I want to bring attention to the fact that W3C will publish ongoing test results on Web technologies, especially HTML5. In addition, a good fraction of those tests were not written by Microsoft: Philipp Taylor wrote the canvas tests for example (and we still need to go through most of them). Regarding the usefulness or quality of the tests, I wouldn't be so affirmative: some of tests were written at the very beginning and the test harness is still evolving. So it's not a surprise that they aged quickly and we already thinking about rewriting them. Like you, I do hope that we'll have more tests and that's why I want to bring attention to the effort.
Im wnt to show a button in a html.in the Firefox or IE is no problem,but in Safari not showed.
why?
Hi Javaer,
If you're willing to write a test that contains the bug you're running into, I'm willing to take your test and submit to the HTML or CSS Working Groups.
I have written a CSS3 test which shows a rounding error with background-size in Opera:
http://winner.cspsx.de/background-size-rounding.zip
Chrome 5? Current stable is 7.0
@Chris, I forwarded your test to the editor of the CSS border and background editor. He'll have a look at it and see if it's relevant for them.
@Igor, other folks have noticed that and submitted test results for Chrome 7. The table in this blog entry is static but we're maintaining the one in the testing report.