MWI Team Blog

Dispatches from members of the W3C Mobile Web Initiative Team

Categories: Current state (32) | Developing Countries (15) | Events (20) | Looking forward (13) | News (42) | Technical (33) |

Mobile Web: test it, use it — 28 May 2007

We started back in December a Working Group of the Mobile Web Initiative dedicated to testing: the Mobile Web Test Suites Working Group, which I have had the pleasure to co-chair.

If you've ever complained that you couldn't use a specific CSS property on a given range of user agents, if you've ever sworn against a badly broken implementation of Javascript on a mobile device, and if more generally you wish the Mobile Web as a platform was much more reliable than it is today, hopefully this group is part of the solution to your problems!

Indeed, our goal is to help identify the bugs and inconsistencies in existing implementations of the Web technologies through the most efficient tool found in computer science: test suites.

As part of this group, we have been considering two sort of test suites: conformance test suites and a community-based test suite; let me explain these two ideas in more details.

What we call "conformance test suites": basically, a set of files (generally very simple Web pages) that illustrates the various features of the tested technology, and which a user agent should render in a well-defined way; W3C has been developing test suites from some of its technologies, OMA has also been working on some of these.

Our first work item has been to review these existing test suites and evaluate how well they served their purposes. A common problem to these tests suites was the difficulty to go from one test to another, as well as to record the results for each individual test case - when you go through 100+ of these, you don't really want to have to take notes on what worked on what didn't.

So, to solve - at least partially - that issue, we developed a simple system to navigate through and record results from existing test suites - which more or less matches what they call a "test harness" in the field. We have already applied that harness to the CSS Mobile Profile test suite, the SVG Tiny 1.1 test suite and the DOM Level 1 test suite, and started to collect results for a few browsers - mostly from the Working Group participants.

While this harness isn't perfect yet - we would like to add the possibility to start and stop testing sessions, so that a user doesn't feel obligated to go through the full test suite in one shot -, it works reasonably well, and anyone should feel free to start submitting test results using it. The Working Group is also interested in any feedback you might have on that system.

Still on the topic of the conformance test suites, one of our great achievements was to make the DOM Level 1 test suite mobile-friendly: the W3C has very complete test suites for each of the three levels of the Document Object Model specifications (the core of what makes Javascript useful), but they are presented in a very desktop-oriented fashion: frames, big scripts to load, heavy network usage, etc.

So we looked at improving that situation, and eventually managed to produce a much more mobile-friendly version of the test suite, while keeping as much automation as possible to make our users' life less tedious - I discovered while looking at the results of that test suite that the DOM engine behind Blazer (Palm's default browser on my phone) and the Netfront browser on my PSP was mostly the same - they fail more or less exactly on the same tests!

Beyond all this work on conformance test suites, we have started to work on quite different scheme as well: what I called above a "community-based test suite". The idea behind this is to build a test suite from test cases submitted by the community, for the community.

In other words, we are setting up a framework that will allow anyone out there (but most likely Web developers, standard evangelists, but maybe also operators) to create test cases that illustrate a specific technology or the combination of several technologies; once reviewed by the Working Group, these test cases will be made available so that anyone (in this case, even random mobile web users e.g. frustrated to see that their browsers don't do what they expect) can test whether their mobile web browsers implement correctly or not the feature under test.

I'm hoping that by opening such a framework, we will be able to create a greater incentive for browsers vendors to improve the quality of their implementations, but also more directly to help Web developers know what features are safe to use on as many user agents as possible, what techniques allows to circumvent bugs in existing implementations, etc. We surveyed the community a few months ago on the interest for such a framework, and the results were fairly positive over all.

Given the rather interesting effect the ACID2 test developed by the WaSP had on the desktop browser market, I think this group can create many opportunities to improve the interoperability situation in the mobile browser space; if you're interested to follow what's going on, feel free to subscribe to our publicly archived mailing list; if you think you can contribute to this work, please get in touch with me ( as you may be eligible to become an Invited Expert in the group.

by Dominique Hazael-Massieux in Current state, Technical Permalink

Comments, Pingbacks:

No Comments/Pingbacks for this post yet...

Contacts: Dominique Hazael-Massieux