Testing/Test Framework TF/WebDriver

Proposed architecture

1. Device visits the test server. If it is identified as being WebDriver-compatible (how do you do this identification still tba), the user is asked to enable WebDriver (how you do this seems device specific and potentially painful) and provide the remote's IP address (via an form input field?).
2. The test server connect to the device using the provided IP address.
3. The test server instructs the device to navigate to each test page it needs to run.
   • if the test page uses testharness.js, the results are sent back to the test server via the JSON protocol.
   • if the test page is a ref test, a screenshot is sent back to the test server and compared to the reference image there.
4. Once the tests are all run, the test server compiles the results and navigates the browser to the results page.

Current implementations

Mozilla

• Desktop and FirefoxOS https://developer.mozilla.org/en-US/docs/Marionette
• Fennec impl stalled: https://bugzilla.mozilla.org/show_bug.cgi?id=787203 not being worked on actively and is blocked by https://bugzilla.mozilla.org/show_bug.cgi?id=787280

Using WebDriver to Test Devices

(Notes from Simon Stewart)

Problem:

We want to host a set of canonical webdriver tests on a publicly addressable server and run these against a device that may not be publicly addressable.

What We Know:

• The server hosting the tests has a functioning web server.
• All WebDriver implementations are required to have an implementation of the JSON over HTTP protocol listed in the W3C WebDriver spec.
• The device being tested can access the Web.
• The device being tested is behind a corporate firewall: it cannot be reached by the server.
• The device adheres to the WebDriver spec.
• It can be assumed that the people running the tests (the testers for wont of a better word) are minimally technical.
• The tests hosted on the server are written using the language bindings of WebDriver and therefore represent the "local end" mentioned in the spec.

Options:

The obvious option, of having the tests run on the server and connecting to the device can't be followed because of the firewall. The following may be considered:

Option 1:

A simple "reflector" is created. This would be hosted on the same network as the device and would be implemented as a basic HTTP server. In order to run the tests, the following steps would be required:

1. The device's WebDriver mode would be enabled.
2. The testers would load the main page of the reflector. This would be a form, allowing the tester to input the URL used by the device for it's WebDriver JSON/HTTP implementation.
3. The reflector opens an HTTP connection to the server, asking to create a new testing session.
4. The server responds with the first WebDriver Command to execute and a key to identify this session with. This causes the HTTP connection to close. The reflector forwards the Command to the device, translating it to the JSON/HTTP protocol as necessary.
5. The device sends the WebDriver Response to the reflector, which opens a new HTTP request to the server and sends the Response and key back as the payload of that request.
6. The process of reflecting continues until the end of the tests.

Option 2:

The test scripts are hosted on the server but are actually run on the same network as the device, with results being uploaded to the server. The following steps would be required:

1. The testers set up a machine on the same network as the device and install python, easy_install and/or pip, as well as virtualenv
2. The device's WebDriver mode would be enabled.
3. The testers run: `curl -L http://server/runtests | DEVICE_URL=http://address.of.device/webdriver/url python`
4. This would execute the downloaded script, which would:
   1. Create a new virtualenv to use for the remaining steps
   2. Install local end bindings for webdriver (probably the python selenium libs)
   3. Download the full suite of tests from the server via HTTP(S)
   4. Run each of those tests
   5. (Optionally) Upload the results to the server.

Problems Not Considered:

What happens when the device is on a 3G/LTE network and is otherwise unreachable (that is, hosting a simple server or reflector on the same network is extremely hard to do) There are commercial and OSS tools that already do something similar, so it's possible.

The WebDriver JSON/HTTP protocol is very chatty. If we lose a single HTTP request or response then the entire test becomes invalid. We could mitigate this by using persistent HTTP connections or another technology entirely (websockets, perhaps?)