Before we start: Logistics

• WIFI: Mozilla-Guest
• IRC: irc.w3.org
  #webtest
• Zakim: +1.617.761.6200 (SIP access)
  passcode 8378 ("TEST")
• Scribing
• Public record?

Why Testing?

• The Open Web Platform is growing
• We're shipping code and devices "HTML5 ready"
• ... and we still have interoperability issues
• Agreeing on standards is one part of getting it right
• Testing is another part
• And fixing code is the third one

Testing at W3C

Currently, from the W3C Process:

Shown that each feature of the technical report has been implemented. Preferably, the Working Group SHOULD be able to demonstrate two interoperable implementations of each feature.

If the Director believes that immediate Advisory Committee review is critical to the success of a technical report, the Director MAY accept to Call for Review of a Proposed Recommendation even without adequate implementation experience;

Implementation of each feature?

Groups usually test:

• existence of the feature
• the feature does what it's supposed in the general case

Minimum to declare victory

Groups usually don't:

• test all the error cases
• look for corner cases

HTML5 CR exit criteria

Interoperable

Qualitatively interoperable at a judgment level, not necessarily for every spec assertion. A test suite may be used as guidance for the qualitative decision.

Judgment level

For features that are well known to be widely implemented and deployed, and where implementations are believed to match the specification, the Working Group will assume effective real-world interoperability without testing.

Mobile and TV Profile: Common parts

• HTTP 1.1
• ECMAScript
• Edition 5.1
• Cross-Origin Resource Sharing
• HTML5
• Scalable Vector Graphics (SVG) 1.1 (Second Edition)
• WOFF
• CSS 2.1
• CSS Backgrounds and Borders
• CSS Color
• CSS Transforms
• CSS Transitions
• CSS Animations
• CSS Fonts
• DOM4 API
• CSSOM View
• Web Storage
• XMLHttpRequest
• Web Workers
• Canvas 2D Context

We have gaps in tests

No test suites:

• HTTP 1.1
• Web Origin concept
• CSS Background and border
• CSS Animations
• CSS Fonts Level 3
• CSS Transitions
• CSS Object Model
• CSSOM View Module

We lack tooling

(depending on the groups)

• submission
• review
• coverage
• documentation

And we lack consistency between the groups

So, what do we want?

• Increase test coverage?
• Better tools?
• Better documentation?

Terminology

test framework

the combination of the test runner and the repository of test suites and produce test reports

test runner

program which is able to run multiple, potentially nested test suites and collect the results

test suite

collection of test pages

test page

single HTML document that contain one or more tests.

feature

a distinguishing characteristic of a specification

test

verify one or more assertions to test a feature

assertion

smallest building block of testing

testharness.js

W3C test page runner for Javascript tests

Level of testing

1. Unit Testing
   1. Mininum to declare victory (aka Minimum)
   2. Test corner and error cases (aka Full)
2. Integration testing
   1. Combine specifications (aka Combination)
   2. Check use cases (aka Use Cases)
3. Regression testing
   • Software defects (aka Regression)

Increase test coverage?

• Vendor contributions
• Crowdsourcing contributions
• Subcontracts

Vendor and crowdsourcing contributions

Despite the appreciated efforts:

• Slow
• Unpredictable
• Uncertain coverage
• Needs education and documentation for ramping up
• Test quality varies
• but does allow for scaling

Subcontracts

• Count around $100 to test each feature (IDL or HTML attributes, etc.)
• HTML5 is around 10,000 features (section 4.8.10 Media elements is 800)
• Will give us a test suite between Level 1/Minimum and Level 2/Full.
• W3C still needs to review the tests

Specifications: Where do we start?

• HTML5
• HTML5 Audio and Video
• HTML5 Tables
• CSS Animations
• HTTP
• ...

Better tools?

1. Test management system
2. Test Framework

Test management system

• Test coverage reports: where are the test gaps for the Mobile and TV profile?
• submission tool
• test review tool

Ensure proper documentation

Make it consistent among groups

Test framework

Is it important for W3C to:

1. provide a test framework?
2. produce test reports using this framework?

Better documentation?

• Identify the gaps
• Facilitate entry
• Improve the test quality

Agenda

1. Introduction
2. Testing and accessibility
3. Proposals from W3C Members
4. Goals and requirements
5. Priorities
6. Strategy for Success

So,

• What do you want to test?
• What tool would you like?
• And do you propose we get those?