W3C

Testing

29 Jan 2013

Contents

  1. Outcome
  2. Background
  3. Goals
  4. Meeting agenda
  5. Participation

The meeting's minutes are available.

Outcome

W3C should improve the interoperability of the Web platform, and support external efforts in testing Web technologies. As such, several task forces were created during the testing meeting:

  1. Test Framework Task Force: scope, requirements, plan for a W3C testing framework (lead by Tobie Langel)
  2. Test Management Task Force: scope, requirements, plan for a W3C test management system (lead by Tobie Langel)
  3. Resource Center Task Force: getting documentation started and off the ground (lead by Tobie Langel)

Separately, there will be investigation around:

  1. the test coverage of the HTML5 and CSS Transform specifications, including the coverage in vendor tests (HTML5 is done by Robin Berjon, CSS Transform is done by Rebecca Hauck, Bryan Sullivan will look at others)
  2. the amount of resources needed into converting existing vendors into W3C tests (lead by Tobie Langel)
  3. adapt testharness.js so as to facilitate vendor test conversion (e.g. through shims) and look at synchronization between the W3C repositories and vendor repositories (lead by Tobie Langel)

Depending on the gaps in tests identify by the investigations above, W3C should find ways to fill the gaps. As such, W3C will keep looking for resources (contributions, human, and funding) to expand its testing effort.

Follow-up discussion with the task forces are expected to happen on the public-test-infra mailing list (subscribe). All related information will be available on the Wiki.

Background

There have many discussion around testing since last year. During TPAC 2012 there were 3 back-to-back sessions related to Testing (Testing, Test Infrastructure, Test the Web Forward), as well as the creation of a Testing task force in the Web and TV Interest Group.We have been having discussion since then with various organizations, including potential companies willing to produce tests for the Open Web Platform. Both the television and mobile industries are actively looking at testing the Open Web Platform. The Web & TV IG just started a requirements gathering task force and the mobile industry has been active in the CoreMob community group. The television and mobile profiles share some common specifications.

Drawing a picture of the current situation, we obtain overlaps between the different efforts and profiles:

Testing-graph.png

Goals

Due to the industry uptake and the increasing size of the Open Web Platform, this testing project will be at a level that has never been done before at W3C and, as a result, the meeting will help determine the size of the problem and identify the resources.

We would like to convene the W3C Members who are interested in contributing to the testing of the Open Web Platform to a high-level meeting where participants will develop a strategy for testing the Open Web Platform, and establish priorities, objectives, and funding sources for 2013 and beyond.

Using the television and mobile profiles as starting point, we will establish the priorities for the testing effort in 2013 and beyond.

Contributions towards the testing effort may take many shapes:

This is going to be a strategy meeting and, as such, technical topics like choice of framework or technical comparison of testing methods will be avoided.


Meeting agenda

The meeting will be starting at 9am PT and finish by 5pm PT.

  1. 8:30-9:00 : Breakfast
  2. 9:00-9:30 : Introduction and Objectives (Philippe Le Hegaret) (slides)
    1. Terminology
    2. A subset of the Web platform: Mobile and TV Profiles
    3. Differences of television and mobile profiles
    4. Common profiles: Current Status and gaps, Standards for Web Applications on Mobile: current state and roadmap
    5. Vendors offer
  3. 9:30-9:50 : Accessibility and testing (Judy Brewer) (slides)
  4. Proposals from W3C Members:
      1. Which specs are the most important for you to start with?
      2. How can your organization contribute to testing those specifications?
    1. 9:50-10:10 : AT&T (Bryan Sullivan) - slides
    2. 10:10-10:30 : Comcast (Mark Vickers) (slides)
    3. 10:30-11:00 : Break
    4. 11:00-11:20 : CableLabs (Bob Lund) (slides)
    5. 11:20-11:40 : Mozilla (Jet Villegas) (slides)
    6. 11:40-12:00 : Facebook (Tobie Langel) (slides)
    7. 12:00-13:00 : Lunch
    8. 13:00-13:20 : Adobe (Larry McLister) - Onramping and scaling
    9. 13:20-13:40 : Masahiro Wada (slides)
    10. 13:40-14:00 : Keio Research Institute at SFC / Tomo-Digi (Yosuke Funahashi) (slides)
  5. 14:00-15:00 : Goals and requirements
    1. Testing requirements
      1. Level 1: Working Group test suite, ie the minimum to declare victory and move a specification to Recommendation
      2. Level 2: tests all assertions and corner cases of the specifications
      3. Level 3: Beyond the specifications: performance, etc.
    2. Tooling: test framework, etc.
      1. Is W3C Test Framework enough?
    3. Documentation
  6. 15:00-15:30 : Break
  7. 15:30-16:00 : Priorities
    1. Specifications: Tests and Reviews
    2. Tooling: Framework, submission, etc.
    3. Guidelines and Documentation
  8. 16:00-17:00 Strategy for success
    1. Increasing the number of tests
      1. Contributions
      2. Outsourcing (aka Work for Hire)
      3. Test Fund
    2. Test Review
    3. Better Framework and tooling
      1. Contributions
      2. Outsourcing

Participants

David Baron (Mozilla), Michael Cooper (W3C), Art Barstow (Nokia), Judy Brewer (W3C), Clarke Stevens (CableLabs), Glenn Adams (Cox Communications), Mark Vickers (Comcast), Tobie Langel (Facebook), Lars Erik Bolstad (Opera), Rebecca Hauck (Adobe), Larry McLister (Adobe), Filip Maj (Adobe), Charles McCathie Nevile (Yandex), Jet Villegas (Mozilla), Kaz Ashimura (W3C), Yosuke Funahashi (Tomo Digi), Peter Linss (HP), Hiroyuki Aizu (Toshiba), Alan Stearns (Adobe), Graham Clift (Sony), Jeff Jaffe (W3C), Bob Lund (CableLabs), Michael Champion (Microsoft), Philippe Le Hegaret (W3C), Dan Sun (Verizon), Masahiro Wada (KDDI), Bryan Sullivan (AT&T), Edward O'Connor (Apple), Paul Irish (Google), Cyril Rickelton-Abdi (Turner), Andrea Trasatti (Nokia), Jennifer Leong (AT&T), Olu Akiwumi-Assani (Verizon)