Test the Web Forward
Next Steps

Cambridge, MA

Advisory Committee Meeting

Plan 2013: A Value Based Proposition

  • Goal: Increasing the interoperability and Reliability of the Open Web Platform
  • W3C seeks sponsors (financially and through resources) to test the standards and build a solid Testing Infrastructure
  • Objectives:
    1. Comprehensive, automated, and freely available system for testing Web products
    2. Database of test results of tested products
    3. Collaborative process for determining requirements and priorities
    4. Minimize industry costs using a coordinated testing initiative

Outcome

  • Almost no interest in financial support
  • However, was educational and helped changed attitudes
  • Got great contributions/minds to leverage: Adobe, Opera, Facebook, Mozilla, Intel

By end of 2013

  • Vendor contributions: Mozilla started work on infrastructure
  • Intel was ramping up its test contributions
  • Tobie Langel (Facebook fellow) has revamped our approach to testing to lower the entry bar
  • Adobe initiative launched Test the Web Forward and gave the brand to W3C

Past six months

  • Vendor and crowd sourcing effort revived by changed of attitudes
  • Contributions have doubled
  • Repository (WPT) now contains 60 different test suites, but coverage varies (DOM has ~47K tests, HTML has ~142K tests)
  • James Graham, Mozilla, is doing tremendous work on the infrastructure
  • 3 Test the Web Forward events (Seattle, Seoul, Tokyo)
  • Web&TV test task force is reevaluating its goals and priorities

Community Contributions

2013AprilJulyOctober2014April01002003004005006007008009001,000All Contributions (merged and pending)Pending Contributions (80% of them are unreviewed)

Infrastructure

  • Goal: Modular and simple way for everyone to submit and run tests
  • Single repository for tests and tools
  • testhtewebforward.org is used for events and documentation
  • Easier deployment (wptserve server)
  • More complete tool for test execution (wptrunner)
  • Automatic contribution checks (travis integration)

What's next?

  • Foster the crowd sourcing effort
  • It's a community effort: working groups, web developers at large
  • Encourage individuals to contribute and review tests

Ideas

We don't lack ideas…

  • More tests, more cleanup
  • Improve wptrunner
  • Improve automatic test checker
  • Getting more browser implementors writing directly for WPT
  • Nightly test results from browser vendors
  • Reintroduce support for HTTPS
  • WPT and CSS repo cooperation
  • Saucelabs results?
  • Implicit meta information, including monitoring coverage?

Take aways

  • Back in early 2013, we faced two problems:
    1. We didn't know how to scale up testing due to its complexity
    2. Industry wanted us to do more on testing
  • Testing proposition didn't get support, but the focus changed attitudes
  • Improving infrastructure facilitated test contributions
  • W3C Members can help by contributing and reviewing tests

testthewebforward.org