TestSubmissionFramework

From W3C Wiki
Jump to: navigation, search

It is often useful to gather contributions from various organizations and individuals when building a test suite.

This page tries to identify the various existing solutions and policies existing to make this possible, and is also a place to discuss what an integrated system that would ease this process would need.

Requirements

Requirements vary immensely depending on whether the contribution system will be used mostly for large-scale contributions from a limited number of organizations, or for atomic contributions by individuals.

The common denominator for all systems would be:

  • identifying the contributor
  • provide a way for the contributor to waive IP rights, agree to licensing terms
  • provide a mechanism for test description (documented format, or User Interface)

atomic contributions-oriented Test Contribution System

Such a system is rather similar to many bug-tracking systems. We can't expect thousands of contributors to learn and understand complex contribution guidelines, so the contribution interfae should be as constrained as possible

  • Web UI
  • upload interface for test subject
  • upload interface for test result
  • drop-down / radio buttons to select scope/area of the test
  • free input field for test description, with guidelines
  • input of pointer to relevant part of spec
  • UI (radio button) for contributor to declare (s)he holds IP rights to the test case
  • UI (radio button) for contributor to declare (s)he agrees to w3c licensing terms

Bulk-oriented Test Contribution System

This is the case of one organization holding thousands, sometimes tens of thousands, of test cases and metadata. The issue here is to be able to import them into a system, which should therefore have a clear:

  • documented format for test case metadata
  • (or) API for automated bulk contribution

IP rights and license agreement declarations can be done on a case-by-case basis.

Test Cases Management System

... beyond the contribution of test cases, how to view, query, edit test case collections

QA qa.png