Results of Technical Plenary session on Joint Test Suite

During the Plenary Day of the W3C Technical Plenary Week 2005 in Boston, MA, Wendy Chisolm and John Gunderson organized a session on Joint Tests Suites Development (slides), which proposed two introductory presentations from the QA Working Group (slides) and from the Compound Document Formats Working Group (slides).

These presentations (IRC Log minutes) were followed by a lunch discussion, where participants were invited to discuss related topics and answering the following questions:

What are the features needed in joint test suites for your specification?
What are the resources and coordination that would facilitate development of joint test suites?
What is the importance of joint test suites in insuring interoperability?

Wendy then presented the first results gathered after the lunch during a lightning talk (IRC Log minutes, summary).

This document gathers the various points that were made during the lunch discussions, as reported by the volunteer scribes.

model to organize and describe test cases
possibility to link test cases back to use cases
test cases description format easy to merge and reuse (sugg: RDF/OLW)
knowledge base about tests cases and test suites
dependencies between test suite/test cases should be mapped formally
maintenance of test suites should be handled above the WG level (e.g. to handle the case where a WG closes), esp. for the benefits of W3C outsiders
common vocabulary to describe joint test suites
collections of test cases should follow enforced rules (like pubrules)
WG not allowed to close until test materials are well-documented
satisfaction of dependencies to exit CR as a new requirement
joint tests suites very important for interoperability
difficulty of referencing test suites at different levels of maturity - how to determine it?
needs for more coordination (coordination groups, QA ?)
needs more consistency wrt communicating information about test suites
object of the test suite: document, schema, user agent, authoring tool, spec itself?
reporting/gathering results from testers, esp. important for subjective testing, done by outsiders (a WBS-like system for test results?)
shared infrastructure needed to have a common basis, e.g. not have to redo all aspects of XML coverage
hierarchical test suites (?)
developing test suite while developing spec helpful
IPR issues can raise barriers to joint test suites
how to avoid hard coding limits in test cases?
subjective testing is harder
versioning is critical, esp. for combined test suites
how to test complex interactions across spec, and what coordination between developers could help doing that?
test suite also useful for pressuring vendors
can't test purely semantics document
hard to know in advance what a future WG may need from a test suite being developed
finding resources for test suite development is hard
automation helps
extensibility mechanisms need to be tested in joint test suite (xml:id use case)
Web Service for on-line test suites?
is it ok to have priotization among features to test?
who should write test cases: spec writers? specific test-oriented person?
what constitues a feature? not all specs can be expressed as a feature
joint test suites efforts should be done by working groups with interdependences working together
need for a W3C-wide test harness, test cases management system, test results reporting system
best practices for test suites needed
management enforcement needed
protocol testing sometimes requires a standard api to test behavior
test suites can also help learning a new technology
test suites tools can re-use CVS, Bugzilla
motivation and training needed to get more test suites
difficult to map test suite to specs when not envisioned from the begining of writing spec
proof that developing test during spec more efficient?
having explanation on how to structure work to make testing less burdensome at the beginning of the group helps
implementors of spec must have test suites; do we need to ask them to contribute more?

aside: Want to hear back of actions resulting from the exercice