Lunch discussion summary

background

inspired by a political campaign where feedback from a blog created real-time changes in how the campaign was being run.

Main points

knowledge problem about what they are and how to create a good one.
resource issue - complex issue, want tools, skills within WGs, keeping up-to-date, managing connections/
what about test suites after groups progress past Rec
issues about proprietary tests and anonymity
lots of interesting ways that test suites are used in groups (to facilitate process), to test the spec, and then outside of w3c for communication.
different types of tests (boolean, subjective), conformance vs implementation

General observations about test suites

test suites are good
concern about resources in general (for creating test suites - time, skills, tools)
- W3C resources
  - maintain a repository?
  - a tool to generate implementation reports
  - WBS-like to generate test cases
  - a language like xmlspec to write test cases
- WG resources - what do WG participants know about test suites? about best practices? (seem to be lack of awareness of QA materials, esp matrix). what skills are required to write them?
  - who writes the tests? the person who writes the spec or someone else? some ppl lfelt the spec writer will be best, other ppl feel that diff enough skills should be different, others that a good test of the spec if someone other than authors write them.
if there is a dependency between test suites and there is a change, how manage?
proprietary issues. anonymity issues.
ability to extend test suites and reusability between test suites (ala Patrick and vincent)
Testing your own spec and not that of others. e.g., test css not html or asst technologies (where does the test fail?)
expectations for joint test suite work:
- groups expecting joint test suite work/needs: mmi, uawg, wcag wg, atag wg, SWBPDWG
- groups not expecting joint test suite work: voicewg (separate ssml, voicexml)
subjective tests/those that require human judgement - concerns about, but needs for.
how do test suites incorporate optional features?
some specs are orthogonal, e.g., xml and soap. if soap works and xml works don't need xml+soap test.

how test suites are used

interoperability not just w/our own specs, but with other organizations.
test suites as communication tool both within and outside of w3c
how different groups use test suites. for example, In Schema before a change is made a test has to be created.
differences in when WG start and how they go about it. some start early, some not until CR. some incorporate into spec, some don't. some start at the tests and work backwards, some start with the spec and generate tests from there. therefore, if a tool or process via w3c, then have to consider all of that. rare that a given participant has worked on multiple test suites.

possible recommendations

QA "summit" to raise awareness about testing and the testing process, available QA resources
test suites are part of public WDs? have at least identified testable assertions (SpecGL??)
better orientation for new WG participants? part of it gives examples of good test suites (several ppl are saying, "don't know what a good test suite looks like" or "don't know anything about test suites.") Perhaps a QA orientation page. "here are good examples" samples of parts of test suites to get a feel for what they should do (instead of just long list of them)
suggest a mailing list for discussion of join test suites.
Requirements for Joint Test Suites
a) Each spec. independently testable;

b) Identify dependencies on other specs/groups at the testing level;

c) Create a process for coordination.
next steps: sending all of this data to the QAWG to process.
- agenda item on the next chairs meeting? or a focus of a chairs meeting?
- best practices test suite area? more than the matrix, good examples.

observations about the lunchtime "experiment"

some groups really got into it, and others didn't. 5 groups didn't even hand in notes.
format we received feedback in: hand written notes, to email, to web pages, to short verbal summary (multimodal) some short, some long


1
2	"regrets"
3	w3c should have a higher-level management that tracks test suites and implementation reports. should not be working group owned, it should be centrally located and searchable. tests will live beyond the WG. WG should not cease existence until they complete well-documented test suite. Issues with test suites at different levels of maturity - versioning or a completeness issue? does the test suite properly reflect the requirements in the specification? is there a formal process for validating test suites?
4	CSS tests are organized by metadata, try to avoid to use too complex format support so as to as to test only the CSS aspects [issue of testing only your spec and n ot the other technologies. css - html or "host" tech overlap) big efforts for VoiceXML were put into creating an abstract metalanguage that could be re-purposed for testing VoiceXML independently of the context it is used for identified who is likely to need joint test suites, who not. differences between Implementation Testing and Conformance Testing, but the former tends to get close to the latter when it is maintained and integrates e.g. bugs reports
5	The group basically exchanged ideas on good principles for test suites making the test suite open for contributions, and with that in mind make it as easy to contribute as possible, and not as easy to automate as possible. Joint testing perhaps not as important as doing good test suites and making it easy for others to contribute and use. But have to be careful with centralization. W3C style may be to define a common reporting scheme for groups that want to. http://www.w3.org/Style/CSS/Test/testsuitedocumentation http://www.w3.org/Style/CSS/Test/guidelines.html
6
7	w3c develop test harness that can we used by all. w3c create test suite management system. w3c can help by: developing tools, educate WGs, "enforcement" to create test suites, some coordinates [QA]
8	resource needs: take advantage of cvs, bugzilla. finding knowledgeable people - ppl who write specs often have different skills that ppl who create tests. managing dependencies is a resource demand. takes time to coordinate. to get the right people looking at it and determining what needs to be tested. if making a test suite, makes your spec better. you can refernce in the spec what to do. some groups only seem to do minimum needed for CR. Evidence that if create test suite during spec creation is it actually going to be a better spec? Needs to be central place [needs to be more awareness about QA and matrix?]
9	need a test suite coordination group. WG should document a procedure for submitting new tests, so public can contribute
10
11
12	issue: what are we testing? WBS-like tool for generating test cases. XML Binary - some be open source, some by proprietary some tests may mean different things to different ppl. may not always be the same output. test suites may be hierarchical - some global, some not if testing a protocol, not always possible to test behavior of an implmentation w/out a "standard" api
13	test suites can be used to learn a new technology. YEAH tests!!!
14	test suites need to be in the culture of the WG, not just the responsibility of a couple of people. DAWG chair is insistent that WG members provide test cases on disputed issues.
15
16	See a need for a framework or environment into which you can drop test suites to get interoperability and composability. Need to think about how to integrate with technologies from other organizations. For example, Choreography needs to be able to generate BPEL endpoints, and uses reliable messaging. XML depends on Unicode. In SMiL, needed two tests for every feature, since the data format wasn't universal.
17	using the test suites to pressure browser makers to support rendering + technology need to make the test suite while you make the spec can't test formal semantics document. other expect ~250K test cases??? voicexml, ssml both develoepd by VBWG but aren't joint.
18	discussion about wcag testing, desire for xmlspec-like framework. difficulty of creating. perhaps limit scope tests? discussion of binary xml taking over the world.
19	specs need to be testable. testable assertions should be included in spec intself. not all specs expressible as a set of features each of which may be implented or not. [are some specs not testable?] how do test suites incorporate optional features?
20	http://www.w3.org/XML/Group/2005/03/test-suite-lunch-20.txt
21	concern about ppl outside of group misinterpreting arbitrary limits? Not all requirements are machine-testable. versioning an issue concern about testing complex interactions. when putting together a variety of technologies, how do you test all of that? concern about decoupling test suites (and then merging back together?)
22	what are the features needed for a test suite? normative parts of spec if there is a dependency between test suites and there is a change, how manage? development of test suites should start early. person who writes the spec is best person is to write the tests.
23	questions about the question.who owns it? what constitutes a test? suggest a mailing list for discussion of join test suites. cases where a test can pass but result isn't useful. [issue of spec or test?} In Schema before a change is made a test has to be created. Neil/Dave/Judy/Bill... we pretty much all agreed on nap and showering areas available in offices. Matt: -- Personally -- I think we need more tools for producing specs, cross linking, etc. Having spent a while hand tweaking the DPF spec, only to realize the XML Spec schema was out there was kind of a shock -- Ideally I could link my test suites easily into another specs test suites, etc.
24	proprietary issues.
25	Lots of really good questions about what test suites are and how they are generated. Requirements for Joint Test Suites a) Each spec. independently testable; b) Identify dependencies on other specs/groups at the testing level; c) Create a process for coordination.
26	need for central repository to help manage test suite creationg. xforms (embedded tech) creating tests that can be reused (where it will be embedded) Subsets... Is it OK to pass some tests and not all? It may be useful to separate the spec for a language (eg data formats) from the spec for a processor of that language (otherwise I can create a processor that does useful stuff but with a subset of the language).
27	how does a WG know what a future group will ask of their test suite? [versioning an issue] how find test suites? [unawareness of testing. is the QA page hard to find?] Reusability an issue. Ability to extend existing test suites. Online web service for test suites.
28	did it, but didn't get to the notes in time (before presentation)
29	they talked about how the different groups they participated in could intergrate, e.g., SMIL, SVG, I18N
30
31	test suites are good. test suites as communication tool
32	it would be good to have a process document about how to write test suites and how to link the tests to the concepts in the spec. oftentimes it's hard to generate the test suite early before you know what the spec will look like.
33	Issue: IPR (ppl contributing), open source [learn more about the test suite submission process] Issue: could the w3c license test suites to peple? [emphasize Patent Policy] When publish a WD, should publish a test suite. Transforms spec into test suites. PPL don't have enough resources in own group, joint test suites be a lot more work. Some tests are boolean, some need a human interpretation. Is everyone aware of that? some tests are very hardware/software specific and difficult to replicate. encryption tests - some may be proprietary relationship of w3c standards and working with other oganizaations technologies and tests. interoperability not just w/our own specs, but with other people's.