W3C QA Workshop, at NIST. Washington D.C Area. U.S.A. April 3-4 2001 Presents: Daniel Dardailler (W3C, co-chair), Lynne Rosenthal (NIST, co-chair), David Marston (IBM), Thierry Kormann (ILOG), Mark Skall (NIST), Dan Montgomery (RealNetworks), Rob Lanphier (RealNetworks), Alex Rousskov (Measurement Factory), Anthony Wiles (ETSI), Dianna Callesen (Adobe), Christine Hoffman (Netscape), Scott Vesey (Boeing), Karl Best (OASIS), Peter Wiggin (O'Reilly and Associates), Paul Cotton (Microsoft), Mary Brady (NIST), Andrew Thackrah (TOG), Karl Dubost (W3C), Lofton Henderson (OASIS), Josef Dietl (Mozquito), Susan Lesch (W3C), Dimitris Dimitriadis (Improve AB), Chris Lilley (W3C, scribe), Gerald Oskoboiny (W3C, scribe), Charles McCathieNeville (W3C), Kati Haritos-Shea (NTIS), Janet Daly (W3C, scribe), Philippe Le Hégaret (W3C, scribe), Allen Brown (Microsoft), Henry Thompson (Univ of Edinburgh/W3C, scribe for Susan Lesch's talk), John Williams (Exodus Performance Labs), Markus Mielke (Microsoft), Jason Brittsan (Microsoft) + couple of locals from NIST ================================================================= Minutes of Tuesday morning discussions Scribe: Chris Lilley FIRST SESSION: Why we're here daniel dardailler: separating the test from the spec development - need to find a balance. Good to have someone that is not the spec writer and not the programmer to do tests. But also, spec writer needs to say which is the correct way (and fix spec if necessary). Need to close the loop. Several ways this can happen - two working groups (neds good coordination) and WA activity will promote this coordination. Past experience of external development that the spec WG was unaware of. dave marston: is it a foregone conclusion that a separate AQA activity is the best way daniel dardailler: no, we can talk about that. Some wgs are split into task forces etc so it could be done like that paul cotton: outside is "black box" testing, "white box" is by developers or spec writers who can look at and affect the implementation or the spec. question: nist doing lots of stuff, how does that correlate with what oasis is doing? Need good coordination : oasis talk said work on xml, xsl/xpath, dom, schema - where is this work being done? want the overall org chart. lynne rosenthal: develop conf tests both within oasis and within w3c, always in touch with the spec developers. xml tersts release 2 were to bring it up to spec with the latest erratta. DOM tests were done more independently but with tsome liaison with the then chair (lauren wood). For schema testing, .. (something about datatypes, missed) .. For FO, we were approached by Sharon adler to see how to test the candidate rec. Whether we are leadin or participating, it is essential to corrdinate with the producers of the spec. josef deitl: spec developers and conformance testing; there is also a third party, like a QA WG perhaps, might improve the cordination 9or just adaniel dardailler to the discussion). W3C has always developed sample implementations not "reference" implementations which make the actual spec onselete. Transfer this experience to the conformance development, as daniel said keep the authority within the WG. Entire W3C process is focussed on getting that second opinion thierry korman: what kind of problems should test suite solve? three kinds, some should be done inside and some outside - will talk about this later. henry thompson: schema wg is clear that they must retain responsibility for "what does the spec say". But that is the only part of QA that schema has signed up for ... so we need an external group to do the rest and an external QA WG could be useful to broker the interactioin between test suite developers, implementors, WGF .. essental do do this on a case by case basis. daniel dardailler: qa wg will also produce documents to explain how to do qa, rob lampier: some discussion as to whether wg should take control of their qa. looking at WAI, it has been incredibly valuable and they are clearly needed. So taking that precedent, a QA activity on similar lines would be great. daniel dardailler: horizontal activities such as WAI, I18N QA mary brady: Experience of both close coupled and loose coupled, danger of testing the groups understanding not the spec - but in practicve these issues get brought up and resolved. robert neff: time to market for this - test requirements might be different in different cases. no one framework is appropriate. janet daly: when there have been issues with external test packages, w3c hears about it. Sometimes there are very independent third party tests, but w3c is still the accountable party so i agree with henry that looking at test suites and tools case by case is true, but there must always be W3C accountability for things related to our specs and this rests within the W3C WG to be aware of what is going on. charles mckathie-neville: in accessibility there are complex test cases, hypothetical situations, important that test suite allows complex real-worlsd intersection of multiple problems at once. WAI guidelines have simple requirements then a big bunch of tests, techniques and recommendations. mary brady: coordination difficult. 20000 tests or more, going through them takes time and there will always be issues. need to be solved by whoever has the resources to get the job done in a timely manner. ---- after mark skall's talk ---- See slides charles mckathie-neville: depends on locality eg for wai , in the us conformance requires 11 things to be tested but for australia, conformance would require infinite falsification testing ht: iso 9126 on software - comes in from left field and defines a bunch of taxonomy but I found it very useful. Can define axes of performance - metrics. then methods to determine performance alomng this axis. then finally the measurement of this metric using this method. Some specs like xml have an error terminology others like html and svg define behaviour but not errors; for xml there are two kinds of errors. errors in the implementation, and correct processing of defined error conditions. mary brady: generally only create correct tests but can also create known invaliud tests to check that errors are correctly ken holman: experience of tweaked implementations that passed the suite but failed in the real world. mark skall: and what is the problem with that? ken: nalps(?) software was tweaked to position for a 35mm slide, etc, and then point to their conformance results mark: now we dont do validation and certification, no easy soliution daniel dardailler: glossary is online from this paper, some terms have been used already. QA will try to use these terms consistently eg validatiojn of software vs validation of content .. glossary will be very important paul cotton: there are already xml query performance benchhmarks apprearing from third parties. performance is one particular metric, not discussed yet. mark: if the spec has performance requirements these should be tested. If not ... cmn: can have a grab bag of additional tests (drawn at randon) to check for tweaking. ---- after Karl Dubost's talk -- charles mckathie-neville: how to say a test suite is rubish? lofton henderson: yes, need to do this daniel dardailler: QA of the QA lofton henderson: svg experience was that implementos found the broken tests *and proved* it to the WG janet daly: implicit endorsement of a link from w3c to somewhere else, so be careful david marston: aditional column on the matricx whether they define behaviour of a particular thing or not, whether they are testable christine hoffman: there is an assumed credinvility of tests linked from w3c alex rousskov: important service to link all test suites so we know what is out there, not just w3c endorsed products janet daly: reason for not linking - we dont know or we don't like it. decentralised - anyone can make a test suite. QA provides the opportunity for WG review chris lilley: if we don't link to a test suite because it is broken, then either it should be fixed or if needed the spec is made less ambiguous gerald oskiboiny: lots more time to implement broken fallback behaviour than the core w3c specs" ----- after geralds talk ----- daniel dardailler: can it use other DTDs gerald oskiboiny: yes, will download dtds and validate against them. Uses james Clark's SP janet daly: how much time do you spend on this gerald oskiboiny: not much, less than a day a month since the first release, but current contract has 1 day/week charles mckathie-neville: icab has a smiley face if the current page is valid chris lilley: mozilla integration is in composer, its a button to do a validation on the server karl dubost: bbedit on mac has validation. good trend. rob lanphier: how much resource is spent on the validation service, is it significant gerald oskiboiny: until recently it was running on an old pentium pro that was salvaged, now a dual pentim 111 and no load problems henry thompson: ted guild spends a lot of time on load issues etc on the validator philippe le hégaret: already a problem with update of the validator. more work can make stuff that used to seem balid be shown to be invalid. paul cotton: shipped xml query with input files for two parser generators, and got implementations withing one hour. also use cases (preferably in separate files or a zip file) - helped with very rapid implementation. However, over time, public should pick up maintenance of the parser generator files. This also has a great PR benefitr by the way. --- after chaalz's talk ---- dd: implementation, publicity, testing tool ... lots of value. So that was several talks on validation of content. Tomorrow we should talk about integrating these tools, how to validate multi-namespace documents. Now, test suites for implementations... --- after loftons talk --- dd: recording of results is not automated? lh: correct dd: break for lunch ======================================================================== Tuesday 3 April 2001 Afternoon Scribe: Janet Daly XSLT/XPath COnformance TC in OASIS presenter: Ken Holman Notes: the TC uses the xml version as normative. This TC did not have formal coordination with the Working Group. There was no explanation provided, only a statement that the TC attempted to coordinate, and that they placed the blame on the W3C XSLT WG. Ken explained the variety of tasks the TC faced, and explained that interpretation of the specification is explicitly not included in their work. PC: In ISO, there are three requests that may be made, and the highest priority item is a request for interpretation. We must instill this philosphy in W3C WGs. KH: What about disposition of comments? Are these documents part of the W3C Process? JD: It's required to exit last call, and a similar document (the implementation report) is required for exiting CR. RL: The movement to producing disposition of comments is a big improvement to the development of W3C Recommendations. However, an issue tracking system would aid this useful effort. SL: Is there a description/style guide of how OASIS technical reports look? I've been impressed with what I have seen. KH: No, not at this time. CMCn: W3C has made progress in dedicating more efforts for public comment accountability. ---- Mary Brady/Rob Lanphier/Alex Rousskov/Katie&Charles Alex: CL: Users subscribe to a special mailing list for changes to the SVG test suite HTTP agent testing? A desire to see this done in W3C, as IETF has not picked it up, and will not. Comments: This is a problem that starts with the authoring tools and the forgiveness of ua. The most common tools are not set up to make valid markup by default. CL:However, svg forces the hand of validity. RL: In SMIL, 1.0 produced broken implementations. But for 2.0, in order to implement new features, you must use the ns, wich means errors will be thrown back to the user. RN: EO is a critical piece - these need to be introduced at the initial level. There are so many rfps out there which don't include valid markup. This needs to change. ---- Anthony Wiles, etsi no questions ---- mary: AR: Why use XML? MB: When you use XML in a declarative fashion, it may be easily reused as an abstract definition. A domain-specific xml vocab is possible. (issues from alex, convinced of xml's lack of appeal to him) dd: how much of the automated process starts with the spec? mb: It doesn't in this particular case. We'd like to do that, tho. ht: But it starts from the semantic requirements or the schema for schema, which you clearly did in other cases. This is a good start. --- DD's presentation --- Dimitris (presentation) dan: How do you handle a commenter who claims errors within the spec? dim: I hear the questions, and try to resolve them, If I cannot, I then bring it to the wg. dan: What is the timing for the production of the test suits, wrt spec maturity? plh: It's written into the charter that we need them in order to advance DOM Level 3. The DOM Level 2 test suites are necessary, as are DOM L3. this will help us move DOM L3 from cr to pr. This will be done by the DOM wg ts. pc: How will these test suites (and changes) be managed within w3c? dim: CVS is one tool we already have for version control. Further, we have a testsuite to start with. Our goal is to get the tools for managing the testsuite together (i.e. issue tracking). We hope to incorporate others' experience to build the ts and come up with best practces. kd: Further, it's my job to internally that we need tools. cmcn: If I post a test, and it's rubbish, then what? dim: We compare the test to the spec, as the spec is the normative reference. In some cases, the test is wrong, and we would communicate the result to the submitter. In other cases, it may be a duplicate. pc: Mary's DOM demo looks interactive. How will this work? Do I need to look at results as a person, or will this be batch oriented? dim: I think people will want error messages. mb: The tests are separate from the harness, which you saw in my demo. The menu-driven option is an example of one way to use the tests. MS: Do you provide instruction to those who submit faulty tests, to help them make a correct test? DD: Absolutely! ================================================================== Wednesday April 4, 2001 (morning) ---- Experience in certification Andrew Thackrah, The Open Group (TOG) Alex Rousskov, Measurement Factory: what if you have severals releases of products? Andrew Thackrah, TOG: need to identify the version even if you have a new release each month. Paul Cotton, Microsoft: how test the implementations? Andrew Thackrah, TOG: mainly self-testing. the companies are submitting their results to our labs. Paul Cotton, Microsoft: were you evolve in testing? Andrew Thackrah, TOG: not really. Daniel Dardailler, W3C: so you may get back to them and say their statement where wrong. Henry Thompson, University of Edinburgh/W3C: we didn't want to get the implementation reference business. founding? third-parties? Andrew Thackrah, TOG: a separate group is developing that. Daniel Dardailler, W3C: SVG has a "virtual reference implementation" by having bitmaps to compare with. Henry Thompson, University of Edinburgh/W3C: where do they come from? Lofton Henderson, OASIS: from one of the implementation. we needed to check the bitmaps of course and fixed them if they were wrong. Alex Rousskov, Measurement Factory: the vendor submit partial results to your lab. How can you trust the results? Andrew Thackrah, TOG: not sure we met this case in fact. Alex Rousskov, Measurement Factory: so they feed your servers with fake results? Andrew Thackrah, TOG: yes but they are legally bounds. what we're doing is providing a service for consortium. it's up to them to use them. they don't mention TOG. Daniel Dardailler, W3C: what kind of contracts do you have with your members? Andrew Thackrah, TOG: don't really know. Paul Cotton, Microsoft: I want people to understand this is not unexpensive process. I worked within TOG for SQL in the past. It was all about using a brand for marketing purposes, we finally decided to not go there. Andrew Thackrah, TOG: you don't have to participate in the entire process if you don't want to. You're able to pick what you need. John Williams, Exodus Performance Labs: we're also doing some branding in this area for Sun or RealNetworks. We found through third-parties. What TOG does is for a standard from a company. do they bring third-parties? Andrew Thackrah, TOG: don't know. Charles McCathieNeville, W3C: what do you do with Debian/Linux? they can't claim to be conformant? Rob Lanphier, RealNetworks: they can do their own certification. Some vendors are working with Debian. it costs money to do the certification. if the foundation works, they will. ---- View on Conformance Christine Hoffman, Netscape Karl Dubost, W3C: did you contact the W3C WGs to have tests? Christine Hoffman, Netscape: we use the W3C test suites. we also use some more complex external test suites [for CSS]. ---- Quality Assurance Interoperability Conformance Allen Brown, Microsoft Corporation Daniel Dardailler, W3C: How do you approach the WG for comments/feedbacks? Allen Brown, Microsoft: through your contact (in case of XML Query, it could be the chair) or we send comments to the public mailing lists. Rob Lanphier, RealNetworks: once it reachs REC, the WG does no longer have the entire power of decision. needs for an appeal process. Allen Brown, Microsoft: usually, issues are technicals. they need to be resolved by the technicals. Rob Lanphier, RealNetworks: yes, but the WG might fail to obtain the right solution. (laziness) Allen Brown, Microsoft: having a higher authority might not fix that also. Paul Cotton, Microsoft: I would advise the director to not make technical decisions. With the TAG, the situation should be improved. and if something is broken in the WG process, we need to traine the chairs. We need to make the WG works better. Karl Best, OASIS: how long the WG stay in place after the REC. Paul Cotton, Microsoft: depends, by experience it would say 2 or 3 months. Daniel Dardailler, W3C: and after that, you still have a domain leader. Janet Daly, W3C: the amount of time is going even farther than six months actually [...] ---- Experience in a large end-user company Scott Vesey, Boeing Scott Vesey, Boeing: we update our web techniques every six months. We have a document to enumerate the techs that we considered part of the Web (with MUST, SHOULD, MIGHT, MAY, ...) Scott Vesey, Boeing: every developer is a potential web content developer. Robert Neff, uaccessit.com: what I would to have had is "what documents do you need for intranets?". Tried to reuse boeing techniques but nothing public. Scott Vesey, Boeing: guidelines are so specific that you can't meet them in current authoring tools. Daniel Dardailler, W3C: how do you update your tools? Scott Vesey, Boeing: in most cases, I was able to convince the author to update their contents if necessary. I had to roll back a change in only one case. Charles McCathieNeville, W3C: did you study/recommend the tools that your authors? Scott Vesey, Boeing: we're mostly recommending dreamweaver but not based on a formal study. Gerald Oskoboiny, W3C: Scott Vesey, Boeing: we have some test suites to test the products for internal uses and our costumers. We're also developing some internals tools such as bookmarks converter, ... We're writing full products such as editors. We're looking very hard in alternates devices such as wireless devices. Gerald Oskoboiny, W3C: if you find a bug in the product, what do you do? Scott Vesey, Boeing: even if we're a big company, regarding the download rate of IE or Netscape, we're not as big as being able to change a product. Charles McCathieNeville, W3C: get back to the tools thing, do you extend current tools and publish your extensions? Scott Vesey, Boeing: don't have enough resources to do that. Charles McCathieNeville, W3C: even for quick hacks? Scott Vesey, Boeing: we have an internal software release process. Robert Neff, uaccessit.com: to have a content management tool? Scott Vesey, Boeing: doing a web application is really a documentation process. for the most part, it is informal. some rules are very very strict. Karl: how to improve the customer feedback to the vendors? the W3C might be able to improve the feedback. Scott Vesey, Boeing: I pick things that I think are important. I don't send so many bug reports. I tried to describe as much as possible the problem. Daniel Dardailler, W3C: if a page doesn't follow our guidelines, what can the employee do? Scott Vesey, Boeing: one of the guidelines is the feedback. ---- Daniel Dardailler, W3C: we talked about certification but below, the issue of compatibility chart. Informal publication of results of test suites (CSS chart, ...). Should we go there? Any comments? Lofton Henderson, OASIS: SVG published a chart. they were volonteer. Henry Thompson, University of Edinburgh/W3C: the third-parties can do that. Paul Cotton, Microsoft: depends on the license. You also have to say that where the datas come from. Karl Dubost, W3C: SMIL didn't publish their results. Rob Lanphier, RealNetworks: lots of products included in the results are not publically released yet. Peter Wiggin, O'Reilly and Associates: The CSS charts are extremely popular. people are looking for that kind of things. Mary Brady, NIST: David Brownell hasn't been sued after publishing the results on xml.com. Charles McCathieNeville, W3C: the license still authorizes you to revoke your use of the software. It hasn't been used in the past but still can be. Henry Thompson, University of Edinburgh/W3C: like most people, I don't read the license. Does the IE5 license restrict you of publishing results? Paul Cotton, Microsoft: no but the MSDN license does. Since we're releasing faster (Web release effect), risk for products damages are higher. Janet Daly, W3C: receive a lot of requests to point the users to the most conformant implementation. Currently, the W3C is not in a position to make statements regarding implementations. Josef Dietl, Mozquito Technologies AG: if you have the garantee that the results publisher will update his pages, would you change the license? Alex Rousskov, Measurement Factory: the damages have been done. Peter Wiggin, O'Reilly and Associates: that's a publishing business. not need to go in this area. Lynne Rosenthal, NIST: there is some benefits to have these results published somehow. NIST published results in the past but with permissions/reviews from the vendors. It was in fact vendors results. Janet Daly, W3C: did you have unsatisfactory results? Lynne Rosenthal, NIST: we didn't juge the results. Paul Cotton, Microsoft: in net performance, nobody can published TPC numbers with higher approvals. strict rules. the matrix used for measurements is extremely important. the vendors constraint themself. ---- [break] ---- Implementation Experience Thierry Kormann, ILOG, Apache Batik Team Kati Haritos-Shea, NTIS: sounds like usuability (in general) and there are works in the area already. Janet Daly, W3C: yes, Steven Pemberton is organizing a BOF during WWW10. Kati Haritos-Shea, NTIS: you want to find the best to the information as fast as possible, this is usuability. Janet Daly, W3C: internal QAs within WGs to improve quality of specs. Karl Dubost, W3C: I started to review the specs as the same time as Susan Lesch for that purpose. Thierry Kormann, ILOG: the first readers of the specs are developers. Andrew Thackrah, TOG: granularity of tests and results is an issue. Thierry Kormann, ILOG: this is a problem in SVG. differences between OS, fonts, ... so we're using a granularity of results: "failed", "succeeded" or "we found differences". Andrew Thackrah, TOG: an other proposal would be to make an anonimized version of test results. Lynne Rosenthal, NIST: yes you can have others categories. we published some tests in the past not for conformance testing or any particular purposes. Chris Lilley, W3C: anonimized result is a bad idea. It was discussed in the WG and I rejected the anonimized solution. If you don't want see your results in the SVG chart, you're not part of the chart. Finally, the WG agreed to publish the results with attributions. It really helps. The Matrix is updated and contains a date. Josef Dietl, Mozquito Technologies AG: back to the usuability of specs, it is not entirely obvious who the targeted audience of the spec is. if you repeat the fragments in each spec, you'll have inconsistency problems and we don't know how to resolve that. ---- Writing Testable specifications and errata. David Marston, IBM/Lotus Daniel Dardailler, W3C: I agree with what you said except that I don't need an activity for doing that. how is going to educate the WG? David Marston, IBM: I suggest that Karl can do some of it. Janet Daly, W3C: Susan reviews all tech reports but not semantics, only typos and general consistency. If IBM has some tech writer for us, they're welcome. Thierry Kormann, ILOG: how can you find resources to read the specs, make the tests without participating to the WG? You have to interact everyday with the WG to write the tests. David Marston, IBM: My expectation of this workshop is too increase the expectations from the WG. David Marston, IBM: if the WG designate someone to be the point person, it will improve the situation. Josef Dietl, Mozquito Technologies AG: could be the editor. The process for publication errata is to publish them as soon as there is agreement. David Marston, IBM: in my case, the editor knew the resolution of the issue but didn't have the agreement of the WG yet. Susan Lesch, W3C: one point on David presentation is that he represents the opinion a large group of people (50). ---- Building Experience in the W3C Susan Lesch, W3C today webmaster reviews validity, link correctness, wcag level A, copyright proposed editorial review: spelling,grammar, style, examples, conformance, rfc2119 (must etc.), layout of refs and acks, size proposed architectural review: experts, incl TAG, WAI, I18N: requirements, dependencies Appendices include a list of commonly confused terms Lofton Henderson, OASIS: ISO Style Guide? It's very rigid, big, but produces uniformity. Alex Rousskov, Measurement Factory: Why not enforce, like the pubrules? ITO has done this. Susan Lesch, W3C: We could/should make the guide public, and then make it a 'should'. Alex Rousskov, Measurement Factory: Why not a 'must'? Janet Daly, W3C: Adding mandatory requirements when there are 30 WGs writing specs has a real potential for disruption. Education is a difficult task which would take a lot of time. Alex Rousskov, Measurement Factory: So require as from 2002-01-01. Janet Daly, W3C: QA will look at this among other steps. Daniel Dardailler, W3C: WGs would have to agree. Henry Thompson, University of Edinburgh/W3C: 'must' means more staff Lofton Henderson, OASIS: 'must' means an appeals process; what about rules which pertain to testability? Karl Dubost, W3C: Note the conformance section is not yet enforced as mandatory. Daniel Dardailler, W3C: Similarly the normative/informative refs distinction is not stated anywhere as required. Karl Dubost, W3C: Outside reviews? Susan and Karl _are_ outside reviewers. Chris Lilley, W3C: Template or best practice doc't would be an intermediate move. Could be based on spec-prod DTD. Thierry Kormann, ILOG: Second Chris's point, choose the format (XML,HTML,...) Rob Lanphier, RealNetworks: Metoo, plus standard forms for normative/informative, default, Henry Thompson, University of Edinburgh/W3C: Yes, down to quite a low level. ================================================================== Minutes from q&a after dd's qa presentation: Scribe: Gerald Oskoboiny DD, W3C: presents a "W3C QA Scope" slide, includes three circles: innermost: specs dev: assertions, xml, style guide, reviews, spec-prod? (dtd) middle: tests dev: guide, ext coord, common harness, change control outermost: tests use: metrics, certif, comm After presentation of innermost circle, "specs dev": Henry, W3C: have been working with spec-prod throughout my work as spec editor; we wouldn't have a PR for schema without spec-prod; also conscious of the fact that we're not ready to make that the way that all w3c editors produce specs. Committing to this requires a resource commitment. One of the reasons why xml-based spec authoring is so wonderful is the 'x' in 'xml'. Every spec I am aware of that has edited with spec-prod has added productions specific to their spec. If we are going to expect all authors, we need to help those that are not in a position to do that. DD, W3C: yes, there are resource questions. Hope we will talk about funding model etc. Charter will need to include resource commitments. Henry, W3C: agrees with Dave that the XML version should become the normative spec. Janet: ideally I think that is a goal, but practically, given all the different groups and publishing tools we deal with it is a problem. DD, W3C: presents middle circle, "tests dev". Lynne, NIST: think all the items are important, we should try to capitalize on what is already common between different spec development efforts. DD: yes, need a common model for basic things: test input, output, etc. DD: who is going to host test suites? W3C? or main developer of each test suite? Something that needs to be considered at this level (of circle) Josef, Mozquito: see a need to develop the specs and docs that constitute good quality documents. Also see the need for an IG, because there are plenty of questions in QA that do not need an ongoing commitment, bu do need a place to express an interest, a need, and to provide input that is channeled in a reasonable way to the WG and to the rest of W3C as a whole. At the same time, WG/IG have to operate on a high level, independent of a particular W3C spec; ultimate decisions on how to test a given spec should still remain within the WG. An accredited test suite, whether hosted at W3C or elsewhere, is an implementation detail that should be left to the WG (for example). Alex, Measurement Factory: want to caution people, if you are talking about working with external developers, and testing anything besides your favorite markup language, don't thin you can have common rules; examples I have seen do not fit in several protocols that would be interesting to test (or that I am testing already). May be an abstract model for testing, but do not think there is a formal model that fits all the things we want to test. DD: one of the things I started looking at is a taxonomy for various testing. DD: presents third circle, "tests use", intros Janet who presents the process document, how we go about starting a new activity. Katie: asks about usability Karl: ... Charles, W3C: first thing you need to do with a new WG or IG is staff them; once that happens, you get a sense for which direction they will go in. Chartering a WG is not a nightmarish job. DD: we already have a number of people on e.g. spec-prod Janet: note that two WGs and an IG begs a CG. Plh: DOM is invited member of xml cg, so we have one teleconf per week to discuss dom and all xml-related work. Also involves UI domain groups, e.g. xsl, and XML protocol group, which is not part of xml activity. We have a wg list, an ig list, a public list for comments, and public list for dom test suite work. Works well. Henry, W3C: if it weren't for the administrarive overhead, it seems clear that there is quite a difference in the interests and people that will be involved in one or the other. Or maybe we need an activity as well as two groups? DD: i18n and WAI are two activities; we also have system team and comm team at w3c, which perform stuff that are not activities Henry: maybe we need an 'horizontal concerns' activity, that includes wai, i18n, and qa. Dave, IBM: follow Henry's point about abstract desire for two WGs; would point out that there is a fuzzy line between inner and middle circles: test assertions, catalog of test cases, developing test cases; most of it sounds like it's on the REC-track WG, thinking of the test cases. Rob, Real: suggest that one WG and one IG is the right thing to do, and as part of the WG charter, making it explicit that the WG needs to consider the possibility of proposing charters for new WGs. So it's a WG that's potentially swamped the first day of its creation, but the first thing it needs to do is consider what the taxonomy needs to be; would be worse to start a couple groups sooner and get the taxonomy wrong. (a few people in the audience voice agreement) Lynne, NIST: maybe one WG with two projects? DD: we have WGs that have created task forces, but those are temporary; if you don't have a web page, people don't know what's going on. Janet: if you word the activity proposal to include the possilibity of organizing new groups, ... Josef, Mozquito: wanted to address the idea of having task forces; with one WG and the possilibty to spin off task forces, we address several questions in the area of testing; sorting test cases according to the xsl spec is something that has to cooperate with the xsl group anyway, but creation of test cases seems like something that is specific in time: intense work near the pr(?) step, then not much else. Henry, W3C: exactly, and there's lots of precedent for cross-wg task forces; the task force you just mentioned would have been a testing-WG/xsl-WG joint task force. Paul Cotton: need to both do maintenance (Fixing current things) and lead by example; a doc that we can point a chair to to say "this doc must be on the agenda of your 1st meeting" to set the direction. DD: in the middle circle I was thinking of things that improve the quality of documents/specs Paul Cotton: don't mind you having the remedial approach to this, but doesn't think it's sufficient. Also want to make sure that WG spends part of its time worrying about the disease: education of chairs and WG members, better charters, etc; need documents for those kinds of things. DD: yes, that's part of that. DD: regarding the middle circle, what can we do to improve the number and quality of tests? The matrix we presented is a deliverable of that circle Mark, NIST: ... Lynn, NIST: test development... produce guidelines or framework to WGs or external groups would have guidelines/frameworks that they could follow while testing framework. Could partner or work with groups developing test suites; if you're going to be developing test suites, here are guidelines for getting started. Lofton: that's what you mean by 'guide' in the green (middle) circle, right? (yes) Alex, Measurement Factory: not always possible to have this within a single WG, because some protocols aren't covered by a WG, e.g. HTTP: IETF doesn't care, W3C can't care because there is no WG for it. If I have an HTTP test suite that I want to contribute, there is nobody to accept it. The WG that is proposed would solve that problem, and let me make my suite available to everyone through W3C channels. (break) DD: need to discuss the level of confidentiality we want: already have team-internal, member-internal, public. ... In general, W3C is trying to find a very good reason for keeping an activity member-private these days. DD: Anyone in favor of member-private stuff? (gerald didn't see any hands) Lofton: in the SVG WG, test suite work in progress is in a cvs repository with limited access; public doesn't have access to that repository; in your model, would that be public or private? e.g. mail list could be public, and work in progress be private. DD: the group we're talking about isn't. For WAI groups that have documents in development, they havve a concept of a public "release", even when everything is public anyway; if people wander around on the site, it's all there, but the WG isn't asking for comments. Thierry, Batik: my understanding is that all the work of the central WG has to be public, so public can contribute to quality of the spec. Think there is a need for the QA WG to have a way to discuss things with other WGs. DD: WAI PF was created to review member-private stuff. WGs have the obligation to release a WD every 3 months. Given the number of drafts that PF has to review, we choose the stuff that's public, because that is enough to keep us busy anyway. Not important to be granted member site access. But when we want to discuss things with a WG, e.g. in a teleconf, or a cross-list thread, it's useful to have member access. Paul Cotton: anyone who wants to make public comments to a QA group can do it just by sending comments to the public comments list. Would think that you would want to do exactly that. Thierry: what if the document is not public yet? Paul: if it has never been made public yet, group would be giving advice prematurely DD: could send a representative of QA to the member-private WG, without releasing member-private information. Chris, W3C: that gets tricky (reporting back to a public forum) Josef, Mozquito: all in favor of keeping as much public as possible, but it seems like a group like that does not need private web space, but it does need a member-private mailing list: could be talking about products, embarrassing vendors. Also, not aware that every WG needs to be released a public draft every 3 months. Philippe, W3C: ran into these kinds of issues when we created the DOM test suites process document; what we did was to create the test suite work that is completely public, and DOM group page that's member-restricted, and there are no restrictions between then, anyone can participate in either or both. If the DOM test suite group has a question for the DOM group, they have a special channel for that, and a reserved slot in teleconfs. DD: sounds good, but not all groups will do that. Charles, W3C: would back up what DD said about WAI PF: there are no minutes left to read docuemnts that aren't public working drafts; all the milestone ones (last call, cr, etc) are all public anyway; by the time you have worked through all those, the WG is due to produce something in public, so you don't have time to deal with non-public documents. I18n has another model: WG and IG; all the WG members are in the IG, and IG is the default place for doing anything; that's where mail goes by default; if you want to talk about something that does have a member confidentiality requirement, there is a member-private list. Might be a useful model to look at starting with. Might be one of the issues for an initial WG to think about for the first six months. DD: does I18N group have two web sites? Charles: no, both WG and IG are member-private. Paul: close to how xml protocol works Janet: XP has one public list for all technical discussion; private list is used exclusively for chairing logistics, for meeting info, bridge number and passcode, and if there is a matter that comes under member confidentiality. When they first started, there was more of an inclination to use the member-internal list, but that seems to have faded. Public lists can be difficult to keep up with though. DD: public lists don't necessarily need to allow anyone to post directly to the list; can be partially moderated. David, IBM: ?? doesn't even have a private channel. ?, OASIS: technical discussion is private. Karl, W3C: usually there is an email address to a list to make comments; WG usually asks the public for this kind of feedback. Janet, W3C: in charters and documents, the public list is the list for which the working group is accountable. DD: okay, let's do a public WG, (and an IG?) that can spawn off other groups as needed. Karl: need coord with comm, sys teams Paul Cotton: can include that in the charter, under staff resource commitment. DD: talks about test development: do tests need to be open source? published under w3c license? (software or document)? Or different policies for different groups? How about accepting test cases from an external party like a vendor? Alex: open source doesn't necessarily mean non-commercial, and vice versa. My company propduces open source benchmarks, but we own the benchmark. If you want to produce a standard for testing, you don't want everyone in the world to modify that standard and then publish the result. If you want to work with external entities, should be flexible in what you accept, e.g. don't say "must be gpl". Chris, W3C: was going to make the same point: for svg test suite, don't want someone to remove the parts of the test suite that they fail on, then publish the result. DD: what if someone contributes a biased test suite? Chris: if it's reasonable, we'll link to it whatever; if not, not. Josef: suppose you accept an external test suite, then the vendor changes the suite? Change control must reside with W3C in order to make sure it fits with what the spec is saying. One of the conditions of the licensing agreement is: whoever has proposed the test suite agrees not to change it any more without asking the WG or W3C in general. DD: brings up the question of stable resources or URLs in general. If someone suggests an external test suite, the external entity needs to guarantee it doesn't change. David, IBM: solution could be use CVS; WG can bless a certain tagged rev in CVS. Janet, W3C: in terms of the way a spec develops in W3C, there are lots of opportunities for public and member review; WG has an obligation to respond. In the case of test suite development, that same level of accountability and opportunity for review is mandatory. People will see the test suite as the natural complement to the spec, regardless of where it's created. DD, W3C: the way we lay out the tech reports is that they always start with "this version: ...", "latest version: ...". Question I have for Janet is: is it possible to have "this version" uri on w3c's site, with "latest version" pointing to a developer web site? (e.g., nist, or someone else). So the public can trust a document they see as being endorsed by the WG, but if they want to check the latest one, they can go to the developer directly. Janet, W3C: if it's not a w3c uri, I think it creates confusion. Developer might hit "latest version" without reading carefully. We want to encourage users to use the endorsed version. We have never done that in the past, even for acknowledged submissions. David, IBM: we need to start at the top: trying to get more quality in the software and in the specs; workingt down, the next step down is people must be encouraged to write test cases over time; therefore there is a value in telling people who have a deep interest "you can still write tests", and go over here to see what has been done since the last blessed version. So maybe it's time to change that rule about W3C URIs in the interest of quality. Janet, W3C: important for that to be coordinated with the WG. The WG always hears about it sooner or later, no matter where it originated. David: would only be pointing outside if they are granting that outside org some level of trust/credibility; WG wouldn't be doing that for anyone who happens to submit tests. Mark, NIST: procedurally, I see the blessing of a test suite to be similar to certification of software; you test a few times, if it isn't correct you have a few chances to fix it, then retest etc. Re souce code, I'm not concerned about people changing it as long as there is traceability. In our tests we have links from assertions to tests to source code. Philippe, W3C: in DOM, what we did was instead of saying W3C must host the tests, W3C must host a mirror. Two issues are: being able to access the test suites, and... Always say that if W3C is not able to provide the resources we need, we will find another site, and W3C's copy will only be a mirror Rob, Real: re assignment of copyright, ... Karl, W3C: specs are copyrighted, but you can make a copy on another web site, with a special text at the site saying "this is not normative" Rob: Alex: wouldn't give you the copyright; I invested 5 years in this test suite. Henry, W3C: could be a problem; so far, we have said that sofware and doc copyrights are the basis for test suites; not necessarily cast in stone, but that's the starting point. In schema tests, companies have all gone to their legal depts to make sure those terms are okay. Karl, OASIS: OASIS IPR is similar: people can donate; any work that's donated must be unencumbered by licensing or patents etc. DD: Can W3C make a snapshot that is not covered by W3C copyright? Rob: seems risky to have something on w3c's site without having change control DD: reason for w3c to make a snapshot is to say "we have endorsed *this* version" Paul: what if you're wrong DD: then you change the page James? (two rows up): ... DD: Josef: is a potential for a de facto migration of content from the test suite to the spec [did he mean the other way around? -gerald] Therefore, whatever you do in terms of licensing is okay as long as the WG has a say in change control. What is important is that the WG also sees feedback about the test suite; if the test suite is hosted somewhere else and the WG doesn't know that some people have issues about the test suite, that could be bad as well. wrt having a snapshot of someone else's resource, if it's not under the doc and software licenses, we have been through that before: if someone wants to make a snapshot of the whole site, they won't be able to (while they can now.) Janet, W3C: in terms of specs and copyright management, there is one class of documents that carries its own copyright status: the result of a submission. There's no review by a staff member, no editorial control by W3C; submitter may include their own copyright and ipr. Within W3C, we go to pains to emphasize that these are not W3C specs in the REC track sense; W3C is not responsible for revisions, etc. Authors might mention within the document how to provide feedback elsewhere. In the case of test suites, both the spec and the test suite depend on each other; they are by definition siblings, so the idea of carrying a copyright that's outside might be problematic when it may be seen as essential for the implementation of a spec. Alex: you can build whatever rules you like into this model; the only control I am looking for is control of the driver. cf. Perl's license: I want to control Perl, you can write whatever programs you want (cf. test cases). I am sure you can put reasonable terms into the license that accomplishes this. Chris, W3C: for SVG we separated the test cases and the harness: you want to be able to link to the next test case, but don't want to make it difficult to insert a test case. Was originally using embed tags, but I said you can't use that on our site, so we switched to frames, then we switched to SVG for the whole harness(?), and all the code for this is in our distribution.(?) Useful distinction to make between the harness and the test case. DD: issue of funding: Karl: I will come around the tables with a plate to accept offerings DD: ... describes offices, various funding stuff; Janet mentions W3C fellows. Paul: actually, taking the plate around (i.e., sponsorship) sounds good; large vendors are the ones that benefit the most, and they should contribute more. Would be a strong supporter in trying to generate such support where I work. DD: thanks for coming, don't forget to send slides Janet: thanks to NIST (applause) Karl: you can continue to contribute on the public www-qa list. DD: we'll send an email to each of you with instructions on how to subscribe =============== END