Meeting Agenda· 2001-11-12· 2001-11-13·
The Matrix Review· Process WG Life· Documents schedule· Teleconferences·
ACTION Karl: bring up issue of DOM validation with Philippe Le Hégaret and/or DOM WG.
ACTION MaxF: find out if there is another MathML validator besides http://validator.w3.org/
ACTION Karl: ask the XML Core WG to review XML Fragment entry in the Matrix.
ACTION Karl, DD talk to XML Coord Group about XLink
ACTION Karl: update Matrix entry for XML Schema test suites: change from text to icons-with-alts, add a link to W3C's test suite.
ACTION Karl: update Matrix entry for XML Schema test suites: change from text to icons-with-alts, add a link to W3C's test suite.
ACTION Karl: ask Joseph/dsig WG how to validate an XML DSig file/tool.
ACTION Karl: talk to Bert about testing XML stylesheet linking
ACTION IJ: write a proposal to the chairs list that includes an errata process (see below)
ACTION DD: bring to w3m the issue of life after REC for errata
ACTION LH/KG: organized collection of technical data: fill in technical guidelines, start collecting real-world data, sort by taxonomy (e.g., test case description language), testable assertions, what does this test suite/validator do. KG to provide his materials/data for technical framework to LH
ACTION DD/KD: write QA glossary; deadline: mid-December
ACTION KD/DD: write Taxonomy; deadline: mid-December
ACTION OT/SL: write Generic term glossary; deadline: 3 months
ACTION LH: reissue Framework/Primer; deadline: mid-December
ACTION DD: check with other groups to see how that model works
ACTION Lynne: send what she has about certification to the WG list, DD will take it from there.
Scribe: Rob Lanphier
(Agenda bashing)
Daniel: Separate document that becomes recipe
Lofton: Presents recap
(FPWD == First public working draft)
Corresponding example and techniques documents
Open question: where does example charter (w/requirements for test suites, etc) live?
Lofton: Operational E&T can have a detailed look at DOM 'TS Process' -- one of best developed examples of a TS process document.
Ian: This is a moving target
Open question: where should the requirement for having a test suite be placed?
Marc: Sounds like a pretty onerous requirement
Lofton: Only needs to happen...doesn't matter who does it.
Karl: Pretty tough to exit CR without test suite?
Lofton: Should this be a requirement?
Ian: Different classes of specification, such as TAG recommendations
Ian: Requirement is already practically imposed, due to TBL's anecdotal opinion.
Kirill: If we raise the bar in our process document too high, WGs will not put in their charter requirement to conform to QA framework. We should provide set of options to conform to at discretion of WG
Daniel: Should have framework document which provides a menu of options
Marc: Test suite can be in gradations
Lofton: A breadth-first TS should be the minimal requirement by end of CR.
Lofton: Is this something we try to move forward for the Process Document (capital "P")?
Ian: No, because 1. probably not necessary 2. No WAI or I18N requirement
Rob: Should we make the choice more clear?
Lofton: Review of working group charter will expose TS plans to explicit review.
Ian: That makes sense
Daniel: Really needs to be activity proposal, not draft charter.
Marc: Should there be a requirement?
Ian: Not a requirement
Marc: What possible exceptions could there be?
Ian: Some urgent situations may require looser requirements. However, the direction is toward very strong requirements.
Marc: We can still make things loose without backing down on this requirement.
Lofton: How do you demonstrate interop without this anyway? Most groups have made substantial changes in CR
Lynne: Important point. The spec has had a lot of bugs shaken out; this is a way of vetting.
Ian: Test suite is the best practice for getting out of CR, ergo should be started early.
Karl: Does it make sense to create a feature without having a test? Should have test cases as features are added.
Lofton: Breadth first gives most bang for buck anyway
Ian: Should we have every instance or just one? Should UAAG be for HTML, or should multimedia as well?
Rob: Ian's example of UAAG is good, because tests on HTML don't necessarily prove implementability of spec in multimedia.
Lynne: It all depends on goals. Checkbox compliance where easiest is goal, sometimes you may want to seek out corner cases to vet out spec. In any case there needs to be test assertions.
Daniel: Is quality of spec more important that test suite?
Marc: Two are closely interrelated
Daniel: If WG decides to create a test suite, the specification could be bad, and the test suite is good.
Rob: Agreed. Test suite could be treated as sole, normative specification of behavior by WG.
Karl: Maybe basic effectivity suite can be requirement
Kirill: One test per assertion
Lynne: May exceed basic effectivity
Rob: Should approach AC
Marc: Don't like idea of pre-compromising
Ian: What was cost of SVG effectivity test
Lofton: Around 2000 hours total. I put in 700 hours
Ian: Problem is that these things are slow, and perception is that it'll be made slower.
Lynne: 3-4 months spent on XSL which were great investment. May seem slow to some, but incredibly fast by QA standards. Several good examples of WGs that have done right thing.
Marc: New slogan: "It's not that terrible"
Daniel: Others need to understand that we're not here to do their test suite. Need to come up with sponsorship opportunities for QA. This would fulfill the needs that some may be expecting out of this group.
Ian: Where does OASIS get money?
Lynne: No sponsorship
Daniel: Outreach for sponsoring is easier
Karl: Risk is that member will try to use this to route around process.
Rob: Skeptical that unbounded sponsorship would fly.
Daniel: How about pool of engineers
Lynne: NIST is willing to put this up. University projects, perhaps internships.
Rob: Interships very good idea, needs following up
Kirill: Market will provide the incentive. If standard doesn't get the test suite from implementors, it means it doesn't have support from implementors and no real implementations, that means it can not exit CR.
Kirill: Support Lynne's idea of universities being involved in test suites creation
Karl: Problems with releasing test suites and with releasing results
Lynne: Once one company contributes, they all fall in line. Process for doing this helps.
Daniel: We need a document that talks about:
Daniel: Business case: education and outreach for QA. Is this different than WAI?
Karl: Example: WaSP is trying to get Macromedia to produce good code. However, community doesn't know to come to W3C anymore, because emphasis is on implementors and not users.
Daniel: Are education materials on risk of not using specs?
Karl: Needs to be pressure on companies to actually conform.
Rob: Perhaps the way that outreach works is by making the test suites very accessible.
Daniel: Is this the job of QA activity or our Comm team
Ian: MarComm is resource constrained
Daniel: Needs to be coordinated with MarComm.
Ian: WAI handles it by Judy working down the hall from Janet.
Daniel: Certification -- how does W3C interact with external certification groups. Open Group paper addresses this.
Karl: Do businesses allow people to see the test suites?
Daniel: We'd like some control over the process.
Daniel: Open Group wants to certify XHTML.
Lynne: Experience with Open Group for POSIX. They do hybrid private/public testing. Perfectly willing to use public suites.
Daniel: Model is clear. W3C writes test suite, anyone can use it.
Kirill: W3C isn't guaranteeing that the test suite is valid.
Rob: What will the W3C do in the case that someone publishes results?
Daniel: If a business evolves around providing certification, what should W3C do?
Lynne: NIST has experience developing criteria to be used by Certifying Authorities in recognizing testing laboratories. NIST developed the procedures and provided the recognition criteria for the ATA to use in their CGM certification program.
Daniel: We could provide recommendations on the honor system.
Kirill: Should they get permission to publish benchmarks?
Daniel: What we're saying what the test suite does and doesn't do, and provide guidelines for how to publish. Need to provide EARL.
Lynne: Must be careful with failure reports, because implies qualitative judgement.
Kirill: We can't let them say that something conforms
Lynne: We can provide suggested wording, but we can't mandate any beyond that.
Lofton: What about the WAI certification?
Ian: We facilitate unverified claims, but don't verify claims
Ian: We should say: "passing a test suite does not imply conformance. In general, failing a test suite implies non-conformance."
Lynne: Should we provide guidelines for how to set up a certification service?
Daniel: Charter says certification is out-of-scope, but setting boundries is in scope.
Lynne: Plan on putting out "how to interact with cert authorities"
Kirill: MS contributes test suite, we wouldn't like any publications that present results by incorrectly running W3C tests against MS products or other products.
Rob: That's a part of being a good-faith member in the process.
Kirill: The test suites are there for feedback
Karl: What was rationale for SMIL test suite having anonymous results
Rob: 1. RN hadn't released RealOne Player publically, 2. No one committed to updating results.
Karl: What is status of SVG?
Lofton: SVG test suite is in third iteration.
Lynne: Working group decides what the result publication rules are. QA WG can provide recommendations though, and it's in the scope of group.
Daniel: We can provide guidelines for publications
Lynne: Very helpfulScribe: Gerald Oskoboiny
Resumed for the afternoon at 13:30
discussing sec 4. Technical Framework of Lofton's QA Framework document sent to www-qa on Nov 6, 2001
Discussing various validators/tools to see what common themes there are that we can make into a framework.
DD: SVG validator is missing from the Matrix; is that handled by the HTML validator?
GO: I think so
...
DD: would like entries in the Matrix to be complete; if not applicable, should say so
possible to do DOM validation?
discussion/disagreement
ACTION Karl: bring up issue of DOM validation with Philippe Le Hégaret and/or DOM WG.
ACTION MaxF: find out if there is another MathML validator besides http://validator.w3.org/
DD: each of these validators should have an entry page that explains what is being checked, even it's just an interface to a common validator on validator.w3.org.
P3P, PICS, PNG, ...
possible to make a PNG validator? Probably, but not clear it's needed.
DD: Do we want validators for all specs, or only those likely or known to have problems?
RDF: working on test suite, to be published as a NOTE?
DD: should add a column to the Matrix to keep track of these issues.
In WG relationships, should ask all WGs to review their entries in the Matrix and inform us when changes are needed.
SMIL: has a validator run by CWI; might be done by validator.w3.org as well.
SOAP: at WD stage, and are working on a test suite
Kirill: there are a number of validators for SOAP, problem is that many people support the NOTE, not the WD.
URI: test suite for python posted to www-qa
WCAG: have evaluators, working on tests as part of ATAG?
WebCGM: have commercial validators
Lofton: released one to CGMOpen members, will be released to the public in early December. Plan (released Mar 2001) is on the CGMOpen web site.
In the Matrix, should we only reference tools that are publicly available? (e.g., HTTP)
If there are more available, or only ones that are not public, should be included in the QA log, not directly in the Matrix.
Lofton: for SVG, might want to add Tiny and Basic profiles to the matrix.
Karl: that's a question I have: should we include WDs in the Matrix? Currently include REC, CR and PR.
DD: or only include those that do QA
XHTML: ...
XForms: currently in last call, will be included in the Matrix when it goes to CR. They don't plan to do a test suite.
DD: should have an entry for everything that is being worked on. Let's say Last Call goes in systematically, and anything before that goes in if they have test work going on.
XML Fragment: not much is known, should ask the WG about that,
ACTION Karl: ask the XML Core WG to review XML Fragment entry in the Matrix.
XLink:
ACTION Karl, DD talk to XML Coord Group about XLink
XML: several freely-available validators, should point to them from the QA log.
XML Base: ...
XML Infoset: validator n/a, same for test suites. XML Infoset rec currently describes an abstract model. Once serialized infoset becomes a standard or part of the infoset rec, there could be a test suite for serializers/content validators.
XML Schema: "has everything I can imagine" -- dd
ACTION Karl: update Matrix entry for XML Schema test suites: change from text to icons-with-alts, add a link to W3C's test suite.
XPath:
XML Signature: interop report, no test suite or validator. No conformance section in spec. ("implicit conformance clause": need to conform to all the MUSTs, ...)
ACTION Karl: ask Joseph/dsig WG how to validate an XML DSig file/tool.
XML stylesheet link: could use a small test suite
ACTION Karl: talk to Bert about testing XML stylesheet linking
XSL: validator n/a? re test suite, W3C's copy of NIST's tests is a static snapshot; WG isn't chartered to do further work on it.
Brings up issue of life of test suites after closure of WGs. Similar to other issues (e.g. spec errata)
Discussion of errata management, publishing official errate lists vs republishing specs with errors fixed.
IJ: suggest bringing this up on the chairs list.
ACTION IJ: write a proposal to the chairs list that includes:
ACTION DD: bring to w3m the issue of life after REC for errata
ACTION LH/KG: organized collection of technical data: fill in technical guidelines, start collecting real-world data, sort by taxonomy (e.g., test case description language), testable assertions, what does this test suite/validator do. KG to provide his materials/data for technical framework to LH
resolved: next f2f meeting (of WG) will be Friday March 1, 2002. (week of all-group meetings, http://www.w3.org/2001/07/allgroupoverview.html )
ACTION DD/KD: write QA glossary; deadline: mid-December
ACTION KD/DD: write Taxonomy; deadline: mid-December
ACTION OT/SL: write Generic term glossary; deadline: 3 months
ACTION LH: reissue Framework/Primer; deadline: mid-December
how to maintain docs? use xmlspec dtd? no.(?)
4th document: input: Lynne's pdf file, Marston paper, Batik paper. Turn it into a W3C NOTE style document, then circulate to www-qa list, then to chairs. Volunteers? Lynne, with help from Lofton, coordinated on qa-chairs list; Lynne will publish updated version within two weeks, ...
Who is on the lists? www-qa has people who were at the Apr 2001 workshop, plus people who just subscribed. (41 people as of today) www-qa-wg list just has DD, KD and OT so far.
resolved: going to try to use the WG list only for logistic stuff: teleconf info, regrets, minor revisions to specs.(?)
ACTION DD: check with other groups to see how that model works
Certification: Lynne wrote something a while ago, very general. ACTION Lynne: send what she has to the WG list, DD will take it from there.
Education and outreach for W3C standards: ...
Chairs have had a few already; do we want to extend them to anyone on the WG? Yes, but not anyone on the IG.
Resolved: teleconferences will be Thursday 4pm Boston time; first one will be Thurs Dec 6, next one Jan 3, 2002.
Adjourned around 17:10.
Last modified $Date: 2011/12/16 02:56:41 $ by $Author: gerald $