ERT
WG
24/25 June F2F Summary
Nadia Heninger, Ian Hickson, Liddy Nevile, Nick Kew, Jim Ley, Wendy
Chisholm, Max Froumentin, Nick Gibbins, Dan Brickley, Libby Miller, Charles
McCathieNevile, Bert Bos (Tuesday only)
Via phone and/or IRC
Marja-Riitta Koivunen, Eric Prud'hommeaux, Giorgio Brajnik, Sean B.
Palmer
Summaries of Issues
- Similar to confidence axis
- possible enumerations: pass w/minor bug, maybe, no, buggy, destroys,
crash
- A black and white pass/fail is not enough, but ok if what pass/fail is
granular enough.
- Augmenting auotmatic evaluations w/human assertions. Reusing manual
test results.
- Combing results from different tools. I want X from Bobby, y from
AccVerify and z from Page Valet.
Is the following possible to assert in EARL?
I cannot tell if test #53 fails until I run test #63.
- Need to be able to say that X atomic tests add up to checkpoint 1.1.
(If one failure, all fail.)
This issue has two parts:
- test suite (test cases)
- large amount of earl data to test and play with
What needs to be done:
- identify tools (to generate EARL)
- find a place to store the data
- determine what to test against
Start with:
- Josh and Chris' tests (too specific, but a good starting point)
- Live sites
This will tie in well with WCAG test suite work.
Also need:
- manual examples (MUTAT)
- other use case data
- implementation reports
Tools to generate data:
- AccVerify
- MuTAT
- Page Valet
- LIFT (eventually)
- Need DAML+OIL? (the following proposals get rid of reliance on?)
- Split schema into 2
- proposal to get rid of rdf reification names
- proposal to use bnodes for the subject
- nmg's proposed change for results (gets rid of subclasses of
properties)
- proposal to use a constrained RDF syntax (with a DTD)
- proposal to have a simple and a complex version, mappable with XSLT
- Point to test cases not specs.
These proposals had support among everyone in the room. WAC took an action
to discuss with SBP.
- Added several use cases
- Added queries for each use case
- Deleted use cases for
- Explain camelcase (class/Property, RDF convention)
- Danbri suggested we look at Dublin Core "instance documents" and the
RSS documentation for ideas for documentation (good instances of teaching
by example).
- Dave Pawson suggested we create an XML DTD to help people understand
what an instance of EARL would look like. He created one, then we spent
time updating it based on the schema discussions.
- Dave Pawson - send info about ATLAS
- DanBri, Libby, Charles, Wendy, Nadia and Eric (did we
volunteer eric or did he agree?) took an action item to figure out and
create a generic RDF database to store the EARL results (test
corpus).
- Danbri - Once publish EARL 1.0 to TR, solicit review
from RDFIG (and other communities).
- WAC - once publish EARL 1.0 to TR, solicit review from
QA.
- Dave Pawson - agreed to be "guinea pig" for reivew of
spec (as an XML developer).
- NH try to re-order the spec to be clearer about the
role of different pieces (completed it during the meeting @@LINK)
- Nick Kew - write up HTML pointer documentation
- action wendy take latest back to reagle for discussion/ACTION: Wendy -
investigate XML signature technique (xpointer, refer to discussion on 24
june 2002 14:04:41)
- WAC - look into the Dublin Core "instance documents"
and the RSS documentation for ideas for documentation (good instances of
teaching by example). (MaxF is interested in this, DanBri said he would
send references).
- Charles look over the use cases and see what features
they need to use.
- nmg write up list of questions about schema
- wendy to talk to sbp about changes in schema
- DP, CMN, JL discuss TestSubject (discussion begins
~15:21:23 25 June 2002)
- ih: send something to the list re: severity (define
it)
- wac define operator ala prev discussion.
- wac: email dtd (with changes) to the list (done)
- WAC take the issue of Test Point Definition Language
to QA.
$Date: 2002/07/24 16:27:14 $ Wendy
Chisholm