See also: IRC log
SAZ: Welcome. I am Shadi
DR: I work for Segala
CMN: Work for Opera Software
KB: Karima Boudaoud, Associate Professor, Uni of Nice, working on evaluation tools...
DS: Dirk Stegemann, Fraunhofer GMD, work with Carlos Velasco
JK: Johannes Koch, also. We are doing some evaluation tools
CI: Carlos Iglesias, CTIC Foundation works on W3C technologies, we have a tool called TAW http://tawdis.net
CMN: Also vice-president of Fundación Sidar, who make Hera http://www.sidar.org/hera
SAZ: We have a close relation to things like Mobile Web Best Practices, and some other groups
SAZ: Lots of things to talk about...
... Start with scope?
<shadi> http://lists.w3.org/Archives/Public/public-wai-ert/2006Feb/0040.html
SAZ: people, look at it because it was only sent on Friday
CMN: Content labelling XG is doing exactly this bit of work, and I suggest that we work with them rather than doing this on our own.
SAZ: They are not Rec track, so how do we work with them?
CMN: In terms of Rec track, we can take what they do and bolt it in straight away.
SAZ: Right. Question is to make sure we understand better what they are working on, and make sure we don't duplicate it needlessly.
<scribe> ACTION: Chaals to check the focus of the XG and see how it relates to this question [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action01]
<scribe> ACTION: David to check the focus of the XG and see how it relates to this question [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action02]
SAZ: How do they label?
... was hoping to sort this stuff out here and push to last call
SAZ: Location, Scope
JK: Two kinds of location - where is something on the web (where we were using dc:location), and where in a resource the particular assertion is relevant to
SAZ: Evidence, methodology
... noting that TestCase and TestRequirement changed
... conformance section, extending EARL (conformance will be a big piece).
... anything else missing for last call
CMN: Think we could go without scope, evidence,
methodology if we really want.
... don't think there is any more there.
SAZ: We changed the term at last face to face, but now we think we need both...
CMN: Unconvinced by the need for it...
DR: Testcase requirement and test case are commonly used in testing evidence and methodology are not common terms.
SAZ: evidence and methodology I want to get
into later on. TestCase and TestRequirement are a bit different...
... basic problem is two vendors checking the same requirement but doing
different actual tests. How do you identify this?
... think that this will really help.
... had used OWL to describe how a TestRequirement is part of another broader
one.
<scribe> ACTION: Chaals decide whether he is going to object to this today, or drop it... [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action03]
CMN: Went through the exercise, and was happy with OWL. Would rather have this as OWL/DC than built into EARL.
SAZ: For me it was easier to do it. Where the
information is not public, it is better to have something in the language.
... people are prone to mixing up the terms, and mess up their reports.
CMN: Will probably just make a formal objection, so it can run along in the process.
SAZ: Will be interested to see the proposal
CMN: Same as always - passing tests X,Y and not passing Z implies passing test Bar...
SAZ: What does earl:mixed mean?
... had thought it was for blanket statements - i.e. you aren't clarifying
how each of the individual tests were done.
... mainly for conformance claims in the large.
... ("my website foo conforms to big specification bar...")
... do we want to promote such blanket statements? Leave it for RDF-CL?
...
CMN: Not a great fan of it, but if it means what Shadi said above I am fine with it.
SAZ: There are manual and automatic tests, but there are also tests where the tool does something to help the user answer a question.
JK: It is a question about what is the test. If only the deciding part, or finding the instance as well. it isn't a manual test if teh tool is finding the instance and asking the user to decide (or vice versa)
SAZ: For those semi-auto cases mixed is not a good term
KB: Yes, that is what we use
JK: You were talking about a collection of tests.
SAZ: Think we wanted to cover both cases...
... think we should have earl:semiAutomatic, and something to cover the case
of blanket collections of assertions
DS: Two cases - auto and manual. You have searching for a relevant place, and deciding. it is important to be able to say which bit is done manually, which bit automatically
CMN: If we have agreed the meaning Shadi gave
for mixed, then we have enough with what we have in the draft
... that leaves the question of who does what in a given test to the
description of the test.
KB: Think we just need to identify the manual/auto/semi-auto cases
SAZ: Agree - we don't need to do that. We could use mainAssertor / helpAssertor to identify who did what.
JK: May be important for a test case desccription language, but not necessarily for a reporting language.
SAZ: [seems like we have had this discussion
before...]
... Do we want earl:semiAutomatic?
CMN: Not necessary. We have compound assertor that gives us even more flexibility
JK: If we use the assertor, we don't need to have this mode. If the assertor is a person, or a tool, then it is clear, so we don't need manual or automatic
SAZ: If we can't give a test mode what do we
use mixed for?
... questioning test mode as a whole. Except earl:heuristic.
CMN: there is no way of knowing that something was heuristic if you don't say it was manual/auto. manual, auto, heuristic are mutually exclusive
DR: mixed is a mess
SAZ: right. For semi-auto case, it is easier to query the mode than look in the assertor, but you should look there.
CMN: earl:mixed I have mixed feelings about (it is "dunno") but we should have the other three
KB: Don't think it makes sense.
CMN: Think we should have it, but not strongly attached to it.
CI: Think it is not necessary. It is some unknown mode. If it makes it easier to check whether the mode is known, it could be useful, but that is the only reason to have it.
SAZ: In a blanket statement the test mode is known. "unknown" is not quite accurate. "notAvailable"?
CMN: Don't care what you call it. Just don't change it again.
SAZ: mixed was a bad name - caused confusion
CMN: Right. Without an agreed label and description, we are likely to go round in circles.
SAZ: earl:semiautomatic? If we have a testMode to query it is easier. Do we want to introduce earl:semiautomatic?
CMN: no.
CI: Same reason to have semiAutomatic as unknown/mixed.
CMN: changed mind - think it is OK to have it.
SAZ: should reconsider test mode, or provide it all.
CMN: So what is the description / label.
<scribe> ACTION: Chaals propose label/description for mixed / semiauto [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action04]
[proposed: earl:mixed s:label "not available" ; s:comment "There is no detailed information about the test mode available".]
[propose: earl:semiauto s:label "Semi-automatic" ; s:comment "The test was done by a person or people, assisted by an automated tool"]
--break
<drooks> --restart
<drooks> SAZ: location class - dc:location does not exist in rdf. need to define our own location
<drooks> JIM: non web content needs location?
<drooks> SAZ: dc:location defines specific URI. need to introduce EARL URI
<drooks> SAZ: EARL:location - label = web URI, comment = a URI to find this on the Web
<drooks> ACTION: chaals to create EARL:location class [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action05]
<drooks> SAZ: how we describe same result multiple times?
<shadi> http://lists.w3.org/Archives/Public/public-wai-ert/2005Sep/0007.html
<shadi> http://lists.w3.org/Archives/Public/public-wai-ert/2005Sep/0012.html
<shadi> http://lists.w3.org/Archives/Public/public-wai-ert/2005Oct/0002.html
<drooks> JIM: range location not useful
<drooks> JIM: single pointes should support range
<drooks> ACTION: range location should be dropped [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action06]
<drooks> CHAALS: showdow location is a location within a compound location
<drooks> ACTION: change shadow location to property of location [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action07]
<drooks> CHAALS: earl:pointer class needs to be defined
<drooks> SAZ: do we need to define default locations?
<drooks> JIM: create a pointers namspace?
<drooks> JIM: need to define xpointer
<drooks> SAZ: also define xpath
<drooks> SAZ: and xquery
<drooks> ACTION: JIM: write up location types - linecharlength, xpointer, xpath, xquery... [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action08]
[so we have:
earl: location a r:Property ; s:label
"location" ; s:comment "location of the result within the test sbuject" ;
s:range earl:Pointer .
... Pointer a s:Class ; s:label "type of location" ; s:comment "A generic
class for a type of location. Locations used in earl reports should be
described using subclasses of this class" .
... SingleLocation s:subClass earl:Pointer ; s:label "Single location" ;
s:comment "A single location. This may have a number of location properties,
using different types of Pointer, that point to the same location."
... shadowLocation s:subPropertyOf earl:location ; s:label "secondary
location" ; s:comment "This can be used to describe something which is not
where the error is manifested, but is relevant - for example a script which
is responsible for causing the manifested result, or otherwise imnportant to
understanding it." .
... CompoundLocation s:subClassOf earl:Pointer ; s:label "compound location"
; s:comment "this class is used to refer to multiple locations which together
manifest one result. The locations may refer to different single locations"
.
]
<drooks> ACTION: JIM: write up location types - linecharlength, xpointer, xpath, xquery... [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action09]
<drooks> ACTION: JIM to get in touch with annotea & check on namespace for earl:pointer [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action10]
<drooks> CHAALS: methodolgy should be a pointer e.g. this is how i did this
<drooks> CHAALS: methodology use case is within evidence
<drooks> CHAALS: evidence is collection - bounded list of things used
Evidence and methodology postponed until later when chaals returns
Topic EARL Conformance
SAZ: What does it mean to conform to EARL?
... What is the minimum?
... Should we tweak it to be more required/less optional ?
... Does it make sense to have more than one assertor, or more than on
subject?
... simplest thing would be to say there must be 1 of each of the things,
that would be really verbose
... we trade of shortness for ambiguity
DB: Why have testSubject aswell as webcontent/
SAZ: So we can talk about things wider than just web content - webcontent inherits from testSubject
DR: Why would you have more than one webcontent?
JK: web-content is just one resource, so you could have tested more than one resource at the same time
SAZ: Currently the only required thing for a
testSubject to have is a dc:Date
... everything else is optional but user agents should understand.
JK: for multiple test subjects you should repeat earl:subject to do them more
CI: You can also do that with multiple dc:location or earl:URI in a single test:Subject
SAZ: Yes so this is similar to earl:Location we just define the container and people are free to widen it more, that limits testSubject to one instance per report.
JL: I don't like the idea of just a single
testSubject as it limits serialisations more than necessary.
... e.g. when you're testing 100 images in a page and they all pass
SAZ: so agree on only 1 Asserter
... but more than one TestSubject is okay
CMN: Requirement is Required
SAZ: Yes one must be required and requirement seems to be the obvious one
JK: There are tests that apply to several requirements.
SAZ: Let's not get into exactly one.
JL: What does testcase need to be?
CI: it should be very flexible
SAZ: I need to be able to recognise that both A and B have tested the same thing
CMN: no test case, just requirement.
SAZ: I propose requirement that's required, and an optional test case
CI: that is how testers work today
CMN: It's not clear that passing or failing
something means when you've really passed the requirement
... so test case is redundant.
DB: You cannot say anything about pass/failing a test case if the pass/fail is on the requirement.
JK: So if 1 of 5 of a test cases or 4 in 5 both of which result in you failing the requirement is the test as failed.
CI: usually if the test contains 5 atomic testing you always have to test all 5 again even if they succeeded before.
CMN: but in the real world that's not how it works.
CI: but if the tests aren't atomic they should be seperate test cases.
CMN: but we aren't telling people how to design tests
SAZ: Is it true that if a requirements succeeds then a test succeeded?
CMN: No.
... if requirement was the pass/fail then people would seperate tests out
into individual testCases.
JL: Requirement is the publically agreed label,
testCase is what you individually did to meet that.
... So seperate testCase/requirement is fine and one of them are required
with the requirement linked to the testcases by heuristic methods.
CI draws picture
SAZ: A tool can only optionally expose what
tests it did to meet a requirement
... describes model with everyones agreement in CI's drawing.
CMN: You must have a direct record of things you pass or fail to re-enable re-use of test result data
CI: If you have a test case for WCAG 1.1 and
have 5 atomic tests within it, then it doesn't matter which you pass or
fail
... Good test cases should be atomic.
... So there is always a 1 to 1 match with tests.
SAZ: I think your testCase is a single use case.
CI: if we drop testcase, then we drop all the other information too
CMN: You use the same methods to describe
"supporting xhtml 1.0" as a "particularly crazy part of the individual
spec".
... With manual testing we don't want to throw away results if they don't
change because we do something new
... we still allow a WCAG 1.0 requirement to describe that it's built up from
atomic tests
DB: You have to have a requirement to test against
CMN: We disagree here between testing and test restul reporting
CI: I want a class for requirements that only points to requirements, and a different class for test cases.
SAZ: What breaks for you in this model?
JL: What breaks for me is that I don't know what to present a user as most important information so like requirements distinct from test case.
KB: Passing/failing individual tests is more important than requirements, so if we keep both, both should be required, but if we choose testcase is more important
SAZ: This is tough and depends on the user.
... different groups have different granularity
CMN: I don't know how this meets the testcase/requirement model
SAZ: I beleive there should be a top down approach from requirements to tests
KB: Does it make sense to be able to have the choice
JL: TestCase is just a subClassOf Requirement
SAZ: so requirement is a wooly public, testcase is more specific
KB: So can a requirement be multiple test cases.
CMN: What a requirement/testCase is completely
open.
... I like the proposal for subClassOf to give us the distinction.
SAZ: The test requirement and the test case are the same.
CMN: A test case is always a type of requirement whereas not the reverse.
CI: I don't agree they are the same
CMN: earl:PassOrFailable be a super class of earl:Requirement earl:TestCase so that we completely seperate the idea that they are related.
DB: Why can't we make them a subclass assertion?
CMN: Can't be done assertion has lots of things.
SAZ: we now agree that testcase/testrequirement should be one property
JK: Could we just have a property called earl:test and an earl:Test class.
CI: so what we need is a method to distinguish between our tests, and our customers requirements
<scribe> ACTION: CMN to write up proposal for subclass test - due 2006-02-27 [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action11]
<JohannesK> scribe: JohannesK
SAZ: back to conformance
... Assertion: exactly 1 assertedBy, 1 or more subject, 1 or more
testThingy
... exactly 1 result?
... exactly 1 mode?
... and mode required
JJ: not more than 1
<chaals> [exactly 1 result, not more than 1 mode]
<chaals> 1 assertedby, 1 or more subject and testThingy
JJ: I won't query on mode, but want to show
SAZ: consensus: 1 assertedBy, 1 or more subject, 1 or more testThingy, 1 result, 0 or 1 mode
evidence and methodology left open for tomorrow
SAZ: no mandatory properties for Assertor?
JJ: encourage to giv emore information on Person, but don't require
<chaals> [/me in other telecon - sorry, need to pay attention elsewhere]
CI: add foaf:Organization to allowable types of Assertor?
SAZ: delete foaf:Person to allow changes in
foaf
... encourage use of Persion and Organization
<scribe> ACTION: Jim to get Libby for tomorrow's meeting [recorded in http://www.w3.org/2006/02/27-er-minutes.html#action12]
SAZ: JJ said, SingleAssertor and
CompondAssertor are not subclasses of Assertor
... defined in OWL, not in RDFS
... so no problem
... TestSubject
... required: date
JJ: which date?
SAZ: property of TestSubject, so creation of subject
JJ: clear when used with WebContent, but for
different things?
... move date as required to WebContent
SAZ: uri requred for WebContent?
... date of WebContent is date of fetching resource
JJ: leave date optional in TestSubject, required in WebContent
SAZ: what if we have more dates available?
JJ: use a different property
SAZ: uri optional?
JJ: there are use cases for not specifying uri
SAZ: dc:location -> earl:uri (optional)
... TestMode added semiAutomatic, changed mixed -> notAvailable
... TestResult: 1 validity
... 0 or 1 confidence
... ValidityLevel 5 values!
... Allowable types for TestSubject are not really 'allowable'.
JJ: have concern with allowable types for SingleAssetor, be propably happy with foaf:Agent
SAZ: guide for tomorrow
... tomorrow room 147
... 0900