07:23:10 RRSAgent has joined #er 07:23:10 logging to http://www.w3.org/2006/02/27-er-irc 07:23:52 Meeting: ERT WG face-to-face, Day 1 08:11:46 shadi_ has joined #er 08:15:55 pecorra has joined #er 08:16:03 JohannesK has joined #er 08:17:43 chaals has joined #er 08:18:24 Zakim has joined #er 08:18:33 zakim, this will be ert 08:18:33 I do not see a conference matching that name scheduled near this time, chaals 08:18:45 zakim, what conferences do you see? 08:18:45 I see SW_RIF(TP)3:00AM, WS_DescWG(TP)3:00AM active 08:18:46 also scheduled at this time are XML_PMWG(TP)2:00AM, WS_DBWG(TP)3:00AM, PP_PSIG(TP)3:00AM 08:18:53 scribe: chaals 08:19:07 SAZ: Welcome. I am Shadi 08:19:14 DR: I work for Segala 08:19:34 CMN: Work for Opera Software 08:20:00 KB: Karima Boudaoud, Associate Professor, Uni of Nice, working on evaluation tools... 08:20:52 DS: Dirk Stegemann, Fraunhofer GMD, work with Carlos Velasco 08:21:16 JK: Johannes Koch, also. We are doing some evaluation tools 08:22:26 CI: Carlos Iglesias, CTIC Foundation works on W3C technologies, we have a tool called TAW http://tawdis.net 08:23:08 CMN: Also vice-president of FundaciĆ³n Sidar, who make Hera http://www.sidar.org/hera 08:24:08 SAZ: We have a close relation to things like Mobile Web Best Practices, and some other groups 08:24:11 Topic: Agenda 08:24:27 SAZ: Lots of things to talk about... 08:25:04 CarlosI has joined #er 08:25:26 SAZ: Start with scope? 08:25:57 http://lists.w3.org/Archives/Public/public-wai-ert/2006Feb/0040.html 08:27:41 ... people, look at it because it was only sent on Friday 08:30:40 CMN: Content labelling XG is doing exactly this bit of work, and I suggest that we work with them rather than doing this on our own. 08:32:01 SAZ: They are not Rec track, so how do we work with them? 08:34:27 CMN: In terms of Rec track, we can take what they do and bolt it in straight away. 08:34:55 SAZ: Right. Question is to make sure we understand better what they are working on, and make sure we don't duplicate it needlessly. 08:37:03 ACTION: Chaals to check the focus of the XG and see how it relates to this question 08:37:33 ACTION: David to check the focus of the XG and see how it relates to this question 08:37:50 SAZ: How do they label? 08:38:20 drooks has joined #er 08:40:04 ... was hoping to sort this stuff out here and push to last call 08:40:18 Topic: What do we need to go to last call 08:40:27 SAZ: Location, Scope 08:41:09 JK: Two kinds of location - where is something on the web (where we were using dc:location), and where in a resource the particular assertion is relevant to 08:41:20 SAZ: Evidence, methodology 08:41:28 ... noting that TestCase and TestRequirement changed 08:41:46 ... conformance section, extending EARL (conformance will be a big piece). 08:41:58 ... anything else missing for last call 08:42:52 CMN: Think we could go without scope, evidence, methodology if we really want. 08:43:04 ... don't think there is any more there. 08:43:13 Topic: Testcase / testrequirement 08:43:34 SAZ: We changed the term at last face to face, but now we think we need both... 08:44:28 CMN: Unconvinced by the need for it... 08:44:52 DR: Testcase requirement and test case are commonly used in testing evidence and methodology are not common terms. 08:45:23 SAZ: evidence and methodology I want to get into later on. TestCase and TestRequirement are a bit different... 08:45:59 ... basic problem is two vendors checking the same requirement but doing different actual tests. How do you identify this? 08:46:08 ... think that this will really help. 08:46:54 ... had used OWL to describe how a TestRequirement is part of another broader one. 08:47:56 ACTION: Chaals decide whether he is going to object to this today, or drop it... 08:47:56 CarlosI has joined #er 08:49:02 CMN: Went through the exercise, and was happy with OWL. Would rather have this as OWL/DC than built into EARL. 08:49:27 SAZ: For me it was easier to do it. Where the information is not public, it is better to have something in the language. 08:51:24 ... people are prone to mixing up the terms, and mess up their reports. 08:53:56 CMN: Will probably just make a formal objection, so it can run along in the process. 08:54:10 SAZ: Will be interested to see the proposal 08:54:52 CMN: Same as always - passing tests X,Y and not passing Z implies passing test Bar... 08:55:11 Topic: earl:mixed in testMode 08:55:22 SAZ: What does earl:mixed mean? 08:56:07 ... had thought it was for blanket statements - i.e. you aren't clarifying how each of the individual tests were done. 08:56:18 ... mainly for conformance claims in the large. 08:56:48 ... ("my website foo conforms to big specification bar...") 08:57:02 ... do we want to promote such blanket statements? Leave it for RDF-CL? ... 08:58:00 CMN: Not a great fan of it, but if it means what Shadi said above I am fine with it. 08:59:05 SAZ: There are manual and automatic tests, but there are also tests where the tool does something to help the user answer a question. 08:59:46 JK: It is a question about what is the test. If only the deciding part, or finding the instance as well. it isn't a manual test if teh tool is finding the instance and asking the user to decide (or vice versa) 08:59:56 SAZ: For those semi-auto cases mixed is not a good term 09:00:01 KB: Yes, that is what we use 09:00:13 JK: You were talking about a collection of tests. 09:00:23 SAZ: Think we wanted to cover both cases... 09:00:57 ... think we should have earl:semiAutomatic, and something to cover the case of blanket collections of assertions 09:01:35 DS: Two cases - auto and manual. You have searching for a relevant place, and deciding. it is important to be able to say which bit is done manually, which bit automatically 09:03:52 CMN: If we have agreed the meaning Shadi gave for mixed, then we have enough with what we have in the draft 09:04:13 ... that leaves the question of who does what in a given test to the description of the test. 09:04:31 KB: Think we just need to identify the manual/auto/semi-auto cases 09:05:01 SAZ: Agree - we don't need to do that. We could use mainAssertor / helpAssertor to identify who did what. 09:05:21 JK: May be important for a test case desccription language, but not necessarily for a reporting language. 09:06:37 SAZ: [seems like we have had this discussion before...] 09:06:51 SAZ: Do we want earl:semiAutomatic? 09:07:13 CMN: Not necessary. We have compound assertor that gives us even more flexibility 09:07:45 JK: If we use the assertor, we don't need to have this mode. If the assertor is a person, or a tool, then it is clear, so we don't need manual or automatic 09:08:06 SAZ: If we can't give a test mode what do we use mixed for? 09:08:19 ... questioning test mode as a whole. Except earl:heuristic. 09:10:33 CMN: there is no way of knowing that something was heuristic if you don't say it was manual/auto. manual, auto, heuristic are mutually exclusive 09:10:38 DR: mixed is a mess 09:10:59 SAZ: right. For semi-auto case, it is easier to query the mode than look in the assertor, but you should look there. 09:11:49 CMN: earl:mixed I have mixed feelings about (it is "dunno") but we should have the other three 09:12:13 KB: Don't think it makes sense. 09:13:20 CMN: Think we should have it, but not strongly attached to it. 09:14:03 CI: Think it is not necessary. It is some unknown mode. If it makes it easier to check whether the mode is known, it could be useful, but that is the only reason to have it. 09:14:39 SAZ: In a blanket statement the test mode is known. "unknown" is not quite accurate. "notAvailable"? 09:14:52 CMN: Don't care what you call it. Just don't change it again. 09:15:15 SAZ: mixed was a bad name - caused confusion 09:15:29 CMN: Right. Without an agreed label and description, we are likely to go round in circles. 09:16:16 SAZ: earl:semiautomatic? If we have a testMode to query it is easier. Do we want to introduce earl:semiautomatic? 09:16:21 CMN: no. 09:16:41 CI: Same reason to have semiAutomatic as unknown/mixed. 09:17:17 CMN: changed mind - think it is OK to have it. 09:17:34 SAZ: should reconsider test mode, or provide it all. 09:17:55 CMN: So what is the description / label. 09:18:26 ACTION: Chaals propose label/description for mixed / semiauto 09:19:20 [proposed: earl:mixed s:label "not available" ; s:comment "There is no detailed information about the test mode available".] 09:20:09 [propose: earl:semiauto s:label "Semi-automatic" ; s:comment "The test was done by a person or people, assisted by an automated tool"] 09:20:33 --break 10:06:21 --restart 10:07:53 SAZ: location class - dc:location does not exist in rdf. need to define our own location 10:09:39 JibberJim has joined #er 10:10:11 JIM: non web content needs location? 10:13:13 SAZ: dc:location defines specific URI. need to introduce EARL URI 10:14:35 SAZ: EARL:location - label = web URI, location=URI 10:15:37 s/location=URI/comment = a URI to find this on the Web/ 10:15:45 ACTION: chaals to create EARL:location class 10:17:36 SAZ: how we describe same result multiple times? 10:18:42 http://lists.w3.org/Archives/Public/public-wai-ert/2005Sep/0007.html 10:19:17 http://lists.w3.org/Archives/Public/public-wai-ert/2005Sep/0012.html 10:19:45 http://lists.w3.org/Archives/Public/public-wai-ert/2005Oct/0002.html 10:22:58 JibberJim has joined #er 10:23:42 chaals has joined #er 10:24:15 JIM: range location not useful 10:25:35 JIM: single pointes should support range 10:31:49 ACTION: range location should be dropped 10:32:57 CarlosI has joined #er 10:32:58 CHAALS: showdow location is a location within a compound location 10:40:51 Zakim has left #er 10:41:41 ACTION: change shadow location to property of location 10:42:26 CHAALS: earl:pointer class needs to be defined 10:51:52 SAZ: do we need to define default locations? 10:52:11 JIM: create a pointers namspace? 10:52:38 pecorra has joined #er 10:53:54 JIM: need to define xpointer 10:56:55 SAZ: also define xpath 10:56:58 pecorra has joined #er 10:57:28 SAZ: and xquery 10:58:42 ACTION: JIM: write up location types - linecharlength, xpointer, xpath, xquery... 11:02:58 JohannesK has joined #er 11:03:11 JibberJim has joined #er 11:03:49 pecorra has joined #er 11:08:43 chaals has joined #er 11:10:14 drooks has joined #er 11:13:16 [so we have: 11:13:16 earl:location a r:Property ; s:label "location" ; s:comment "location of the result within the test sbuject" ; s:range earl:Pointer . 11:13:16 earl:Pointer a s:Class ; s:label "type of location" ; s:comment "A generic class for a type of location. Locations used in earl reports should be described using subclasses of this class" . 11:13:16 earl:SingleLocation s:subClass earl:Pointer ; s:label "Single location" ; s:comment "A single location. This may have a number of location properties, using different types of Pointer, that point to the same location." 11:13:18 earl:shadowLocation s:subPropertyOf earl:location ; s:label "secondary location" ; s:comment "This can be used to describe something which is not where the error is manifested, but is relevant - for example a script which is responsible for causing the manifested result, or otherwise imnportant to understanding it." . 11:13:22 earl:CompoundLocation s:subClassOf earl:Pointer ; s:label "compound location" ; s:comment "this class is used to refer to multiple locations which together manifest one result. The locations may refer to different single locations" . 11:13:25 ] 11:13:42 shadi has joined #er 11:14:25 ACTION: JIM: write up location types - linecharlength, xpointer, xpath, xquery... 11:14:56 ACTION: JIM to get in touch with annotea & check on namespace for earl:pointer 11:15:13 CHAALS: methodolgy should be a pointer e.g. this is how i did this 11:15:29 CHAALS: methodology use case is within evidence 11:15:42 CHAALS: evidence is collection - bounded list of things used 12:35:02 shadi has joined #er 12:36:18 JibberJim has joined #er 12:39:33 scribenick JibberJim 12:39:39 scribenick: JibberJim 12:39:44 scribe: Jim 12:39:54 JohannesK has joined #er 12:40:23 drooks has joined #er 12:41:35 pecorra has joined #er 12:41:44 Evidence and methodology postponed until later when chaals returns 12:41:53 Topic EARL Conformance 12:42:08 SAZ: What does it mean to conform to EARL? 12:42:13 SAZ: What is the minimum? 12:42:29 SAZ: Should we tweak it to be more required/less optional ? 12:43:42 CarlosI has joined #er 12:43:51 SAZ: Does it make sense to have more than one assertor, or more than on subject? 12:45:01 SAZ: simplest thing would be to say there must be 1 of each of the things, that would be really verbose 12:45:20 SAZ: we trade of shortness for ambiguity 12:46:36 DB: Why have testSubject aswell as webcontent/ 12:47:22 SAZ: So we can talk about things wider than just web content - webcontent inherits from testSubject 12:49:29 DR: Why would you have more than one webcontent? 12:50:09 JK: web-content is just one resource, so you could have tested more than one resource at the same time 12:50:36 SAZ: Currently the only required thing for a testSubject to have is a dc:Date 12:51:06 SAZ: everything else is optional but user agents should understand. 12:52:33 JK: for multiple test subjects you should repeat earl:subject to do them more 12:53:02 CI: You can also do that with multiple dc:location or earl:URI in a single test:Subject 12:54:13 SAZ: Yes so this is similar to earl:Location we just define the container and people are free to widen it more, that limits testSubject to one instance per report. 12:56:07 JL: I don't like the idea of just a single testSubject as it limits serialisations more than necessary. 12:56:27 JL: e.g. when you're testing 100 images in a page and they all pass 12:56:39 SAZ: so agree on only 1 Asserter 12:57:28 SAZ: but more than one TestSubject is okay 12:57:52 CMN: Requirement is Required 12:58:26 SAZ: Yes one must be required and requirement seems to be the obvious one 13:01:01 JibberJim has joined #er 13:01:21 JK: There are tests that apply to several requirements. 13:01:29 SAZ: Let's not get into exactly one. 13:02:51 JL: What does testcase need to be? 13:03:03 CI: it should be very flexible 13:04:04 chaals has joined #er 13:04:45 SAZ: I need to be able to recognise that both A and B have tested the same thing 13:04:55 rrsagent, make log public 13:05:36 CMN: no test case, just requirement. 13:05:46 SAZ: I propose requirement that's required, and an optional test case 13:06:02 CI: that is how testers work today 13:06:48 CMN: It's not clear that passing or failing something means when you've really passed the requirement 13:08:09 CMN: so test case is redundant. 13:08:58 DB: You cannot say anything about pass/failing a test case if the pass/fail is on the requirement. 13:09:48 JK: So if 1 of 5 of a test cases or 4 in 5 both of which result in you failing the requirement is the test as failed. 13:10:19 CI: usually if the test contains 5 atomic testing you always have to test all 5 again even if they succeeded before. 13:10:34 CMN: but in the real world that's not how it works. 13:10:48 CI: but if the tests aren't atomic they should be seperate test cases. 13:11:19 CMN: but we aren't telling people how to design tests 13:13:19 SAZ: Is it true that if a requirements succeeds then a test succeeded? 13:13:22 CMN: No. 13:14:03 CMN: if requirement was the pass/fail then people would seperate tests out into individual testCases. 13:14:56 JL: Requirement is the publically agreed label, testCase is what you individually did to meet that. 13:15:46 JL: So seperate testCase/requirement is fine and one of them are required with the requirement linked to the testcases by heuristic methods. 13:16:29 CI draws picture 13:18:16 SAZ: A tool can only optionally expose what tests it did to meet a requirement 13:19:33 SAZ: describes model with everyones agreement in CI's drawing. 13:20:57 CMN: You must have a direct record of things you pass or fail to re-enable re-use of test result data 13:23:20 CI: If you have a test case for WCAG 1.1 and have 5 atomic tests within it, then it doesn't matter which you pass or fail 13:23:30 CI: Good test cases should be atomic. 13:23:50 CI: So there is always a 1 to 1 match with tests. 13:25:19 SAZ: I think your testCase is a single use case. 13:26:26 CI: if we drop testcase, then we drop all the other information too 13:27:26 CMN: You use the same methods to describe "supporting xhtml 1.0" as a "particularly crazy part of the individual spec". 13:28:08 CMN: With manual testing we don't want to throw away results if they don't change because we do something new 13:29:38 CMN: we still allow a WCAG 1.0 requirement to describe that it's built up from atomic tests 13:31:14 DB: You have to have a requirement to test against 13:32:01 CMN: We disagree here between testing and test restul reporting 13:34:33 CI: I want a class for requirements that only points to requirements, and a different class for test cases. 13:38:09 SAZ: What breaks for you in this model? 13:42:25 JL: What breaks for me is that I don't know what to present a user as most important information so like requirements distinct from test case. 13:43:41 KB: Passing/failing individual tests is more important than requirements, so if we keep both, both should be required, but if we choose testcase is more important 13:44:21 SAZ: This is tough and depends on the user. 13:44:45 SAZ: different groups have different granularity 13:44:52 q+ to make proposal on sub 13:45:17 CMN: I don't know how this meets the testcase/requirement model 13:47:37 SAZ: I beleive there should be a top down approach from requirements to tests 13:47:52 KB: Does it make sense to be able to have the choice 13:49:42 JL: TestCase is just a subClassOf Requirement 13:50:36 SAZ: so requirement is a wooly public, testcase is more specific 13:51:05 KB: So can a requirement be multiple test cases. 13:51:40 CMN: What a requirement/testCase is completely open. 13:54:14 CMN: I like the proposal for subClassOf to give us the distinction. 13:56:43 SAZ: The test requirement and the test case are the same. 13:57:06 CMN: A test case is always a type of requirement whereas not the reverse. 13:57:17 CI: I don't agree they are the same 14:03:54 CMN: earl:PassOrFailable be a super class of earl:Requirement earl:TestCase so that we completely seperate the idea that they are related. 14:06:35 DB: Why can't we make them a subclass assertion? 14:07:19 CMN: Can't be done assertion has lots of things. 14:07:46 SAZ: we now agree that testcase/testrequirement should be one property 14:09:21 JK: Could we just have a property called earl:test and an earl:Test class. 14:11:24 CI: so what we need is a method to distinguish between our tests, and our customers requirements 14:14:01 ACTION: CMN to write up proposal for subclass test - due 2006-02-27 14:54:16 scribe: JohannesK 14:55:47 SAZ: back to conformance 14:56:51 SAZ: Assertion: exactly 1 assertedBy, 1 or more subject, 1 or more testThingy 14:57:11 SAZ: exactly 1 result? 14:58:02 SAZ: exactly 1 mode? 14:59:52 SAZ: and mode required 15:00:03 JJ: not more than 1 15:01:20 [exactly 1 result, not more than 1 mode] 15:01:35 1 assertedby, 1 or more subject and testThingy 15:05:26 pecorra has joined #er 15:05:55 JJ: I won't query on mode, but want to show 15:07:21 SAZ: consensus: 1 assertedBy, 1 or more subject, 1 or more testThingy, 1 result, 0 or 1 mode 15:08:08 evidence and methodology left open for tomorrow 15:08:51 SAZ: no mandatory properties for Assertor? 15:09:18 JJ: encourage to giv emore information on Person, but don't require 15:10:34 [/me in other telecon - sorry, need to pay attention elsewhere] 15:10:41 CI: add foaf:Organization to allowable types of Assertor? 15:11:56 SAZ: delete foaf:Person to allow changes in foaf 15:12:25 SAZ: encourage use of Persion and Organization 15:13:59 ACTION: Jim to get Libby for tomorrow's meeting 15:15:47 SAZ: JJ said, SingleAssertor and CompondAssertor are not subclasses of Assertor 15:17:46 SAZ: defined in OWL, not in RDFS 15:18:18 SAZ: so no problem 15:19:13 SAZ: TestSubject 15:19:45 SAZ: required: date 15:19:58 JJ: which date? 15:20:50 SAZ: property of TestSubject, so creation of subject 15:21:11 JJ: clear when used with WebContent, but for different things? 15:21:46 JJ: move date as required to WebContent 15:22:32 SAZ: uri requred for WebContent? 15:30:53 SAZ: date of WebContent is date of fetching resource 15:31:45 JJ: leave date optional in TestSubject, required in WebContent 15:38:08 SAZ: what if we have more dates available? 15:38:19 JJ: use a different property 15:41:27 SAZ: uri optional? 15:41:49 JJ: there are use cases for not specifying uri 15:44:53 SAZ: dc:location -> earl:uri (optional) 15:45:34 SAZ: TestMode added semiAutomatic, changed mixed -> notAvailable 15:46:04 SAZ: TestResult: 1 validity 15:47:11 SAZ: 0 or 1 confidence 15:47:48 SAZ: ValidityLevel 5 values! 15:59:07 JohannesK has joined #er 15:59:19 SAZ: Allowable types for TestSubject are not really 'allowable'. 15:59:29 JJ: have concern with allowable types for SingleAssetor, be propably happy with foaf:Agent 15:59:57 SAZ: guide for tomorrow 16:00:10 SAZ: tomorrow room 147 16:00:16 SAZ: 0900 16:00:43 pecorra has joined #er 16:07:54 shadi has joined #er 16:10:41 rrsagent, make logs world 16:10:57 rrsagent, make minutes 16:10:57 I have made the request to generate http://www.w3.org/2006/02/27-er-minutes.html shadi 16:11:04 rrsagent, make logs world 16:11:12 rrsagent, bye 16:11:12 I see 12 open action items saved in http://www.w3.org/2006/02/27-er-actions.rdf : 16:11:12 ACTION: Chaals to check the focus of the XG and see how it relates to this question [1] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T08-37-03 16:11:12 ACTION: David to check the focus of the XG and see how it relates to this question [2] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T08-37-33 16:11:12 ACTION: Chaals decide whether he is going to object to this today, or drop it... [3] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T08-47-56 16:11:12 ACTION: Chaals propose label/description for mixed / semiauto [4] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T09-18-26 16:11:12 ACTION: chaals to create EARL:location class [5] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T10-15-45 16:11:12 ACTION: range location should be dropped [6] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T10-31-49 16:11:12 ACTION: change shadow location to property of location [7] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T10-41-41 16:11:12 ACTION: JIM: write up location types - linecharlength, xpointer, xpath, xquery... [8] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T10-58-42 16:11:12 ACTION: JIM: write up location types - linecharlength, xpointer, xpath, xquery... [9] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T11-14-25 16:11:12 ACTION: JIM to get in touch with annotea & check on namespace for earl:pointer [10] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T11-14-56 16:11:12 ACTION: CMN to write up proposal for subclass test - due 2006-02-27 [11] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T14-14-01 16:11:12 ACTION: Jim to get Libby for tomorrow's meeting [12] 16:11:12 recorded in http://www.w3.org/2006/02/27-er-irc#T15-13-59