09:02:20 RRSAgent has joined #qa 09:02:20 logging to http://www.w3.org/2005/08/09-qa-irc 09:02:37 Meeting: QA WG Dublin F2F Day 2 am 09:02:41 Chair: Karl 09:03:41 Present: Karl, Lofton, Patrick, Lynne, Mark, Richard, Dom 09:05:41 RRSAgent, make log public 09:06:22 Scribe: dom 09:06:31 Agenda: http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0004.html 09:06:47 RRSagent, draft minutes 09:06:47 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 09:07:22 Topic: Discussion around GP19 09:07:37 Lynne: I think our current text is good 09:07:48 ... but I propose to add clarifications to the example 09:08:15 -> http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0022.html Lynne's rewrite 09:10:24 Karl: works for me 09:11:32 Dom: I think it does satisfy the editorial problem that has appeared during the discussion 09:11:53 ... wrt the more substantive problem, we already said we disagreed with Bjoern's comment 09:13:55 RESOLVED: agreed to incorporate Lynne's change to the example in GP19 09:16:03 lynne has joined #qa 09:16:28 Topic: Test Case data model 09:17:08 -> http://esw.w3.org/topic/TestCaseMetadata Test Case Metadata 09:19:00 karl: I propose "identification" should be renamed as identifier, as in Dublin Core (DC) 09:19:33 patrick: I suggest we actually take the description from dublin core as well 09:20:22 -> http://dublincore.org/documents/dces/#identifier Dublin Core definition for identifier 09:20:40 patrick: so the description should be "An unambiguous reference to the test case" 09:20:55 lynne: re rationale, "buys the ability to uniquely reference the test case" 09:21:15 ... a required element 09:21:33 ... I don't think there should be any restriction for the syntax 09:21:41 karl: should we suggest URIs there? 09:22:10 dom: would make sense to at least suggest it 09:22:27 lofton: but doesn't that make it bound to the versioning aspect? 09:22:58 karl: not necessarily, as in latest/this version in TR 09:23:19 patrick: so we should recommend URIs as a good idea, but not as a restriction 09:23:58 lynne: relevance, both during development and execution 09:24:22 patrick: we should fix "running time" into "test execution" for consistency sake 09:25:37 karl: what about example? do we have a real example to link to? 09:26:50 -> http://www.w3.org/MarkUp/Test/HTML401/20030123/tests/9_1-BF-01.html HTML 4.01 Test Case with a unique identifier "9_1-BF-01" 09:27:15 karl has joined #qa 09:27:26 http://www.w3.org/MarkUp/Test/HTML401/current/tests/sec9_1-BF-01.html 09:28:23 Patrick: N/A for test FAQ 09:29:41 RRSAgent, draft minutes 09:29:41 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 09:30:08 Dom: for SeeAlso, maybe Versioning? 09:30:18 Lynne: put that as temporary item 09:30:38 karl: next item, "name" 09:31:07 patrick: we should change to title as in dublin core 09:31:15 -> http://dublincore.org/documents/dces/#title Title definition in DC 09:31:43 lynne: description is "the name by which the test case is formally known" 09:32:02 patrick: rational is something more usable for humans than an identifier 09:32:24 ... I don't think it should be required, as sometimes the identifier is enough 09:32:28 ... syntax: text 09:32:37 ... relevance: both 09:32:46 ... test faq: N/A 09:33:44 lofton: but isn't a title required for usability? 09:34:04 patrick: I have some test suites I've been using for more than 7 years without anything but identifiers 09:34:22 lynne: maybe we should add a comment about this? 09:34:35 karl: that's a good idea 09:34:46 patrick: so let's add a comment field with that point made 09:35:01 ... and this also means that SeeAlso should reference identifier 09:36:46 http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm 09:37:29 Karl: example from CSS Test suite "Pseudo-elements in selectors", http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm 09:37:49 karl: next is "description" 09:38:27 lynne: definition "a representation in words of the nature and characteristics of the test case" 09:38:35 patrick: also exists in Dublin core 09:38:55 karl: rational? 09:39:35 lynne: to provide additional context about the test case 09:39:40 s/rational/rationale/ 09:39:49 dom: not required, AFAICT 09:39:57 patrick: syntax: hypertext 09:40:01 ... relevance: both 09:40:30 ... test faq: N/A 09:41:51 Dom: how does it related to purpose? is it redundant? 09:42:36 ... I think a broader description than the purpose can be useful 09:42:52 Lofton: [reading an example from WebCGM TS illustrating how they complement each other] 09:43:06 lofton has joined #qa 09:43:13 Test basic CGM-to-HTML link, from a Application Structure with 09:43:13 linkuri APS Attribute within the CGM, to a whole HTML document, with 09:43:13 no behaviors or fragments associated with the link. 09:43:42 Lynne: see also: purpose 09:44:17 karl: next item is "status" 09:44:38 ... useful for development process, but not necessarily at executing test time 09:44:43 patrick: depend on your test suite 09:44:59 ... we have a test suite that contains buggy tests marked as excluded but that we still ship 09:45:00 That descriptions comes from WebCGM test case linking-basicC2H-BE-02 09:45:11 ... in this case, it is relevant at execution time 09:46:05 lynne: there are different states before and after release of the test suite 09:46:21 s/there are/there may be/ 09:46:27 karl: is it required? 09:46:57 patrick: I don't think it should be required; maybe "recommended" (vs optional) 09:47:16 lofton: I think it should be required, maybe with a default value 09:47:37 ... it's not reasonnable to release a test suite without that 09:47:57 lynne: this wouldn't apply to some of my test suites 09:48:18 ... we go through several stages 09:50:55 marc: why don't we just call it "None" but required when someone don't want to use it 09:55:43 [somewhat consensus around "recommended", although lofton and mark think "required" is better] 09:56:40 karl: nothing relevant in DC, as far as I can tell 09:56:48 ... what about description? 09:58:24 patrick,others: "one of an enumerated list of values that can be used to track the state of a test at a given time" 09:59:45 karl: rationale? 10:01:17 patrick: during development, it can be helpful to track the test through the review process 10:01:59 dom: it helps assess a confidence level with the test, maybe? 10:02:06 [rough agreement] 10:02:42 dom: syntax? 10:02:54 patrick: enumerated list 10:03:32 karl: [example from WCAG] 10:03:34 http://www.w3.org/WAI/GL/WCAG20/tests/ctprocess.html 10:03:42 lynne: that's for development time, though 10:04:27 patrick: after release, this could be "good, under investigation, withdrawn" 10:04:46 ... or "correct" instead of "good" 10:05:34 ... test faq: Q6, http://www.w3.org/QA/WG/2005/01/test-faq#review 10:05:45 ... "6. What should we do with test contributions we receive?" 10:17:43 RRSAgent, draft minutes 10:17:43 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 10:21:51 karl has joined #qa 10:22:35 Patrick: let's skip versioning for now, I suggest 10:22:42 Karl: next is "link to spec" 10:23:06 ... maybe relates to dc:source 10:23:22 patrick: I don't think we should use the word "source" 10:24:13 ... but we could say this is equivalent to dc:source? 10:24:22 ... I'm not sure there is a benefit to reference this 10:24:52 karl: so do we keep the name "link to spec"? 10:26:18 patrick: what about "spec reference"? 10:27:05 ... description would be "identification of the portion of the specification tested" 10:27:48 patrick: rationale is really close to what we put in purpose, isn't it? 10:28:39 lynne: "to provide the tracability from the test back to the spec" 10:28:47 dom: required? 10:28:49 lynne: yes 10:28:52 patrick: yes! 10:29:13 dom: syntax? 10:29:36 patrick: ideally, a URI with an XPath pointer, or a free form hyper text 10:29:50 ... we should put as comments that it should be as precise as possible 10:30:00 ... e.g. pointing to a particular string rather than a whole section 10:30:11 dom: relevance: both 10:30:31 patrick: test faq: Q8 is about coverage 10:30:55 lynne: for examples, DOM and XML have this I think 10:32:02 patrick: the examples in Q8 can be re-used 10:32:11 ... e.g. the HTML 4.01 tests 10:34:36 dom: looking at the HTML test suite, they annotate their assertion with their RFC 2119 level requirement 10:34:45 ... i.e. "must" vs "should" 10:34:53 ... is that something we should record in our data model? 10:35:15 patrick: it somewhat relates to the variability discussion 10:35:21 lynne: or grouping? 10:35:30 patrick: let's come back to this later 10:36:49 lynne: going back to the spec ref, we should have an example of each type of the syntax 10:39:08 patrick: ideally, we could constrcut an xpath uri to match the example we had above 10:39:49 karl: what about "link to issue"? 10:40:32 patrick: would argue against that, too specific 10:40:43 dom: what about a more general "see also"? 10:40:50 patrick: sounds good 10:41:14 ... could link to an issue database, a discussion in a mailing list 10:42:09 dom: so title, "see also" 10:42:36 patrick: description is a list of references to external resources relevant to this test case 10:42:45 ... 2 use cases: issue in bug db, discussion ml 10:43:17 ... rational: useful to be able to link to external resources that provide more context to the existence of the test case 10:43:24 dom: not required, obviously 10:43:32 patrick: syntax would be list of URIs 10:44:16 ... relevance: probably more development than execution, but can be useful also after release when a test is challenged 10:44:21 ... e.g. to link to a bug report 10:44:37 http://www.w3.org/QA/WG/2005/01/test-faq#bugs 10:44:43 ... test FAQ Q15 refers to bugs in a test suite, so maybe relevant 10:45:32 ... examples should reference the 2 uses cases 10:46:21 karl: it could also be other test cases? 10:46:25 patrick: I guess so 10:47:07 ... which makes me think that a URI as such isn't useful if there is not human-readable info about the said URI 10:47:25 ... so syntax should be a list of pairs of (URI, human readable title) 10:48:01 dom: next one is "dependencies" that we suggested should be called "pre-conditions" 10:48:37 ... separate from input 10:49:11 patrick: description: "conditions that must be met before this test is executed" 10:50:14 ... rationale: any such condition must be understood before the test case can be successfully executed 10:50:21 ... syntax: hypertext 10:50:26 .. relevance: both 10:50:31 ... test faq: N/A 10:51:05 ... required: not 10:51:38 ... I actually think stateless test suites are better 10:51:56 lynne: but not every pre-conditions come from dependencies between test cases 10:52:03 ... so not necessarily bad 10:52:29 patrick: as an example, such a server process must be running for the test case to be executable 10:53:36 patrick: let's skip grouping and variability-driven filtering criteria for now 10:53:46 dom: next are "input" and "expected results" 10:54:27 patrick: name is "input" or "inputs"? 10:58:52 ... description: "parameters or data that must be supplied to the test at execution time"? 11:00:03 [discussing about whether "supplied to" is appropriate for a test] 11:00:20 karl: "parameters or data that are needed for the test execution" 11:00:29 patrick: rationale is obvious 11:01:02 ... optional since in some cases it doesn't apply 11:01:18 karl: syntax, none, since that's implementation dependant 11:01:25 ... relevant, both 11:01:52 patrick: but probably more importantly at execution time 11:02:25 lofton: I think it should be required, with "none" as a possibility 11:02:41 patrick: ok, I agree 11:02:45 karl: I disagree 11:02:50 Lynne: me too 11:03:35 ... I may have an entire test suite without input at all 11:04:21 [optional seems to gain back consensus] 11:04:30 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 11:04:59 patrick: test FAQ Q11 may be somewhat relevant 11:05:47 karl: example could be HTTP interactions? 11:06:18 ... e.g. an HTML file may be served under different conditions (encoding, MIME Type) 11:09:10 dom has joined #qa 11:09:39 RRSAgent, draft minutes 11:09:39 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 11:10:00 RRSAgent, bye 11:10:00 I see no action items 12:59:19 RRSAgent has joined #qa 12:59:19 logging to http://www.w3.org/2005/08/09-qa-irc 13:00:05 ScribeNick: karl 13:00:11 Topic: Afternoon session 13:00:27 Chair: Dom 13:01:07 dom: there is still one easy target - expected results 13:01:21 patrick: expected results or output 13:01:41 dom: we had the discussion yesterday, that sometimes it's not an ouput 13:02:04 ... we keep expected results. agreement of the WG 13:02:45 (discussion between people about the definition) 13:03:19 description: The results that a correct implementation will produce 13:04:40 lynne: the results is know a priori before running the test then we need something which capture that 13:04:43 patrick: "the results that a conformant implementation is expected to produce"? 13:05:23 dom: I don't think we need a rationale for something which is that simple 13:05:35 ... required is mandatory 13:05:40 patrick: yes 13:05:49 dom: syntax is implementaiton dependant 13:06:20 dom: what about tesfaq? 13:06:29 patrick: TestFAQ 9 13:07:57 ... in test FAQ 9, I don't like the word outcome 13:08:19 dom: no link for the test FAQ 9 13:08:29 lofton has joined #qa 13:08:39 patrick: before we had the CSS Test Result: background green 13:08:45 http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm 13:09:10 See also - nothing in particular 13:09:33 dom: we are done with the simple ones 13:09:42 ... should we tackle the difficult ones 13:10:44 ... Grouping is a super property of variability 13:11:01 lofton: David proposed grouping? 13:11:20 dom: david proposed specific driven variarability 13:13:07 patrick: generic keywords would be useful to identify for the test case. 13:13:24 dom: keywords are not a property of test case 13:14:54 patrick explaining his position about dealing with grouping 13:15:31 patrick: Do we reduce grouping and variability to only one property. 13:16:03 lofton: (reading the comment of variability in the wiki) 13:16:27 lofton: it seems that David agrees with the notion of grouping 13:16:40 patrick: yes you can have multiple ways of grouping 13:17:25 dom: mAybe it's so generic than it's not tied to the test case, and we could write prose to encourage people 13:17:36 ... to use grouping for the process management 13:18:03 patrick: There may be multiple ways of slicing data in different sets. 13:18:17 dom: we should maybe separate it 13:18:43 patrick: maybe it fits in this template, but I see the problem of free form. 13:19:07 dom: Maybe it deserves more space. It something we should add before in prose 13:19:21 patrick: it's something which might be addressed in the test FAQ 13:21:22 dom, patrick: Grouping? (maybe to be modified if a better name) 13:21:28 karl: what about keywords? 13:21:34 dom: too specific. 13:21:36 karl: ok 13:24:37 ... after discussion Description: "mechanism for classifying test cases into groups" 13:25:46 patrick: you want to be able to deal with variety of subsets 13:26:16 ... then the rationale will be 13:26:26 lynne: agreed 13:26:43 dom: I think we should include some examples 13:27:39 karl: why should I care? what will you answer me? 13:28:01 patrick: A subset which shares some common characterics. 13:28:25 ... To be able to select a subset which shares common characteristics. 13:28:38 ... This is the rationale. 13:29:00 Required: no. agreement of the group 13:30:08 patrick: I would say one or more enumerated list 13:30:14 lofton: it's too limited 13:30:30 ... the syntax is variable and include blablabla 13:31:17 patrick: syntax: Implementation dependent, some possibilities include naming conventions, enumerated lists, and naming conventions, and others. 13:31:27 dom: relevance? 13:31:32 ... both 13:31:37 dom: TestFAQ? 13:32:18 patrick: we need to add a question to the Test FAQ 13:32:29 ... how to deal with DOV 13:32:40 dom: I don't think we should have DOV in the question 13:32:54 patrick: something explaining the DOV. 13:33:03 dom: Example? 13:33:26 ... interactive versus automated. 13:33:33 ... voice versus dtmg 13:33:42 s/dtmg/dtmf/ 13:34:11 ... tests marked for various profile 13:34:44 ... SVG Tiny/Full had a subnaming convention. 13:35:11 dom: Comments: @@to be done with the capture of the discussion@@ 13:35:47 lofton: what about DOV keyword? 13:36:00 dom: we just discussed that it was a kind of grouping 13:36:13 ... what would be the gain? 13:38:30 lofton: For groupings other that driven variability ones, arbitrary groupings, I don't think there's a compelling argument for calling them recommended. 13:38:48 ... for variability it's tied to the conformance clause 13:39:10 dom: but you said the recent SVG Test suite didn't deal with it 13:39:25 lofton: the first incarnation dealt with it. 13:40:02 ... you can't test if you don't have grouping for variability 13:40:47 lynne: maybe we need something for conformance and variability testing 13:41:36 lofton: I don't think it should be buried in grouping 13:44:11 karl: I don't think it's part of the test suite 13:44:22 s/test suite/test case/ 13:45:00 dom: lofton is saying that it's part of the conformance criteria and the test case should say which profiles belong to 13:45:38 lynne: the reasons you would like to group is related to the conformance. 13:48:14 karl: could we work with a practical example? 13:48:20 ... (discussion) 13:55:01 ... explaining that a specific test case can be called by many specifications 13:56:17 ... and then we will have a huge list of conformance grouping. 13:56:27 discussion between lynne and patrick 13:57:50 patrick: we have to be very explicit in the examples and comments about the ways to use grouping 13:57:59 ... there are many different cases. 13:58:34 karl: (examples for WebCGM, XSLT, Java world, xml:lang attributes, etc.) 14:01:41 karl: I think it's a meta level which is not tied to the test case 14:01:53 lofton: yes I can buy that. 14:02:14 dom: (it seems we have an agreement) 14:02:21 ... versioning? 14:02:31 patrick: we will have the same problem. 14:03:05 dom: version of the test case? version of the test suite? version of the specification? 14:04:52 patrick: a test which is valid for a spec A at a time doesn't say that the test will not be useful for another spec B later on. 14:05:24 dom: it seems to be another kind grouping 14:06:03 karl: how do we relate versioning and status? 14:06:24 dom: you might want to keep version control for the development of your test case. 14:07:00 patrick: Identifier blablabla version X for keeping an history in time. Ok I agree with that 14:08:11 lynne: this is the version of the test itself. 14:08:19 dom: Name: version 14:12:22 QAWG: Description: "Identifier which allows one to distinguish between different revisions of test case" 14:13:11 QAWG: Rationale: "It @@provides@@ an history of the evolution of the test case." 14:13:21 QAWG: Required yes 14:13:36 QAWG: Relevance: both 14:14:25 QAWG: Syntax: Free to implementation (see examples) 14:16:14 QAWG: TestFAQ: Link 14. 14:16:47 QAWG: Examples: look at CSS TS with latest etc. @@@@ 14:18:14 (break) 14:34:33 QAWG: name: Contributor 14:34:50 ... See Dublin Core 14:35:41 lofton has joined #qa 14:37:42 ... discussion about ownership, about people submitting test but didn't want to assume the management of the test 14:39:05 patrick: you might want to know, who did this test so that it can be fix by the same entity who created the test. 14:41:12 dom: we have several questions. 14:41:20 ... do we keep the metadata 14:41:26 ... do we keep that name 14:42:00 lynne: I want to know who and/or where the test case came from. I need at least one of those. 14:43:20 patrick: you want to be able to identify the legal ownership of the test before the donation. 14:44:02 QAWG: name chosen is Contributor 14:47:08 ... description: "Individual or organization that contributed this test case." 14:48:06 ... Rationale: "It might be important to track the origin of the test for technical reasons, legal issues, modifications, etc." 14:49:35 ... Required: optional 14:49:59 ... Syntax: hypertext 14:50:10 ... relevance: primary development 14:50:45 ... TestFAQ: Question 10 14:51:11 ... Example: XML Test Suite attribution to different contributors 14:52:49 ... includes contribution from Microsoft, IBM, Sun, etc. @@link to an example@@ 14:52:56 dom: dates? 14:54:07 karl: we can have dates about many things 14:54:19 dom; we should skip it 14:54:35 lynne: wait a minute, I don't want to get rid completely of it. 14:55:54 karl: I agree with lynne. And I think we should deal with it in the introduction. 14:56:26 QAWG: It needs to be discussed somewhere. 14:56:44 Lynne: Maybe in a note section after the metadata elements 14:56:54 karl: agreed 14:57:10 QAWG: Types: it's just a kind of grouping 14:57:21 ... everyone agreed 14:57:34 QAWG: Source? 14:57:41 ... Not that important 14:57:54 QAWG: Language? 14:58:00 ... should be like date in the note. 14:58:24 ... Coverage? 14:58:37 ... Dropped 14:58:42 ... Rights? 14:59:48 patrick: copyrights are dealt with a tangible form already. We should not try to address in a particular way. 15:02:30 ... we may be stepped on lawyer's feet. 15:02:51 dom: We are not saying how they should be managed, we are giving a pointer. 15:05:14 karl: it's a pointer to the rights, not the rights 15:05:31 QAWG: Rights 15:05:36 ... description 15:06:00 ... Description: Information about rights held in and over the test case. 15:07:59 ... Rationale: Publishers and Users of the test case need to know where the legal rights lie 15:08:23 ... Requires: Optional 15:08:45 ... Syntax: Ask your lawyer. 15:08:52 ... Relevance: both 15:09:11 ... Test FAQ: Q. 10 15:10:37 ... Examples: @@To find an example@@ 15:10:56 ... Priority? no interest 15:11:06 ... References? no interest 15:11:40 See mail for the previous discussion: http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0020.html 15:11:57 dom: what do we do with that? 15:29:37 QAWG: We decided to publish this as a W3C WG Note. Patrick Curran will write the prose, introduction + notes + Issues, The text will be reviewed by Richard Kennedy 15:29:55 ... The text will be put in a ready to publish form by Karl 15:30:07 ... before August 31. 15:31:44 dom: what do people think about creating actual schema for this Test Case Model. 15:31:49 QAWG: Great Idea 15:32:46 karl: do we try to gather people around it? 15:33:15 patrick: We should try first to create a first version of an XML Schema and/or RDF Schema and try to gather people after it. 15:33:56 ... I would focus more on tools and very practical things. 15:34:02 karl: by tools? 15:34:56 patrick: something concrete and specific, the schema is too abstract, but something that can be usable: tools, programs, templates 15:35:03 ... that might interest people to work on it 15:36:46 dom: Agenda for tomorrow morning. QA IG deliverables for rechartering. 15:36:51 ... teleconferences 15:36:58 ... next F2F? 15:37:14 lofton: Next TP in France end of Feb 2006. 15:37:47 karl: Basically future work 15:38:00 patrick: shopping list of what we could for the next months 15:38:49 RRSAgent, draft minutes 15:38:49 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 15:38:53 ADJJOURNED 15:39:50 Meeting: QA WG Dublin F2F Day 2 15:39:52 RRSAgent, draft minutes 15:39:52 I have made the request to generate http://www.w3.org/2005/08/09-qa-minutes.html dom 15:43:53 lofton has left #qa