QA WG Dublin F2F Day 2 am

9 Aug 2005


See also: IRC log


Karl, Lofton, Patrick, Lynne, Mark, Richard, Dom




<dom> Scribe: dom

Discussion around GP19

Lynne: I think our current text is good
... but I propose to add clarifications to the example

-> http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0022.html Lynne's rewrite

Karl: works for me

Dom: I think it does satisfy the editorial problem that has appeared during the discussion
... wrt the more substantive problem, we already said we disagreed with Bjoern's comment

RESOLUTION: agreed to incorporate Lynne's change to the example in GP19

Test Case data model

-> http://esw.w3.org/topic/TestCaseMetadata Test Case Metadata

karl: I propose "identification" should be renamed as identifier, as in Dublin Core (DC)

patrick: I suggest we actually take the description from dublin core as well

-> http://dublincore.org/documents/dces/#identifier Dublin Core definition for identifier

patrick: so the description should be "An unambiguous reference to the test case"

lynne: re rationale, "buys the ability to uniquely reference the test case"
... a required element
... I don't think there should be any restriction for the syntax

karl: should we suggest URIs there?

dom: would make sense to at least suggest it

lofton: but doesn't that make it bound to the versioning aspect?

karl: not necessarily, as in latest/this version in TR

patrick: so we should recommend URIs as a good idea, but not as a restriction

lynne: relevance, both during development and execution

patrick: we should fix "running time" into "test execution" for consistency sake

karl: what about example? do we have a real example to link to?

-> http://www.w3.org/MarkUp/Test/HTML401/20030123/tests/9_1-BF-01.html HTML 4.01 Test Case with a unique identifier "9_1-BF-01"

<karl> http://www.w3.org/MarkUp/Test/HTML401/current/tests/sec9_1-BF-01.html

Patrick: N/A for test FAQ

Dom: for SeeAlso, maybe Versioning?

Lynne: put that as temporary item

karl: next item, "name"

patrick: we should change to title as in dublin core

-> http://dublincore.org/documents/dces/#title Title definition in DC

lynne: description is "the name by which the test case is formally known"

patrick: rational is something more usable for humans than an identifier
... I don't think it should be required, as sometimes the identifier is enough
... syntax: text
... relevance: both
... test faq: N/A

lofton: but isn't a title required for usability?

patrick: I have some test suites I've been using for more than 7 years without anything but identifiers

lynne: maybe we should add a comment about this?

karl: that's a good idea

patrick: so let's add a comment field with that point made
... and this also means that SeeAlso should reference identifier

<karl> http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm

Karl: example from CSS Test suite "Pseudo-elements in selectors", http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm
... next is "description"

lynne: definition "a representation in words of the nature and characteristics of the test case"

patrick: also exists in Dublin core

karl: rationale?

lynne: to provide additional context about the test case

dom: not required, AFAICT

patrick: syntax: hypertext
... relevance: both
... test faq: N/A

Dom: how does it related to purpose? is it redundant?
... I think a broader description than the purpose can be useful

Lofton: [reading an example from WebCGM TS illustrating how they complement each other]

<lofton> Test basic CGM-to-HTML link, from a Application Structure with

<lofton> linkuri APS Attribute within the CGM, to a whole HTML document, with

<lofton> no behaviors or fragments associated with the link.

Lynne: see also: purpose

karl: next item is "status"
... useful for development process, but not necessarily at executing test time

patrick: depend on your test suite
... we have a test suite that contains buggy tests marked as excluded but that we still ship

<lofton> That descriptions comes from WebCGM test case linking-basicC2H-BE-02

patrick: in this case, it is relevant at execution time

lynne: there may be different states before and after release of the test suite

karl: is it required?

patrick: I don't think it should be required; maybe "recommended" (vs optional)

lofton: I think it should be required, maybe with a default value
... it's not reasonnable to release a test suite without that

lynne: this wouldn't apply to some of my test suites
... we go through several stages

<karl> marc: why don't we just call it "None" but required when someone don't want to use it

[somewhat consensus around "recommended", although lofton and mark think "required" is better]

karl: nothing relevant in DC, as far as I can tell
... what about description?

patrick,others: "one of an enumerated list of values that can be used to track the state of a test at a given time"

karl: rationale?

patrick: during development, it can be helpful to track the test through the review process

dom: it helps assess a confidence level with the test, maybe?

[rough agreement]

dom: syntax?

patrick: enumerated list

karl: [example from WCAG]

<karl> http://www.w3.org/WAI/GL/WCAG20/tests/ctprocess.html

lynne: that's for development time, though

patrick: after release, this could be "good, under investigation, withdrawn"
... or "correct" instead of "good"
... test faq: Q6, http://www.w3.org/QA/WG/2005/01/test-faq#review
... "6. What should we do with test contributions we receive?"
... let's skip versioning for now, I suggest

Karl: next is "link to spec"
... maybe relates to dc:source

patrick: I don't think we should use the word "source"
... but we could say this is equivalent to dc:source?
... I'm not sure there is a benefit to reference this

karl: so do we keep the name "link to spec"?

patrick: what about "spec reference"?
... description would be "identification of the portion of the specification tested"
... rationale is really close to what we put in purpose, isn't it?

lynne: "to provide the tracability from the test back to the spec"

dom: required?

lynne: yes

patrick: yes!

dom: syntax?

patrick: ideally, a URI with an XPath pointer, or a free form hyper text
... we should put as comments that it should be as precise as possible
... e.g. pointing to a particular string rather than a whole section

dom: relevance: both

patrick: test faq: Q8 is about coverage

lynne: for examples, DOM and XML have this I think

patrick: the examples in Q8 can be re-used
... e.g. the HTML 4.01 tests

dom: looking at the HTML test suite, they annotate their assertion with their RFC 2119 level requirement
... i.e. "must" vs "should"
... is that something we should record in our data model?

patrick: it somewhat relates to the variability discussion

lynne: or grouping?

patrick: let's come back to this later

lynne: going back to the spec ref, we should have an example of each type of the syntax

patrick: ideally, we could constrcut an xpath uri to match the example we had above

karl: what about "link to issue"?

patrick: would argue against that, too specific

dom: what about a more general "see also"?

patrick: sounds good
... could link to an issue database, a discussion in a mailing list

dom: so title, "see also"

patrick: description is a list of references to external resources relevant to this test case
... 2 use cases: issue in bug db, discussion ml
... rational: useful to be able to link to external resources that provide more context to the existence of the test case

dom: not required, obviously

patrick: syntax would be list of URIs
... relevance: probably more development than execution, but can be useful also after release when a test is challenged
... e.g. to link to a bug report

<karl> http://www.w3.org/QA/WG/2005/01/test-faq#bugs

patrick: test FAQ Q15 refers to bugs in a test suite, so maybe relevant
... examples should reference the 2 uses cases

karl: it could also be other test cases?

patrick: I guess so
... which makes me think that a URI as such isn't useful if there is not human-readable info about the said URI
... so syntax should be a list of pairs of (URI, human readable title)

dom: next one is "dependencies" that we suggested should be called "pre-conditions"
... separate from input

patrick: description: "conditions that must be met before this test is executed"
... rationale: any such condition must be understood before the test case can be successfully executed
... syntax: hypertext
... relevance: both
... test faq: N/A
... required: not
... I actually think stateless test suites are better

lynne: but not every pre-conditions come from dependencies between test cases
... so not necessarily bad

patrick: as an example, such a server process must be running for the test case to be executable
... let's skip grouping and variability-driven filtering criteria for now

dom: next are "input" and "expected results"

patrick: name is "input" or "inputs"?
... description: "parameters or data that must be supplied to the test at execution time"?

[discussing about whether "supplied to" is appropriate for a test]

karl: "parameters or data that are needed for the test execution"

patrick: rationale is obvious
... optional since in some cases it doesn't apply

karl: syntax, none, since that's implementation dependant
... relevant, both

patrick: but probably more importantly at execution time

lofton: I think it should be required, with "none" as a possibility

patrick: ok, I agree

karl: I disagree

Lynne: me too
... I may have an entire test suite without input at all

[optional seems to gain back consensus]

patrick: test FAQ Q11 may be somewhat relevant

karl: example could be HTTP interactions?
... e.g. an HTML file may be served under different conditions (encoding, MIME Type)

<scribe> ScribeNick: karl

Afternoon session

<scribe> Chair: Dom

dom: there is still one easy target - expected results

patrick: expected results or output

dom: we had the discussion yesterday, that sometimes it's not an ouput
... we keep expected results. agreement of the WG

(discussion between people about the definition)

description: The results that a correct implementation will produce

lynne: the results is know a priori before running the test then we need something which capture that

<dom> patrick: "the results that a conformant implementation is expected to produce"?

dom: I don't think we need a rationale for something which is that simple
... required is mandatory

patrick: yes

dom: syntax is implementaiton dependant
... what about tesfaq?

patrick: TestFAQ 9
... in test FAQ 9, I don't like the word outcome

dom: no link for the test FAQ 9

patrick: before we had the CSS Test Result: background green


See also - nothing in particular

dom: we are done with the simple ones
... should we tackle the difficult ones
... Grouping is a super property of variability

lofton: David proposed grouping?

dom: david proposed specific driven variarability

patrick: generic keywords would be useful to identify for the test case.

dom: keywords are not a property of test case

patrick explaining his position about dealing with grouping

patrick: Do we reduce grouping and variability to only one property.

lofton: (reading the comment of variability in the wiki)
... it seems that David agrees with the notion of grouping

patrick: yes you can have multiple ways of grouping

dom: mAybe it's so generic than it's not tied to the test case, and we could write prose to encourage people
... to use grouping for the process management

patrick: There may be multiple ways of slicing data in different sets.

dom: we should maybe separate it

patrick: maybe it fits in this template, but I see the problem of free form.

dom: Maybe it deserves more space. It something we should add before in prose

patrick: it's something which might be addressed in the test FAQ

dom, patrick: Grouping? (maybe to be modified if a better name)

karl: what about keywords?

dom: too specific.

karl: ok
... after discussion Description: "mechanism for classifying test cases into groups"

patrick: you want to be able to deal with variety of subsets
... then the rationale will be

lynne: agreed

dom: I think we should include some examples

karl: why should I care? what will you answer me?

patrick: A subset which shares some common characterics.
... To be able to select a subset which shares common characteristics.
... This is the rationale.

Required: no. agreement of the group

patrick: I would say one or more enumerated list

lofton: it's too limited
... the syntax is variable and include blablabla

patrick: syntax: Implementation dependent, some possibilities include naming conventions, enumerated lists, and naming conventions, and others.

dom: relevance?
... both
... TestFAQ?

patrick: we need to add a question to the Test FAQ
... how to deal with DOV

dom: I don't think we should have DOV in the question

patrick: something explaining the DOV.

dom: Example?
... interactive versus automated.
... voice versus dtmf
... tests marked for various profile
... SVG Tiny/Full had a subnaming convention.
... Comments: @@to be done with the capture of the discussion@@

lofton: what about DOV keyword?

dom: we just discussed that it was a kind of grouping
... what would be the gain?

lofton: For groupings other that driven variability ones, arbitrary groupings, I don't think there's a compelling argument for calling them recommended.
... for variability it's tied to the conformance clause

dom: but you said the recent SVG Test suite didn't deal with it

lofton: the first incarnation dealt with it.
... you can't test if you don't have grouping for variability

lynne: maybe we need something for conformance and variability testing

lofton: I don't think it should be buried in grouping

karl: I don't think it's part of the test case

dom: lofton is saying that it's part of the conformance criteria and the test case should say which profiles belong to

lynne: the reasons you would like to group is related to the conformance.

karl: could we work with a practical example?
... (discussion)
... explaining that a specific test case can be called by many specifications
... and then we will have a huge list of conformance grouping.

discussion between lynne and patrick

patrick: we have to be very explicit in the examples and comments about the ways to use grouping
... there are many different cases.

karl: (examples for WebCGM, XSLT, Java world, xml:lang attributes, etc.)
... I think it's a meta level which is not tied to the test case

lofton: yes I can buy that.

dom: (it seems we have an agreement)
... versioning?

patrick: we will have the same problem.

dom: version of the test case? version of the test suite? version of the specification?

patrick: a test which is valid for a spec A at a time doesn't say that the test will not be useful for another spec B later on.

dom: it seems to be another kind grouping

karl: how do we relate versioning and status?

dom: you might want to keep version control for the development of your test case.

patrick: Identifier blablabla version X for keeping an history in time. Ok I agree with that

lynne: this is the version of the test itself.

dom: Name: version

QAWG: Description: "Identifier which allows one to distinguish between different revisions of test case"
... Rationale: "It @@provides@@ an history of the evolution of the test case."
... Required yes
... Relevance: both
... Syntax: Free to implementation (see examples)
... TestFAQ: Link 14.
... Examples: look at CSS TS with latest etc. @@@@


QAWG: name: Contributor
... See Dublin Core
... discussion about ownership, about people submitting test but didn't want to assume the management of the test

patrick: you might want to know, who did this test so that it can be fix by the same entity who created the test.

dom: we have several questions.
... do we keep the metadata
... do we keep that name

lynne: I want to know who and/or where the test case came from. I need at least one of those.

patrick: you want to be able to identify the legal ownership of the test before the donation.

QAWG: name chosen is Contributor
... description: "Individual or organization that contributed this test case."
... Rationale: "It might be important to track the origin of the test for technical reasons, legal issues, modifications, etc."
... Required: optional
... Syntax: hypertext
... relevance: primary development
... TestFAQ: Question 10
... Example: XML Test Suite attribution to different contributors
... includes contribution from Microsoft, IBM, Sun, etc. @@link to an example@@

dom: dates?

karl: we can have dates about many things

dom; we should skip it

lynne: wait a minute, I don't want to get rid completely of it.

karl: I agree with lynne. And I think we should deal with it in the introduction.

QAWG: It needs to be discussed somewhere.

Lynne: Maybe in a note section after the metadata elements

karl: agreed

QAWG: Types: it's just a kind of grouping
... everyone agreed
... Source?
... Not that important
... Language?
... should be like date in the note.
... Coverage?

<scribe> ... Dropped


patrick: copyrights are dealt with a tangible form already. We should not try to address in a particular way.
... we may be stepped on lawyer's feet.

dom: We are not saying how they should be managed, we are giving a pointer.

karl: it's a pointer to the rights, not the rights

QAWG: Rights
... description
... Description: Information about rights held in and over the test case.
... Rationale: Publishers and Users of the test case need to know where the legal rights lie
... Requires: Optional
... Syntax: Ask your lawyer.
... Relevance: both
... Test FAQ: Q. 10
... Examples: @@To find an example@@
... Priority? no interest
... References? no interest

See mail for the previous discussion: http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0020.html

dom: what do we do with that?

QAWG: We decided to publish this as a W3C WG Note. Patrick Curran will write the prose, introduction + notes + Issues, The text will be reviewed by Richard Kennedy
... The text will be put in a ready to publish form by Karl
... before August 31.

dom: what do people think about creating actual schema for this Test Case Model.

QAWG: Great Idea

karl: do we try to gather people around it?

patrick: We should try first to create a first version of an XML Schema and/or RDF Schema and try to gather people after it.
... I would focus more on tools and very practical things.

karl: by tools?

patrick: something concrete and specific, the schema is too abstract, but something that can be usable: tools, programs, templates
... that might interest people to work on it

dom: Agenda for tomorrow morning. QA IG deliverables for rechartering.
... teleconferences
... next F2F?

lofton: Next TP in France end of Feb 2006.

karl: Basically future work

patrick: shopping list of what we could for the next months


<dom> Meeting: QA WG Dublin F2F Day 2

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.126 (CVS log)
$Date: 2005/08/09 15:40:03 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.126  of Date: 2005/05/16 16:49:48  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/rational/rationale/
Succeeded: s/there are/there may be/
Succeeded: s/dtmg/dtmf/
Succeeded: s/test suite/test case/
Found Scribe: dom
Inferring ScribeNick: dom
WARNING: No scribe lines found matching previous ScribeNick pattern: <karl> ...
Found ScribeNick: karl
ScribeNicks: karl, dom
Present: Karl Lofton Patrick Lynne Mark Richard Dom
Agenda: http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0004.html
Got date from IRC log name: 9 Aug 2005
Guessing minutes URL: http://www.w3.org/2005/08/09-qa-minutes.html
People with action items: 

[End of scribe.perl diagnostic output]