See also: IRC log
<dom> Scribe: dom
Lynne: I think our current text
is good
... but I propose to add clarifications to the example
-> http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0022.html Lynne's rewrite
Karl: works for me
Dom: I think it does satisfy the
editorial problem that has appeared during the discussion
... wrt the more substantive problem, we already said we
disagreed with Bjoern's comment
RESOLUTION: agreed to incorporate Lynne's change to the example in GP19
-> http://esw.w3.org/topic/TestCaseMetadata Test Case Metadata
karl: I propose "identification" should be renamed as identifier, as in Dublin Core (DC)
patrick: I suggest we actually take the description from dublin core as well
-> http://dublincore.org/documents/dces/#identifier Dublin Core definition for identifier
patrick: so the description should be "An unambiguous reference to the test case"
lynne: re rationale, "buys the
ability to uniquely reference the test case"
... a required element
... I don't think there should be any restriction for the
syntax
karl: should we suggest URIs there?
dom: would make sense to at least suggest it
lofton: but doesn't that make it bound to the versioning aspect?
karl: not necessarily, as in latest/this version in TR
patrick: so we should recommend URIs as a good idea, but not as a restriction
lynne: relevance, both during development and execution
patrick: we should fix "running time" into "test execution" for consistency sake
karl: what about example? do we have a real example to link to?
-> http://www.w3.org/MarkUp/Test/HTML401/20030123/tests/9_1-BF-01.html HTML 4.01 Test Case with a unique identifier "9_1-BF-01"
<karl> http://www.w3.org/MarkUp/Test/HTML401/current/tests/sec9_1-BF-01.html
Patrick: N/A for test FAQ
Dom: for SeeAlso, maybe Versioning?
Lynne: put that as temporary item
karl: next item, "name"
patrick: we should change to title as in dublin core
-> http://dublincore.org/documents/dces/#title Title definition in DC
lynne: description is "the name by which the test case is formally known"
patrick: rational is something
more usable for humans than an identifier
... I don't think it should be required, as sometimes the
identifier is enough
... syntax: text
... relevance: both
... test faq: N/A
lofton: but isn't a title required for usability?
patrick: I have some test suites I've been using for more than 7 years without anything but identifiers
lynne: maybe we should add a comment about this?
karl: that's a good idea
patrick: so let's add a comment
field with that point made
... and this also means that SeeAlso should reference
identifier
<karl> http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm
Karl: example from CSS Test suite
"Pseudo-elements in selectors",
http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm
... next is "description"
lynne: definition "a representation in words of the nature and characteristics of the test case"
patrick: also exists in Dublin core
karl: rationale?
lynne: to provide additional context about the test case
dom: not required, AFAICT
patrick: syntax: hypertext
... relevance: both
... test faq: N/A
Dom: how does it related to
purpose? is it redundant?
... I think a broader description than the purpose can be
useful
Lofton: [reading an example from WebCGM TS illustrating how they complement each other]
<lofton> Test basic CGM-to-HTML link, from a Application Structure with
<lofton> linkuri APS Attribute within the CGM, to a whole HTML document, with
<lofton> no behaviors or fragments associated with the link.
Lynne: see also: purpose
karl: next item is "status"
... useful for development process, but not necessarily at
executing test time
patrick: depend on your test
suite
... we have a test suite that contains buggy tests marked as
excluded but that we still ship
<lofton> That descriptions comes from WebCGM test case linking-basicC2H-BE-02
patrick: in this case, it is relevant at execution time
lynne: there may be different states before and after release of the test suite
karl: is it required?
patrick: I don't think it should be required; maybe "recommended" (vs optional)
lofton: I think it should be
required, maybe with a default value
... it's not reasonnable to release a test suite without
that
lynne: this wouldn't apply to
some of my test suites
... we go through several stages
<karl> marc: why don't we just call it "None" but required when someone don't want to use it
[somewhat consensus around "recommended", although lofton and mark think "required" is better]
karl: nothing relevant in DC, as
far as I can tell
... what about description?
patrick,others: "one of an enumerated list of values that can be used to track the state of a test at a given time"
karl: rationale?
patrick: during development, it can be helpful to track the test through the review process
dom: it helps assess a confidence level with the test, maybe?
[rough agreement]
dom: syntax?
patrick: enumerated list
karl: [example from WCAG]
<karl> http://www.w3.org/WAI/GL/WCAG20/tests/ctprocess.html
lynne: that's for development time, though
patrick: after release, this
could be "good, under investigation, withdrawn"
... or "correct" instead of "good"
... test faq: Q6, http://www.w3.org/QA/WG/2005/01/test-faq#review
... "6. What should we do with test contributions we
receive?"
... let's skip versioning for now, I suggest
Karl: next is "link to
spec"
... maybe relates to dc:source
patrick: I don't think we should
use the word "source"
... but we could say this is equivalent to dc:source?
... I'm not sure there is a benefit to reference this
karl: so do we keep the name "link to spec"?
patrick: what about "spec
reference"?
... description would be "identification of the portion of the
specification tested"
... rationale is really close to what we put in purpose, isn't
it?
lynne: "to provide the tracability from the test back to the spec"
dom: required?
lynne: yes
patrick: yes!
dom: syntax?
patrick: ideally, a URI with an
XPath pointer, or a free form hyper text
... we should put as comments that it should be as precise as
possible
... e.g. pointing to a particular string rather than a whole
section
dom: relevance: both
patrick: test faq: Q8 is about coverage
lynne: for examples, DOM and XML have this I think
patrick: the examples in Q8 can
be re-used
... e.g. the HTML 4.01 tests
dom: looking at the HTML test
suite, they annotate their assertion with their RFC 2119 level
requirement
... i.e. "must" vs "should"
... is that something we should record in our data model?
patrick: it somewhat relates to the variability discussion
lynne: or grouping?
patrick: let's come back to this later
lynne: going back to the spec ref, we should have an example of each type of the syntax
patrick: ideally, we could constrcut an xpath uri to match the example we had above
karl: what about "link to issue"?
patrick: would argue against that, too specific
dom: what about a more general "see also"?
patrick: sounds good
... could link to an issue database, a discussion in a mailing
list
dom: so title, "see also"
patrick: description is a list of
references to external resources relevant to this test
case
... 2 use cases: issue in bug db, discussion ml
... rational: useful to be able to link to external resources
that provide more context to the existence of the test case
dom: not required, obviously
patrick: syntax would be list of
URIs
... relevance: probably more development than execution, but
can be useful also after release when a test is
challenged
... e.g. to link to a bug report
<karl> http://www.w3.org/QA/WG/2005/01/test-faq#bugs
patrick: test FAQ Q15 refers to
bugs in a test suite, so maybe relevant
... examples should reference the 2 uses cases
karl: it could also be other test cases?
patrick: I guess so
... which makes me think that a URI as such isn't useful if
there is not human-readable info about the said URI
... so syntax should be a list of pairs of (URI, human readable
title)
dom: next one is "dependencies"
that we suggested should be called "pre-conditions"
... separate from input
patrick: description: "conditions
that must be met before this test is executed"
... rationale: any such condition must be understood before the
test case can be successfully executed
... syntax: hypertext
... relevance: both
... test faq: N/A
... required: not
... I actually think stateless test suites are better
lynne: but not every
pre-conditions come from dependencies between test cases
... so not necessarily bad
patrick: as an example, such a
server process must be running for the test case to be
executable
... let's skip grouping and variability-driven filtering
criteria for now
dom: next are "input" and "expected results"
patrick: name is "input" or
"inputs"?
... description: "parameters or data that must be supplied to
the test at execution time"?
[discussing about whether "supplied to" is appropriate for a test]
karl: "parameters or data that are needed for the test execution"
patrick: rationale is
obvious
... optional since in some cases it doesn't apply
karl: syntax, none, since that's
implementation dependant
... relevant, both
patrick: but probably more importantly at execution time
lofton: I think it should be required, with "none" as a possibility
patrick: ok, I agree
karl: I disagree
Lynne: me too
... I may have an entire test suite without input at all
[optional seems to gain back consensus]
patrick: test FAQ Q11 may be somewhat relevant
karl: example could be HTTP
interactions?
... e.g. an HTML file may be served under different conditions
(encoding, MIME Type)
<scribe> ScribeNick: karl
<scribe> Chair: Dom
dom: there is still one easy target - expected results
patrick: expected results or output
dom: we had the discussion
yesterday, that sometimes it's not an ouput
... we keep expected results. agreement of the WG
(discussion between people about the definition)
description: The results that a correct implementation will produce
lynne: the results is know a priori before running the test then we need something which capture that
<dom> patrick: "the results that a conformant implementation is expected to produce"?
dom: I don't think we need a
rationale for something which is that simple
... required is mandatory
patrick: yes
dom: syntax is implementaiton
dependant
... what about tesfaq?
patrick: TestFAQ 9
... in test FAQ 9, I don't like the word outcome
dom: no link for the test FAQ 9
patrick: before we had the CSS Test Result: background green
http://www.w3.org/Style/CSS/Test/CSS2.1/current/t0510-c25-pseudo-elmnt-00-c.htm
See also - nothing in particular
dom: we are done with the simple
ones
... should we tackle the difficult ones
... Grouping is a super property of variability
lofton: David proposed grouping?
dom: david proposed specific driven variarability
patrick: generic keywords would be useful to identify for the test case.
dom: keywords are not a property of test case
patrick explaining his position about dealing with grouping
patrick: Do we reduce grouping and variability to only one property.
lofton: (reading the comment of
variability in the wiki)
... it seems that David agrees with the notion of grouping
patrick: yes you can have multiple ways of grouping
dom: mAybe it's so generic than
it's not tied to the test case, and we could write prose to
encourage people
... to use grouping for the process management
patrick: There may be multiple ways of slicing data in different sets.
dom: we should maybe separate it
patrick: maybe it fits in this template, but I see the problem of free form.
dom: Maybe it deserves more space. It something we should add before in prose
patrick: it's something which might be addressed in the test FAQ
dom, patrick: Grouping? (maybe to be modified if a better name)
karl: what about keywords?
dom: too specific.
karl: ok
... after discussion Description: "mechanism for classifying
test cases into groups"
patrick: you want to be able to
deal with variety of subsets
... then the rationale will be
lynne: agreed
dom: I think we should include some examples
karl: why should I care? what will you answer me?
patrick: A subset which shares
some common characterics.
... To be able to select a subset which shares common
characteristics.
... This is the rationale.
Required: no. agreement of the group
patrick: I would say one or more enumerated list
lofton: it's too limited
... the syntax is variable and include blablabla
patrick: syntax: Implementation dependent, some possibilities include naming conventions, enumerated lists, and naming conventions, and others.
dom: relevance?
... both
... TestFAQ?
patrick: we need to add a
question to the Test FAQ
... how to deal with DOV
dom: I don't think we should have DOV in the question
patrick: something explaining the DOV.
dom: Example?
... interactive versus automated.
... voice versus dtmf
... tests marked for various profile
... SVG Tiny/Full had a subnaming convention.
... Comments: @@to be done with the capture of the
discussion@@
lofton: what about DOV keyword?
dom: we just discussed that it
was a kind of grouping
... what would be the gain?
lofton: For groupings other that
driven variability ones, arbitrary groupings, I don't think
there's a compelling argument for calling them
recommended.
... for variability it's tied to the conformance clause
dom: but you said the recent SVG Test suite didn't deal with it
lofton: the first incarnation
dealt with it.
... you can't test if you don't have grouping for
variability
lynne: maybe we need something for conformance and variability testing
lofton: I don't think it should be buried in grouping
karl: I don't think it's part of the test case
dom: lofton is saying that it's part of the conformance criteria and the test case should say which profiles belong to
lynne: the reasons you would like to group is related to the conformance.
karl: could we work with a
practical example?
... (discussion)
... explaining that a specific test case can be called by many
specifications
... and then we will have a huge list of conformance
grouping.
discussion between lynne and patrick
patrick: we have to be very
explicit in the examples and comments about the ways to use
grouping
... there are many different cases.
karl: (examples for WebCGM, XSLT,
Java world, xml:lang attributes, etc.)
... I think it's a meta level which is not tied to the test
case
lofton: yes I can buy that.
dom: (it seems we have an
agreement)
... versioning?
patrick: we will have the same problem.
dom: version of the test case? version of the test suite? version of the specification?
patrick: a test which is valid for a spec A at a time doesn't say that the test will not be useful for another spec B later on.
dom: it seems to be another kind grouping
karl: how do we relate versioning and status?
dom: you might want to keep version control for the development of your test case.
patrick: Identifier blablabla version X for keeping an history in time. Ok I agree with that
lynne: this is the version of the test itself.
dom: Name: version
QAWG: Description: "Identifier
which allows one to distinguish between different revisions of
test case"
... Rationale: "It @@provides@@ an history of the evolution of
the test case."
... Required yes
... Relevance: both
... Syntax: Free to implementation (see examples)
... TestFAQ: Link 14.
... Examples: look at CSS TS with latest etc. @@@@
(break)
QAWG: name: Contributor
... See Dublin Core
... discussion about ownership, about people submitting test
but didn't want to assume the management of the test
patrick: you might want to know, who did this test so that it can be fix by the same entity who created the test.
dom: we have several
questions.
... do we keep the metadata
... do we keep that name
lynne: I want to know who and/or where the test case came from. I need at least one of those.
patrick: you want to be able to identify the legal ownership of the test before the donation.
QAWG: name chosen is
Contributor
... description: "Individual or organization that contributed
this test case."
... Rationale: "It might be important to track the origin of
the test for technical reasons, legal issues, modifications,
etc."
... Required: optional
... Syntax: hypertext
... relevance: primary development
... TestFAQ: Question 10
... Example: XML Test Suite attribution to different
contributors
... includes contribution from Microsoft, IBM, Sun, etc. @@link
to an example@@
dom: dates?
karl: we can have dates about many things
dom; we should skip it
lynne: wait a minute, I don't want to get rid completely of it.
karl: I agree with lynne. And I think we should deal with it in the introduction.
QAWG: It needs to be discussed somewhere.
Lynne: Maybe in a note section after the metadata elements
karl: agreed
QAWG: Types: it's just a kind of
grouping
... everyone agreed
... Source?
... Not that important
... Language?
... should be like date in the note.
... Coverage?
<scribe> ... Dropped
UNKNOWN_SPEAKER: Rights?
patrick: copyrights are dealt
with a tangible form already. We should not try to address in a
particular way.
... we may be stepped on lawyer's feet.
dom: We are not saying how they should be managed, we are giving a pointer.
karl: it's a pointer to the rights, not the rights
QAWG: Rights
... description
... Description: Information about rights held in and over the
test case.
... Rationale: Publishers and Users of the test case need to
know where the legal rights lie
... Requires: Optional
... Syntax: Ask your lawyer.
... Relevance: both
... Test FAQ: Q. 10
... Examples: @@To find an example@@
... Priority? no interest
... References? no interest
See mail for the previous discussion: http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0020.html
dom: what do we do with that?
QAWG: We decided to publish this
as a W3C WG Note. Patrick Curran will write the prose,
introduction + notes + Issues, The text will be reviewed by
Richard Kennedy
... The text will be put in a ready to publish form by
Karl
... before August 31.
dom: what do people think about creating actual schema for this Test Case Model.
QAWG: Great Idea
karl: do we try to gather people around it?
patrick: We should try first to
create a first version of an XML Schema and/or RDF Schema and
try to gather people after it.
... I would focus more on tools and very practical things.
karl: by tools?
patrick: something concrete and
specific, the schema is too abstract, but something that can be
usable: tools, programs, templates
... that might interest people to work on it
dom: Agenda for tomorrow morning.
QA IG deliverables for rechartering.
... teleconferences
... next F2F?
lofton: Next TP in France end of Feb 2006.
karl: Basically future work
patrick: shopping list of what we could for the next months
ADJJOURNED
<dom> Meeting: QA WG Dublin F2F Day 2
This is scribe.perl Revision: 1.126 of Date: 2005/05/16 16:49:48 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/rational/rationale/ Succeeded: s/there are/there may be/ Succeeded: s/dtmg/dtmf/ Succeeded: s/test suite/test case/ Found Scribe: dom Inferring ScribeNick: dom WARNING: No scribe lines found matching previous ScribeNick pattern: <karl> ... Found ScribeNick: karl ScribeNicks: karl, dom Present: Karl Lofton Patrick Lynne Mark Richard Dom Agenda: http://lists.w3.org/Archives/Public/www-qa-wg/2005Aug/0004.html Got date from IRC log name: 9 Aug 2005 Guessing minutes URL: http://www.w3.org/2005/08/09-qa-minutes.html People with action items:[End of scribe.perl diagnostic output]