- From: Andrew Thackrah <andrew@opengroup.org>
- Date: Wed, 11 Jun 2003 14:39:42 +0100 (BST)
- To: <www-qa-wg@w3.org>
DRAFT MINUTES QA Working Group Teleconference Monday, 09-June-2003 -- Scribe: Andrew Thackrah Attendees: (PC) Patrick Curran (Sun Microsystems) (KD) Karl Dubost (W3C, WG co-chair) (PF) Peter Fawcett (RealNetworks) (KG) Kirill Gavrylyuk (Microsoft) (LH) Lofton Henderson (CGMO - WG co-chair) (LR) Lynne Rosenthal (NIST - IG co-chair) (MS) Mark Skall (NIST) (AT) Andrew Thackrah (Open Group) (DM) David Marston (IBM, guest) Regrets: (KD) Karl Dubost (W3C, WG co-chair) (DH) Dominique Hazaël-Massieux (W3C) (SM) Sandra Martinez (NIST) Absent: (DD) Dimitris Dimitriadis (Ontologicon) Summary of New Action Items: [no action items were assigned in this meeting] Agenda: <a href="http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0019.html"> http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0019.html</a> Previous Telcon Minutes: <a href="http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0010.html"> http://lists.w3.org/Archives/Public/www-qa-wg/2003Jun/0010.html</a> Minutes: Agenda 1. Roll call, 1100 EDT 1b. Crete F2F attendance 2. Last Call reviews 3. SpecGL issue processing 4. specGL extras 1. Roll call See above 1b. Crete face-to-face meeting atendance [This ad oc item was added during the meeting.] Daniel Dardailler will only be present for two day (Mon, Tue) All W3C team shoul dbe present LH, PC, MS, (and DD??) confirmed attendance. AT said yes, subject to travel approval. 2 last call reviews [none] 3 SpecGL issue processing ref2a from agenda: LR summarises: purpose is to stimulate discusion re. test assertions. e.g. definition of a Test Assertion (TA). Should SpecGL require TAs to be included? How formal should TAs be? LH - do we still have any fundamental confusion about TA definition? Seattle resolution added GL14 definition to the glossary. [LH reads this definition] MS - Is there a link to this definition? LH - Agenda link [1] then from TOC jump to GL14. Also in agenda is the ref to AT's explanation of the difference between a conformance requirement (CR) and an assertion and MS's example exercise. I believe a spec could be written with embedded TAs - but doesn't have to be that way. See my 3 examples in mail [ref] MS - difference of opinion. conformance requirements != assertions. We need to clarify this. LH - consider a spec with a formal notation: TAs are irrelevant because tests can be auto-generated LR - MUSTs etc are important and should be ID'd or tagged - it is not necessary to have TAs. Spec authors need to concentrate on the identification of CR that will result in TAs LH - SpecGL is an an example - not written with TAs MS - My example suggests that SpecGL is not not exact LH - SpecGL should be considered a reasonable way to write a spec MS - consider the issue of different people producing different TAs from the same document DM - an aside - even with TAs incuded - it will still be important to identify testable prose - ie we need tags for prose regardless of TAs KG - So what is the difference betweem TA & CRs? LH - agenda ref 2c - long discussion of this from AT. there was general agreement on this definition PC - there may be a many-to-one relationship between TA & CR... small CRs, large # TAs. For example- Java api specs: total number of TAs is 5 figures (tens of thousands). I want a spec to have assertions tagged. But there are only a small number of CRs - all high level. AT - are all TAs published? PC - yes, if possible. non-contiguous fragments of test may make up assertions, which is why it is important to be able to relate every TA back to spec using tags etc. LH - We had similar experience for SVG detailed tests. PC - So I think we should encourage (inclusion of) TAs where appropriate but not mandate. LR, LH, AT - agree MS - There are still two different things, logically: do we require spec authors to develop TSs when possible or leave it to test developers? PC - I'm in favour of encouragment, not mandates. In perfect world - spec developers would do it. MS - If the CRs are only in spec author's mind - then their TAs will be biased, wheres test developer's TAs will be objective DM - BNF/schema are sufficiently formal to be considered as good as TAs [group decides to ignore specs with formal grammars for the moment] PC - The fuzzier the language, the less useful. tagging testability helps remove subjectivity MS - But a TA must be mapped to normative spec language LH - The job of who does the work is irrelevant - so long as TAs are published in some form? MS - we should confirm who we want TAs to come from - test developers or spec authors KG - If TAs are written by spec authors - quality may be low. we use TAs a mechanism for spec reviews. So we have someone other than spec author make TAs - as a QA on the spec. [general agreement on value of this] LH - Should we require TAs by time we get to Rec.? KG - If it's priority 2 , then yes, desirable DM - It's a priority 3 to me. PC - Test case is useless without assertion [noisy discussion follows. Conclusion is that it is desirable (essential to some) to be able to trace an assertion back to a specific section (or sections) of spec. text. TA is link between test and spec. Test has no validity without this link.] LH - So TAs, ideally not created by spec author, give positive benfit to quality of spec. right? MS - Quality yes, also one set of TAs creates consistency of testing. LH - straw poll - keep GL 14? [all YES in principle] Three spin-off issues are identified that should be addressed in future meetings: 1 Priority 2 Creator (spec author | test developer ) 3 Auto-generation (spec->test via formal grammar) [argument about relationship between normative text and TAs follows] LR - There is the danger of setting the bar too high for spec writers to rephrase text as an assertion. [discussion follows regarding clarity of spec language ] MS - need to avoid words like obvious, clear. [referring to sugestions that TAs may be explicit if spec CRs are sufficiently clear. Considered too subjective]. AT - should TAs be mandatory? LH - issue 98 - decided at Tokyo. [There is discussion about who is mandated - test developers or product implementers] PC - We recognize that our tests can not give 100% coverage. We have a conformance clause - obliged to implement regardless. DM - Also need to handle bad assertions PC - TAs should not be mandatory for implementers - only for test developers. [There is some agreement that test developers should be mandated to implement any testable assertions but that it is undesirable for product implement to use TAs - they should implement directly from the specification conformance requirements] [summing up] LH - please give opionions on remainder of issues in [2a] in advance of crete. [agenda items 4+ not addressed] Adjourned 12:00
Received on Wednesday, 11 June 2003 09:39:48 UTC