13:45:35 RRSAgent has joined #mwts 13:45:35 logging to http://www.w3.org/2007/01/25-mwts-irc 13:45:47 Meeting: MWI Test Suites Working Group F2F, Cambridge MA 13:45:53 Chair: Dom, Carmelo 13:46:03 Agenda: http://www.w3.org/2005/MWI/Tests/Meetings/Cambridge/agenda 13:46:07 RRSAgent, make log public 13:46:15 Zakim, this will be MWTS 13:46:15 I do not see a conference matching that name scheduled within the next hour, dom 13:46:21 Zakim, this will be TS 13:46:21 ok, dom; I see MWI_TSWG()9:00AM scheduled to start in 14 minutes 14:03:31 joachim has joined #mwts 14:03:47 Agenda: http://www.w3.org/2005/MWI/Tests/Meetings/Cambridge/agenda 14:06:43 Present: Allen, Carmelo, Christophe, Joachim, Dom 14:12:05 Regrets: Lynne, Till, Dmitri, DavidRooks, WukKim, Andrea 14:12:08 Christophe has joined #mwts 14:12:35 Allen has joined #mwts 14:12:38 Topic: Logistics, Agenda review 14:14:02 carmelo has joined #mwts 14:22:33 MWI_TSWG()9:00AM has now started 14:22:40 +??P6 14:26:14 Zakim, code? 14:26:14 the conference code is 8794 (tel:+1.617.761.6200 tel:+33.4.89.06.34.99 tel:+44.117.370.6152), dom 14:26:20 dsilaev has joined #mwts 14:26:43 hi dmitri, we'll start the call soon 14:26:55 Hi! 14:27:02 OK 14:27:17 allen: our Java ME test suites are used to make sure the devices under test conform to a set of well-known devices 14:27:31 ... my interests are around Web development and java programming 14:27:49 ... in terms of expected outcome, I'm quite open and would like to hear other people opinions 14:28:18 ... We need to deliver some kind of test suites that can run on different UA on devices 14:28:29 ... whether it is conformance test suite or something else, we need to make a decision 14:28:52 ... I have some experience in conformance testing on devices, some of which I can share 14:28:59 ... in case it's useful for this group 14:31:13 + +1.617.324.aaaa 14:31:22 Zakim, aaaa is roomD407 14:31:22 +roomD407; got it 14:31:32 dsilaev, can you hear us? 14:32:54 Carmelo: working for NIST (US National Institute of Standard and Technologies) 14:33:04 ... been doing testing for the past 20 years 14:33:47 ... was involved with the DOM Test Suites 14:33:51 ... the XSL FO Test Suite 14:33:58 ... XSLT Test suite 14:34:08 ... and most recently, been with the XML Query WG 14:34:22 ... which has just released their Recommendation 14:34:30 ... NIST wrote about 60% of the test suite 14:35:01 ... new to mobile web groups 14:35:39 ... in terms of expectations, we may be doing both conformance and interoperability testing 14:36:18 ... I would like to suggest that we look at the existing work that exists for catalog, infrastructure, ... 14:36:27 ... so that we don't reinvent the wheel 14:36:39 ... NIST is very happy to be part of this 14:37:18 Christophie Strobel, from louvain university (belgium): involved with accessibility groups 14:37:32 ... working esp. on test suites for WCAG 2.0 14:38:05 ... WCAG and ERT (evaluation and repair tools) WG have set up a task forces to develop test suites for WCAG 2 14:38:27 ... we have a test case description language developed for our European project (ben2web?) 14:38:44 ... we have set up some infrastructure (cvs, wiki) for setting up this tests 14:38:56 s/this/these/ 14:39:14 ... I'm hoping we can share ideas and maybe infrastructure for test suites in W3C 14:39:55 Joachim, Drutt: business of mobile delivery platform, primarily for mobile operators 14:40:06 ... have been involved in the device description Working Group for about a year 14:40:13 ... I'm interested to see what's this WG is about 14:40:19 ... and whether Drutt should be involved 14:40:31 ... We have a test suite for devices testing 14:40:47 ... with a pragmatic appraoch: trying to see what a device can do wrt our delivery platform 14:40:52 ... Basic testing 14:40:55 s/Christophie Strobel, from louvain university (belgium)/Christophe Strobbe, from Katholieke Universiteit Leuven (Belgium)/ 14:41:21 I am going to recall due to bad connection 14:41:21 ... Would be interesting to see whether this group can come up with better/automated testing methodology 14:41:26 -??P6 14:41:52 ... one of the goals being to check that the content produced by our platform appears the way it is supposed to 14:42:55 +??P6 14:43:22 Zakim, ??P6 is Dmitri 14:43:22 +Dmitri; got it 14:44:38 [Allen presents slides on "TCK 101"] 14:45:14 TCK stands Technology Compatibility Kit 14:45:27 it has test cases, tools, and harness 14:45:42 ... plus set of documentations (users guide, releases notes, rules for conformance testing) 14:46:06 ... TCK testing is strictly based on specifications 14:46:16 ... no assumptions on the OS or hardware, blackbox testing 14:46:31 ... we don't the quality of an inmplementation, nor robustness, performance, etc. 14:47:24 ... Java ME contains at least a dozen specs called JSR, which are under the Java community process 14:47:31 ... each JSR is targeted at a given platform 14:47:40 ... targets a certain of functionalities in the devices 14:48:02 ... specs are layered in stacks, e.g. a given spec is based on a set of underlying specs 14:48:47 ... each approved JSR must come with a tck that enforces the compatibility 14:49:08 ... these tck are available under licenses 14:49:23 ... (but can be obtained for free under certain conditions) 14:50:16 -> http://jcp.org/en/jsr/all List of Java Specification Requests (JSR) 14:52:37 ... We have a test harness called JavaTest 14:52:45 ... it provides quite a lot of functions 14:53:10 ... test execution, requiring a plugin to communicate with the device 14:53:30 ... configuration also done through the harness 14:53:35 ... tests selection 14:54:08 -> http://java.sun.com/developer/technicalArticles/JCPtools2/ The Java Compatibility Test Tools: JavaTest Harness 14:55:34 ... JavaTest used only on Java technologies at some point - don't know if that's an option 14:55:55 ... if the user agent is written in java, shouldn't be a difficulty 14:56:13 ... our tests are mostly positive, some are negative 14:56:27 ... we test whatever is specified in JSR 14:56:54 ... the tests give a clear result (pass, fail, error) 15:06:23 ... [describing the test development process, incl. test assertions] 15:08:10 -> http://java.sun.com/developer/technicalArticles/JCPtools/ The Java Compatibility Test Tools 15:18:12 ... javatest is open source 15:29:47 -> http://www.jcp.org/en/resources/tdk#java_ctt Java™ Compatibility Test Tools 15:40:49 dsilaev has joined #mwts 16:32:03 List of test suites (some possibly relevant to mobile web): http://www.bentoweb.org/html/BenToWeb_D4.1.html#heading28 16:41:35 acid2 test 16:42:42 URL: http://www.webstandards.org/action/acid2/ 16:55:44 DOM Test Suite Methodology Report: http://www.itl.nist.gov/div897/ctg/conformance/DOMTSmethod.pdf 17:07:40 [demo of the MIDP TCK] 17:19:16 dom shows MWI Best Practices tests for Encdoding Declaration Support: public-mwts-request@w3.org 17:19:56 s/public-mwts-request@w3.org/'http://www.w3.org/2005/MWI/BPWG/techs/EncodingDeclarationSupport'/ 17:27:46 Allen has left #mwts 17:28:30 -roomD407 17:28:31 MWI_TSWG()9:00AM has ended 17:28:32 Attendees were +1.617.324.aaaa, roomD407, Dmitri 17:29:58 dsilaev has left #mwts 17:32:00 Zakim has left #mwts 18:10:53 Christophe has joined #mwts 18:11:22 Zakim has joined #mwts 18:26:58 dom has joined #mwts 18:27:19 Zakim, code? 18:27:19 sorry, dom, I don't know what conference this is 18:27:33 Zakim, this is tswg 18:27:34 dom, I see MWI_TSWG()9:00AM in the schedule but not yet started. Perhaps you mean "this will be tswg". 18:27:41 Zakim, this will be tswg 18:27:41 ok, dom; I see MWI_TSWG()9:00AM scheduled to start 267 minutes ago 18:27:47 Zakim, code? 18:27:47 the conference code is 8794 (tel:+1.617.761.6200 tel:+33.4.89.06.34.99 tel:+44.117.370.6152), dom 18:28:40 MWI_TSWG()9:00AM has now started 18:28:47 +roomD407 18:30:38 +??P3 18:30:53 Zakim, ??P3 is dmitri 18:30:53 +dmitri; got it 18:33:52 carmelo has joined #mwts 18:35:25 dsilaev has joined #mwts 18:36:16 montane has joined #mwts 18:42:14 +Lynne_Rosenthal 18:43:18 Allen has joined #mwts 18:46:29 carmelo has joined #mwts 18:46:58 Topic: scope of our work 18:47:03 Dom: conformance or interop testing? 18:47:17 carmelo: both would be interesting 18:47:25 ... maybe with a greater focus on conformance 18:47:38 ... depending on the interest from the browser community on interop 18:47:54 ... would be interesting to know what user agents are looking for 18:48:05 s/user agents/user agents developers/ 18:48:14 ... and see what's their needs are 18:49:03 Allen: the kind of ts we will produce depends on our charter 18:49:25 ... our charter mentions conformance testing 18:50:14 ... but dom mentioned we could orient it one way or the other 18:50:20 ... shouldn't we try to finalize the charter? 18:50:27 dom: would like to do so in this very discussion, indeed 18:50:38 allen: what's the most desired outcome of this WG? 18:50:45 ... is conformance the most important or interop? 18:50:56 carmelo: probably equally important 18:51:06 ... if you're conformant, it's likely that there is interop 18:51:12 ... but the reverse may not be true 18:51:43 -Lynne_Rosenthal 18:54:46 dom: problem with conformance testing is that there may not be so many requirements established for user agents 18:55:01 ... e.g. XHTML Basic defines conformance more for documents than for user agents 18:55:35 carmelo: given the nature of browsing, it may be more interesting to focus on interop 18:55:39 ... e.g. on rendering 18:56:02 ... looking at acid2 and how it appeared in IE was interesting 18:59:52 dmitri: how do you define interop testing? 19:00:40 dom: testing that focuses on how user agents react to actual authoring practices (vs ill-defined requirements) 19:01:01 carmelo: don't think there has been much effort in interop testing for browsers 19:01:07 ... at least in W3C 19:21:06 ACTION: carmelo to look if NIST has a test cases/results submission system 19:22:59 ACTION: Allen to check if there is any test case submission system/process as part of the JCP 19:25:03 ACTION: Dom to look into test cases submissions systems 19:33:11 WAI: Techniques for WCAG 2.0 submission form: http://www.w3.org/WAI/GL/WCAG20/TECHS-SUBMIT/ 19:33:52 This form submits the data to a mailing list with public archive (http://lists.w3.org/Archives/Public/public-wcag2-techs/); Working Group reviews them later. 19:37:29 Christophe has joined #mwts 19:37:48 ACTION: Carmelo to report on the XSLT/XQuery metadata for test cases 19:39:17 -> http://www.w3.org/TR/2005/NOTE-test-metadata-20050914/ Test Metadata 19:39:46 Carmelo: in case of our interops test, we would also need to categorize tests in terms of technologies involved (css, markup, etc) 19:43:24 -roomD407 19:43:25 -dmitri 19:43:27 MWI_TSWG()9:00AM has ended 19:43:29 Attendees were roomD407, dmitri, Lynne_Rosenthal 19:43:37 dsilaev has left #mwts 20:19:33 Christophe has joined #mwts 20:20:36 test suites licenses 20:20:47 public availability 20:22:57 conformance / interoperability 20:23:14 coverage / test assertions avaibility 20:23:21 mobile web relevance 20:23:49 test harness 20:23:53 automated or not 20:26:06 goals: creating a portal of available test suites 20:26:19 ... evaluate the possibility of packaging them together 20:27:58 stability / status 20:28:12 maintenance process 20:36:33 2 main deliverables: 20:36:54 * Survey of existing "conformance" test suites (based on criteria established above) 20:37:02 Zakim has left #mwts 20:37:06 ... possibly leading to packaging them together 20:37:14 ... and advertizing them to user agent developers 20:37:22 * a contributions-based test suites 20:37:36 ... accepting contributions both for test cases and test results 20:37:37 Zakim has joined #mwts 20:38:00 ... with 2 outcomes: a packaged set of test cases 20:38:17 ... a report of how current user agents do on these test cases 20:40:13 ... that would need to define a test process 20:43:44 ... including to define what's in scope and not (e.g. is non-conformant content acceptable?) 20:43:50 ... and a formal review process by the WG 20:45:27 WCAG TSD TF review process: http://www.w3.org/WAI/ER/2006/tests/process 20:47:10 ... to identify existing sources of tests cases 20:47:48 http://www.w3.org/XML/Query/test-suite/Guidelines%20for%20Test%20Submission.html - URL for Query Test Submissions 20:48:06 ... tools to submit and review the test cases 20:48:39 ... tools to gather test results 20:51:25 ... a preliminary set of tests results based on testing internal to the WG 20:53:44 ... a simple test harness to navigate through the tests 20:55:25 [note on properties for existing test suites review: how to navigate through the test cases?] 20:59:53 ACTION: Dom to turn the list of work items into a roadmap with a schedule 21:02:15 Topic: F2F scheduling 21:02:21 Carmelo: flexible in frequency 21:02:36 Allen: may be difficult to go outside of the US 21:02:59 ... but if in Europe, Dmitri could attend 21:07:25 Dom: periods would be either April or May 21:07:35 Carmelo: prefers meeting starting in the middle of the week 21:10:32 RRSAgent, draft minutes 21:10:32 I have made the request to generate http://www.w3.org/2007/01/25-mwts-minutes.html dom 21:14:57 s/middle of the week/middle or the end of the week/ 21:15:00 RRSAgent, draft minutes 21:15:00 I have made the request to generate http://www.w3.org/2007/01/25-mwts-minutes.html dom 21:23:45 [adjourned] 21:24:16 ACTION: carmelo to send agenda for Tuesdday meeting 21:25:52 RRSAgent, bye 21:25:52 I see 6 open action items saved in http://www.w3.org/2007/01/25-mwts-actions.rdf : 21:25:52 ACTION: carmelo to look if NIST has a test cases/results submission system [1] 21:25:52 recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-21-06 21:25:52 ACTION: Allen to check if there is any test case submission system/process as part of the JCP [2] 21:25:52 recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-22-59 21:25:52 ACTION: Dom to look into test cases submissions systems [3] 21:25:52 recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-25-03 21:25:52 ACTION: Carmelo to report on the XSLT/XQuery metadata for test cases [4] 21:25:52 recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-37-48 21:25:52 ACTION: Dom to turn the list of work items into a roadmap with a schedule [5] 21:25:52 recorded in http://www.w3.org/2007/01/25-mwts-irc#T20-59-53 21:25:52 ACTION: carmelo to send agenda for Tuesdday meeting [6] 21:25:52 recorded in http://www.w3.org/2007/01/25-mwts-irc#T21-24-16