IRC log of mwts on 2007-01-25

Timestamps are in UTC.

13:45:35 [RRSAgent]
RRSAgent has joined #mwts
13:45:35 [RRSAgent]
logging to http://www.w3.org/2007/01/25-mwts-irc
13:45:47 [dom]
Meeting: MWI Test Suites Working Group F2F, Cambridge MA
13:45:53 [dom]
Chair: Dom, Carmelo
13:46:03 [dom]
Agenda: http://www.w3.org/2005/MWI/Tests/Meetings/Cambridge/agenda
13:46:07 [dom]
RRSAgent, make log public
13:46:15 [dom]
Zakim, this will be MWTS
13:46:15 [Zakim]
I do not see a conference matching that name scheduled within the next hour, dom
13:46:21 [dom]
Zakim, this will be TS
13:46:21 [Zakim]
ok, dom; I see MWI_TSWG()9:00AM scheduled to start in 14 minutes
14:03:31 [joachim]
joachim has joined #mwts
14:03:47 [dom]
Agenda: http://www.w3.org/2005/MWI/Tests/Meetings/Cambridge/agenda
14:06:43 [dom]
Present: Allen, Carmelo, Christophe, Joachim, Dom
14:12:05 [dom]
Regrets: Lynne, Till, Dmitri, DavidRooks, WukKim, Andrea
14:12:08 [Christophe]
Christophe has joined #mwts
14:12:35 [Allen]
Allen has joined #mwts
14:12:38 [dom]
Topic: Logistics, Agenda review
14:14:02 [carmelo]
carmelo has joined #mwts
14:22:33 [Zakim]
MWI_TSWG()9:00AM has now started
14:22:40 [Zakim]
+??P6
14:26:14 [dom]
Zakim, code?
14:26:14 [Zakim]
the conference code is 8794 (tel:+1.617.761.6200 tel:+33.4.89.06.34.99 tel:+44.117.370.6152), dom
14:26:20 [dsilaev]
dsilaev has joined #mwts
14:26:43 [dom]
hi dmitri, we'll start the call soon
14:26:55 [dsilaev]
Hi!
14:27:02 [dsilaev]
OK
14:27:17 [dom]
allen: our Java ME test suites are used to make sure the devices under test conform to a set of well-known devices
14:27:31 [dom]
... my interests are around Web development and java programming
14:27:49 [dom]
... in terms of expected outcome, I'm quite open and would like to hear other people opinions
14:28:18 [dom]
... We need to deliver some kind of test suites that can run on different UA on devices
14:28:29 [dom]
... whether it is conformance test suite or something else, we need to make a decision
14:28:52 [dom]
... I have some experience in conformance testing on devices, some of which I can share
14:28:59 [dom]
... in case it's useful for this group
14:31:13 [Zakim]
+ +1.617.324.aaaa
14:31:22 [dom]
Zakim, aaaa is roomD407
14:31:22 [Zakim]
+roomD407; got it
14:31:32 [dom]
dsilaev, can you hear us?
14:32:54 [dom]
Carmelo: working for NIST (US National Institute of Standard and Technologies)
14:33:04 [dom]
... been doing testing for the past 20 years
14:33:47 [dom]
... was involved with the DOM Test Suites
14:33:51 [dom]
... the XSL FO Test Suite
14:33:58 [dom]
... XSLT Test suite
14:34:08 [dom]
... and most recently, been with the XML Query WG
14:34:22 [dom]
... which has just released their Recommendation
14:34:30 [dom]
... NIST wrote about 60% of the test suite
14:35:01 [dom]
... new to mobile web groups
14:35:39 [dom]
... in terms of expectations, we may be doing both conformance and interoperability testing
14:36:18 [dom]
... I would like to suggest that we look at the existing work that exists for catalog, infrastructure, ...
14:36:27 [dom]
... so that we don't reinvent the wheel
14:36:39 [dom]
... NIST is very happy to be part of this
14:37:18 [dom]
Christophie Strobel, from louvain university (belgium): involved with accessibility groups
14:37:32 [dom]
... working esp. on test suites for WCAG 2.0
14:38:05 [dom]
... WCAG and ERT (evaluation and repair tools) WG have set up a task forces to develop test suites for WCAG 2
14:38:27 [dom]
... we have a test case description language developed for our European project (ben2web?)
14:38:44 [dom]
... we have set up some infrastructure (cvs, wiki) for setting up this tests
14:38:56 [dom]
s/this/these/
14:39:14 [dom]
... I'm hoping we can share ideas and maybe infrastructure for test suites in W3C
14:39:55 [dom]
Joachim, Drutt: business of mobile delivery platform, primarily for mobile operators
14:40:06 [dom]
... have been involved in the device description Working Group for about a year
14:40:13 [dom]
... I'm interested to see what's this WG is about
14:40:19 [dom]
... and whether Drutt should be involved
14:40:31 [dom]
... We have a test suite for devices testing
14:40:47 [dom]
... with a pragmatic appraoch: trying to see what a device can do wrt our delivery platform
14:40:52 [dom]
... Basic testing
14:40:55 [Christophe]
s/Christophie Strobel, from louvain university (belgium)/Christophe Strobbe, from Katholieke Universiteit Leuven (Belgium)/
14:41:21 [dsilaev]
I am going to recall due to bad connection
14:41:21 [dom]
... Would be interesting to see whether this group can come up with better/automated testing methodology
14:41:26 [Zakim]
-??P6
14:41:52 [dom]
... one of the goals being to check that the content produced by our platform appears the way it is supposed to
14:42:55 [Zakim]
+??P6
14:43:22 [dom]
Zakim, ??P6 is Dmitri
14:43:22 [Zakim]
+Dmitri; got it
14:44:38 [dom]
[Allen presents slides on "TCK 101"]
14:45:14 [dom]
TCK stands Technology Compatibility Kit
14:45:27 [dom]
it has test cases, tools, and harness
14:45:42 [dom]
... plus set of documentations (users guide, releases notes, rules for conformance testing)
14:46:06 [dom]
... TCK testing is strictly based on specifications
14:46:16 [dom]
... no assumptions on the OS or hardware, blackbox testing
14:46:31 [dom]
... we don't the quality of an inmplementation, nor robustness, performance, etc.
14:47:24 [dom]
... Java ME contains at least a dozen specs called JSR, which are under the Java community process
14:47:31 [dom]
... each JSR is targeted at a given platform
14:47:40 [dom]
... targets a certain of functionalities in the devices
14:48:02 [dom]
... specs are layered in stacks, e.g. a given spec is based on a set of underlying specs
14:48:47 [dom]
... each approved JSR must come with a tck that enforces the compatibility
14:49:08 [dom]
... these tck are available under licenses
14:49:23 [dom]
... (but can be obtained for free under certain conditions)
14:50:16 [dom]
-> http://jcp.org/en/jsr/all List of Java Specification Requests (JSR)
14:52:37 [dom]
... We have a test harness called JavaTest
14:52:45 [dom]
... it provides quite a lot of functions
14:53:10 [dom]
... test execution, requiring a plugin to communicate with the device
14:53:30 [dom]
... configuration also done through the harness
14:53:35 [dom]
... tests selection
14:54:08 [dom]
-> http://java.sun.com/developer/technicalArticles/JCPtools2/ The Java Compatibility Test Tools: JavaTest Harness
14:55:34 [dom]
... JavaTest used only on Java technologies at some point - don't know if that's an option
14:55:55 [dom]
... if the user agent is written in java, shouldn't be a difficulty
14:56:13 [dom]
... our tests are mostly positive, some are negative
14:56:27 [dom]
... we test whatever is specified in JSR
14:56:54 [dom]
... the tests give a clear result (pass, fail, error)
15:06:23 [dom]
... [describing the test development process, incl. test assertions]
15:08:10 [dom]
-> http://java.sun.com/developer/technicalArticles/JCPtools/ The Java Compatibility Test Tools
15:18:12 [dom]
... javatest is open source
15:29:47 [dom]
-> http://www.jcp.org/en/resources/tdk#java_ctt Java™ Compatibility Test Tools
15:40:49 [dsilaev]
dsilaev has joined #mwts
16:32:03 [Christophe]
List of test suites (some possibly relevant to mobile web): http://www.bentoweb.org/html/BenToWeb_D4.1.html#heading28
16:41:35 [dom]
acid2 test
16:42:42 [Christophe]
URL: http://www.webstandards.org/action/acid2/
16:55:44 [Christophe]
DOM Test Suite Methodology Report: http://www.itl.nist.gov/div897/ctg/conformance/DOMTSmethod.pdf
17:07:40 [dom]
[demo of the MIDP TCK]
17:19:16 [Christophe]
dom shows MWI Best Practices tests for Encdoding Declaration Support: public-mwts-request@w3.org
17:19:56 [Christophe]
s/public-mwts-request@w3.org/'http://www.w3.org/2005/MWI/BPWG/techs/EncodingDeclarationSupport'/
17:27:46 [Allen]
Allen has left #mwts
17:28:30 [Zakim]
-roomD407
17:28:31 [Zakim]
MWI_TSWG()9:00AM has ended
17:28:32 [Zakim]
Attendees were +1.617.324.aaaa, roomD407, Dmitri
17:29:58 [dsilaev]
dsilaev has left #mwts
17:32:00 [Zakim]
Zakim has left #mwts
18:10:53 [Christophe]
Christophe has joined #mwts
18:11:22 [Zakim]
Zakim has joined #mwts
18:26:58 [dom]
dom has joined #mwts
18:27:19 [dom]
Zakim, code?
18:27:19 [Zakim]
sorry, dom, I don't know what conference this is
18:27:33 [dom]
Zakim, this is tswg
18:27:34 [Zakim]
dom, I see MWI_TSWG()9:00AM in the schedule but not yet started. Perhaps you mean "this will be tswg".
18:27:41 [dom]
Zakim, this will be tswg
18:27:41 [Zakim]
ok, dom; I see MWI_TSWG()9:00AM scheduled to start 267 minutes ago
18:27:47 [dom]
Zakim, code?
18:27:47 [Zakim]
the conference code is 8794 (tel:+1.617.761.6200 tel:+33.4.89.06.34.99 tel:+44.117.370.6152), dom
18:28:40 [Zakim]
MWI_TSWG()9:00AM has now started
18:28:47 [Zakim]
+roomD407
18:30:38 [Zakim]
+??P3
18:30:53 [dom]
Zakim, ??P3 is dmitri
18:30:53 [Zakim]
+dmitri; got it
18:33:52 [carmelo]
carmelo has joined #mwts
18:35:25 [dsilaev]
dsilaev has joined #mwts
18:36:16 [montane]
montane has joined #mwts
18:42:14 [Zakim]
+Lynne_Rosenthal
18:43:18 [Allen]
Allen has joined #mwts
18:46:29 [carmelo]
carmelo has joined #mwts
18:46:58 [dom]
Topic: scope of our work
18:47:03 [dom]
Dom: conformance or interop testing?
18:47:17 [dom]
carmelo: both would be interesting
18:47:25 [dom]
... maybe with a greater focus on conformance
18:47:38 [dom]
... depending on the interest from the browser community on interop
18:47:54 [dom]
... would be interesting to know what user agents are looking for
18:48:05 [dom]
s/user agents/user agents developers/
18:48:14 [dom]
... and see what's their needs are
18:49:03 [dom]
Allen: the kind of ts we will produce depends on our charter
18:49:25 [dom]
... our charter mentions conformance testing
18:50:14 [dom]
... but dom mentioned we could orient it one way or the other
18:50:20 [dom]
... shouldn't we try to finalize the charter?
18:50:27 [dom]
dom: would like to do so in this very discussion, indeed
18:50:38 [dom]
allen: what's the most desired outcome of this WG?
18:50:45 [dom]
... is conformance the most important or interop?
18:50:56 [dom]
carmelo: probably equally important
18:51:06 [dom]
... if you're conformant, it's likely that there is interop
18:51:12 [dom]
... but the reverse may not be true
18:51:43 [Zakim]
-Lynne_Rosenthal
18:54:46 [dom]
dom: problem with conformance testing is that there may not be so many requirements established for user agents
18:55:01 [dom]
... e.g. XHTML Basic defines conformance more for documents than for user agents
18:55:35 [dom]
carmelo: given the nature of browsing, it may be more interesting to focus on interop
18:55:39 [dom]
... e.g. on rendering
18:56:02 [dom]
... looking at acid2 and how it appeared in IE was interesting
18:59:52 [dom]
dmitri: how do you define interop testing?
19:00:40 [dom]
dom: testing that focuses on how user agents react to actual authoring practices (vs ill-defined requirements)
19:01:01 [dom]
carmelo: don't think there has been much effort in interop testing for browsers
19:01:07 [dom]
... at least in W3C
19:21:06 [dom]
ACTION: carmelo to look if NIST has a test cases/results submission system
19:22:59 [dom]
ACTION: Allen to check if there is any test case submission system/process as part of the JCP
19:25:03 [dom]
ACTION: Dom to look into test cases submissions systems
19:33:11 [Christophe]
WAI: Techniques for WCAG 2.0 submission form: http://www.w3.org/WAI/GL/WCAG20/TECHS-SUBMIT/
19:33:52 [Christophe]
This form submits the data to a mailing list with public archive (http://lists.w3.org/Archives/Public/public-wcag2-techs/); Working Group reviews them later.
19:37:29 [Christophe]
Christophe has joined #mwts
19:37:48 [dom]
ACTION: Carmelo to report on the XSLT/XQuery metadata for test cases
19:39:17 [dom]
-> http://www.w3.org/TR/2005/NOTE-test-metadata-20050914/ Test Metadata
19:39:46 [dom]
Carmelo: in case of our interops test, we would also need to categorize tests in terms of technologies involved (css, markup, etc)
19:43:24 [Zakim]
-roomD407
19:43:25 [Zakim]
-dmitri
19:43:27 [Zakim]
MWI_TSWG()9:00AM has ended
19:43:29 [Zakim]
Attendees were roomD407, dmitri, Lynne_Rosenthal
19:43:37 [dsilaev]
dsilaev has left #mwts
20:19:33 [Christophe]
Christophe has joined #mwts
20:20:36 [dom]
test suites licenses
20:20:47 [dom]
public availability
20:22:57 [dom]
conformance / interoperability
20:23:14 [dom]
coverage / test assertions avaibility
20:23:21 [dom]
mobile web relevance
20:23:49 [dom]
test harness
20:23:53 [dom]
automated or not
20:26:06 [dom]
goals: creating a portal of available test suites
20:26:19 [dom]
... evaluate the possibility of packaging them together
20:27:58 [dom]
stability / status
20:28:12 [dom]
maintenance process
20:36:33 [dom]
2 main deliverables:
20:36:54 [dom]
* Survey of existing "conformance" test suites (based on criteria established above)
20:37:02 [Zakim]
Zakim has left #mwts
20:37:06 [dom]
... possibly leading to packaging them together
20:37:14 [dom]
... and advertizing them to user agent developers
20:37:22 [dom]
* a contributions-based test suites
20:37:36 [dom]
... accepting contributions both for test cases and test results
20:37:37 [Zakim]
Zakim has joined #mwts
20:38:00 [dom]
... with 2 outcomes: a packaged set of test cases
20:38:17 [dom]
... a report of how current user agents do on these test cases
20:40:13 [dom]
... that would need to define a test process
20:43:44 [dom]
... including to define what's in scope and not (e.g. is non-conformant content acceptable?)
20:43:50 [dom]
... and a formal review process by the WG
20:45:27 [Christophe]
WCAG TSD TF review process: http://www.w3.org/WAI/ER/2006/tests/process
20:47:10 [dom]
... to identify existing sources of tests cases
20:47:48 [carmelo]
http://www.w3.org/XML/Query/test-suite/Guidelines%20for%20Test%20Submission.html - URL for Query Test Submissions
20:48:06 [dom]
... tools to submit and review the test cases
20:48:39 [dom]
... tools to gather test results
20:51:25 [dom]
... a preliminary set of tests results based on testing internal to the WG
20:53:44 [dom]
... a simple test harness to navigate through the tests
20:55:25 [dom]
[note on properties for existing test suites review: how to navigate through the test cases?]
20:59:53 [dom]
ACTION: Dom to turn the list of work items into a roadmap with a schedule
21:02:15 [dom]
Topic: F2F scheduling
21:02:21 [dom]
Carmelo: flexible in frequency
21:02:36 [dom]
Allen: may be difficult to go outside of the US
21:02:59 [dom]
... but if in Europe, Dmitri could attend
21:07:25 [dom]
Dom: periods would be either April or May
21:07:35 [dom]
Carmelo: prefers meeting starting in the middle of the week
21:10:32 [dom]
RRSAgent, draft minutes
21:10:32 [RRSAgent]
I have made the request to generate http://www.w3.org/2007/01/25-mwts-minutes.html dom
21:14:57 [dom]
s/middle of the week/middle or the end of the week/
21:15:00 [dom]
RRSAgent, draft minutes
21:15:00 [RRSAgent]
I have made the request to generate http://www.w3.org/2007/01/25-mwts-minutes.html dom
21:23:45 [dom]
[adjourned]
21:24:16 [dom]
ACTION: carmelo to send agenda for Tuesdday meeting
21:25:52 [dom]
RRSAgent, bye
21:25:52 [RRSAgent]
I see 6 open action items saved in http://www.w3.org/2007/01/25-mwts-actions.rdf :
21:25:52 [RRSAgent]
ACTION: carmelo to look if NIST has a test cases/results submission system [1]
21:25:52 [RRSAgent]
recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-21-06
21:25:52 [RRSAgent]
ACTION: Allen to check if there is any test case submission system/process as part of the JCP [2]
21:25:52 [RRSAgent]
recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-22-59
21:25:52 [RRSAgent]
ACTION: Dom to look into test cases submissions systems [3]
21:25:52 [RRSAgent]
recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-25-03
21:25:52 [RRSAgent]
ACTION: Carmelo to report on the XSLT/XQuery metadata for test cases [4]
21:25:52 [RRSAgent]
recorded in http://www.w3.org/2007/01/25-mwts-irc#T19-37-48
21:25:52 [RRSAgent]
ACTION: Dom to turn the list of work items into a roadmap with a schedule [5]
21:25:52 [RRSAgent]
recorded in http://www.w3.org/2007/01/25-mwts-irc#T20-59-53
21:25:52 [RRSAgent]
ACTION: carmelo to send agenda for Tuesdday meeting [6]
21:25:52 [RRSAgent]
recorded in http://www.w3.org/2007/01/25-mwts-irc#T21-24-16