13:00:21 <RRSAgent> RRSAgent has joined #testing
13:00:21 <RRSAgent> logging to http://www.w3.org/2011/04/04-testing-irc
13:00:32 <shadi> shadi has joined #testing
13:00:44 <plh> zakim, room for 6?
13:00:46 <Zakim> ok, plh; conference Team_(testing)13:00Z scheduled with code 26631 (CONF1) for 60 minutes until 1400Z
13:00:50 <Zakim> Team_(testing)13:00Z has now started
13:00:51 <Zakim> +Plh
13:01:03 <plh> oops
13:01:06 <Zakim> +Plh
13:01:09 <shadi> zakim, call shad-617
13:01:09 <Zakim> I am sorry, shadi; I do not know a number for shad-617
13:01:14 <shadi> zakim, call shadi-617
13:01:14 <Zakim> ok, shadi; the call is being made
13:01:16 <Zakim> +Shadi
13:01:20 <Zakim> +francois
13:01:21 <plh> zakim, drop plh
13:01:21 <Zakim> 'plh' is ambiguous, plh
13:01:27 <francois> zakim, mute me
13:01:27 <Zakim> francois should now be muted
13:01:42 <MichaelC> MichaelC has joined #testing
13:01:42 <Zakim> -Plh
13:01:44 <plh> zakim, drop plh
13:01:44 <Zakim> Plh is being disconnected
13:01:46 <Zakim> -Plh
13:01:49 <Zakim> +Plh
13:02:21 <francois> zakim, unmute me
13:02:21 <Zakim> francois should no longer be muted
13:02:49 <Zakim> +Michael_Cooper
13:03:17 <plh> agenda+ Closing on the requirements
13:03:17 <plh> agenda+ Testing Interest Group charter (Plh and Mike)
13:03:17 <plh> agenda+ Mozilla testing framework (Francois)
13:03:17 <plh> agenda+ Starting point for the framework
13:03:17 <plh> agenda+ Documentation for Groups
13:03:18 <plh> agenda+ Organizing w3c-test.org
13:03:25 <plh> agenda?
13:03:25 <MikeSmith> Zakim, call Mike-goog
13:03:33 <plh> zakim, drop agendum 2
13:03:37 <Zakim> ok, MikeSmith; the call is being made
13:03:38 <Zakim> +Mike
13:03:40 <Zakim> agendum 2, requirements page, dropped
13:03:48 <shadi> zakim, who is on the phone?
13:03:48 <Zakim> On the phone I see Plh, Shadi, francois, Michael_Cooper, Mike
13:04:00 <shadi> scribe: shadi
13:04:12 <shadi> zakim, mute me
13:04:12 <Zakim> Shadi should now be muted
13:04:14 <plh> zakim, move to next agendum
13:04:14 <Zakim> agendum 3. "Closing on the requirements" taken up [from plh]
13:04:25 <plh> http://www.w3.org/wiki/TestInfra/goals
13:05:06 <shadi> plh: separated a little more but did not add examples yet
13:05:46 <shadi> q+
13:06:04 <shadi> plh: SAZ said that most accessibility tests may be manual
13:06:07 <MichaelC> q+
13:06:12 <shadi> ack me
13:06:16 <shadi> http://www.w3.org/wiki/Testing/accessibility
13:06:44 <francois> shadi: I added a page on accessibility testing where I tried to expand a little more on the different types of accessibility testing. Some tests can be automatically automated.
13:07:00 <francois> ... In particular, some ARIA test cases can be nicely automated, I think.
13:07:39 <shadi> zakim, mute me
13:07:39 <Zakim> Shadi should now be muted
13:08:17 <shadi> plh: semi-automatic seem to be a mix of self-describing and automatic
13:08:30 <shadi> ...may be similar to geo-location testing
13:08:46 <shadi> ...they would also require human intervention
13:09:05 <shadi> ...interesting case about how people would write their tests
13:09:58 <shadi> ...for some WAI tests the human provides the results whereas with the geo-location the result is provided automatically based on what a human does
13:10:01 <shadi> q+
13:10:16 <MichaelC_> MichaelC_ has joined #testing
13:10:18 <Zakim> +Michael_Cooper.a
13:10:18 <plh> ack Michael
13:11:14 <shadi> mc: had thoughts but did not add them to the wiki yet
13:11:24 <MichaelC_> Test cases separate from test files
13:11:26 <MichaelC_> Test cases supported by 0 or more test files
13:11:28 <MichaelC_> A given test file can be used by 0 or more test cases
13:11:30 <MichaelC_> Test cases and test files can be shared among WGs (implication of above requirements is that WGs could share test files but have different test cases)
13:11:33 <MichaelC_> Metadata not stored in test files
13:11:35 <MichaelC_> Provision for passing and failing test cases
13:11:37 <MichaelC_> Support for automated execution of test cases
13:11:39 <MichaelC_> Support for manual execution of test cases
13:11:41 <MichaelC_> Easy way to store test results
13:11:43 <MichaelC_> (possibly future) way to store and compare test results from different testers, and declare an "authoratative" result
13:11:46 <MichaelC_> Way to associate test cases with specific spec requirement
13:11:47 <MichaelC_> Way to perform tests against multiple user agents (defined broadly as UA on a platform, potentially with a given AT, plugin, or other support tool running)
13:11:51 <MichaelC_> (possibly future) separate analysis against different UA, different OS, different AT, different AAPI, etc.
13:11:53 <MichaelC_> Different test cases allowed for different UAs
13:11:55 <MichaelC_> Way for not-to-technical users to add tests
13:11:57 <MichaelC_> Way to add large amounts of test cases and/or test files at once, e.g., because auto-generated from spec
13:11:59 <MichaelC_> Way to isolate each spec feature tested in order to associate test cases with them
13:12:01 <MichaelC_> Metadata requirements: see TSDTF format
13:12:03 <MichaelC_> (possibly future) way for members of public to submit tests for consideration (would need a vetting process)
13:12:05 <MichaelC_> Way to associate test cases with 1 or more spec features (preference for unit tests associated with one feature, but aggregated tests may be needed)
13:12:08 <MichaelC_> Types of tests: parsing, DOM (or other memory model) effect, presentation, API exposure, reaction
13:12:55 <plh> ack shadi
13:12:57 <plh> q+
13:12:57 <shadi> ack me
13:14:21 <Zakim> -Michael_Cooper
13:14:36 <shadi> zakim, mute me
13:14:36 <Zakim> Shadi should now be muted
13:15:16 <shadi> saz: the break-down of semi-automated testing is useful to WAI too, for instance for user input for script testing
13:15:38 <shadi> ...might be more relevant for test authoring but need to keep that in mind
13:16:06 <shadi> plh: mc, didn't understand the separating test file from test case
13:16:43 <shadi> mc: let's say testing for the display of a particular attribute in a browser vs testing how it is communicated to the API
13:16:57 <shadi> ...two different tests on the same test file
13:17:52 <shadi> ack me
13:18:31 <shadi> plh: should make the test automatic, can't separate the automation
13:18:59 <shadi> mc: by separating test files from test cases, just an attribute in the metadata
13:19:12 <shadi> plh: adds a lot of complexity
13:22:15 <shadi> mc: if just looking at one or two groups' needs for now, then may do differently
13:22:38 <shadi> ..but on the long term, thought we need to support a W3C-wide framework
13:22:45 <shadi> http://www.openajax.org/member/wiki/Accessibility_Rules_Format_1.0
13:23:11 <shadi> plh: how will the test logic look like?
13:24:12 <shadi> mc: by adding the logic into the test files you are making each a small evaluation tool
13:24:34 <shadi> ...think it may be easier to create a single evaluation engine to process script logic
13:25:18 <shadi> plh: just need to write the assertions for testharness.js and then you are done
13:26:10 <shadi> saz: how did we end up with testharness.js?
13:26:27 <shadi> plh: needed a simple approach to carry out a test
13:27:00 <shadi> ...DOM WG and others had something like this years ago
13:27:21 <shadi> ...problem is that other libraries are pretty heavy
13:27:34 <plh> test(function() {assert_true(true)}, "assert_true with true")
13:28:43 <shadi> saz: are we starting with requirements then trying to find tools that match these needs or the other way around?
13:28:55 <shadi> plh: difficult enough to get people to write tests
13:30:21 <shadi> saz: isn't testharness.js a type of framework in itself, and don't people need to write logic anyway?
13:30:41 <shadi> mc: also hard to port to other frameworks in the future
13:30:48 <shadi> plh: not really
13:31:41 <shadi> mc: haven't looked at testharness.js specifically but from experience test metadata does interfere with test cases
13:31:51 <shadi> ...so at least these need to be separated
13:32:31 <shadi> plh: have examples from web performance group and can tell you that separating the logic makes it extremely difficult
13:32:50 <shadi> ...on the other hand, sometimes logic within the test case can get in the way
13:33:13 <shadi> ...like checking the DOM where writing logic would actually change the DOM
13:33:14 <francois> q+
13:33:34 <shadi> ...need to support different methods, do not want to constrain people
13:33:44 <shadi> q+ to ask about different methods
13:35:00 <shadi> zakim, mute me
13:35:00 <Zakim> Shadi should now be muted
13:35:45 <shadi> ack me
13:35:46 <Zakim> shadi, you wanted to ask about different methods
13:35:53 <plh> ack plh
13:35:54 <shadi> q+
13:35:55 <plh> ack francois
13:36:15 <shadi> fd: could have different ways of authoring tests
13:36:58 <shadi> ...could provide a tool that will convert declarative tests into testharness.js format
13:37:07 <shadi> ...not easy to do the other way around
13:38:25 <shadi> ...testharness.js would be the smallest common
13:38:45 <shadi> mc: yes, it is possible but not sure what we are supposed to be doing
13:39:37 <shadi> fd: even if you do it on your own, it would become part of the overall tool
13:39:54 <MichaelC> -> http://www.w3.org/WAI/PF/wiki/ARIA_Test_Plan ARIA Draft test plna
13:40:23 <MichaelC> s/plna/plan/
13:40:45 <shadi> mc: some ARIA tests may fall out of scope of testharness.js
13:40:58 <shadi> fd: this would fall back into the scope of requirements
13:41:40 <shadi> mc: would need to write some tests and bring back the requirements
13:41:50 <shadi> plh: or at least one test
13:42:47 <plh> ack shadi
13:42:48 <shadi> ack me
13:42:58 <shadi> http://www.openajax.org/member/wiki/Accessibility_Rules_Format_1.0
13:44:51 <MichaelC> side note, the work done by OpenAjax on ARIA testing is something we expect to incorporate into the ARIA test plan, this work is done by PFWG members anyways
13:46:08 <shadi> saz: so we have the requirement to support both the declarative and procedural approach, and a converter from declarative to procedural
13:46:52 <MichaelC> q+
13:46:58 <shadi> plh: agree with requirement but have to be careful of scope
13:48:16 <plh> ack MichaelC
13:48:24 <shadi> saz: was thinking of unicorn or some other framework
13:48:40 <shadi> plh: unicorn is more to merge reports
13:48:43 <MichaelC> http://www.w3.org/WAI/GL/WCAG20-TECHS/PDF1#PDF1-tests
13:51:09 <shadi> mc: example linked above, need to support many ways of carring out a test
13:51:20 <shadi> plh: this is what the requirements page needs to cover
13:51:44 <shadi> ...also need to specify what our requirements for the declarative approach would be
13:51:53 <shadi> ...can quickly get out of hand
13:52:13 <shadi> zakim, shut up
13:52:13 <Zakim> I don't understand 'shut up', shadi
13:59:45 <shadi> saz: suggest we need to refine the requirements a little bit more, especially to add the differentiation between declarative and procedural approach
14:00:06 <MikeSmith> many of the requirements we have already added on the wiki page do not necessarily have any level of consensus either
14:00:11 <shadi> mc: was not sure if should put requirements directly in the wiki or if we need more discussion
14:00:14 <plh> zakim, move to next agendum
14:00:14 <Zakim> agendum 4. "Testing Interest Group charter (Plh and Mike)" taken up [from plh]
14:00:17 <shadi> plh: need to put them in now
14:00:31 <shadi> zakim, mute me
14:00:33 <Zakim> Shadi should now be muted
14:01:11 <shadi> ms: have a draft for the IG, very sketchy at this point
14:01:22 <MikeSmith> http://www.w3.org/2011/05/testing-ig-charter.html
14:01:52 <shadi> ms: appreciate comments and feedback
14:02:34 <shadi> ...need to have well-bounded group
14:02:37 <shadi> q+
14:03:21 <MikeSmith> q?
14:03:33 <shadi> ack me
14:05:04 <shadi> saz: concern about the limited scope, more limited than the scope of the Vision TF intended
14:05:26 <shadi> plh: WAI-ARIA and XQuery missing
14:05:54 <shadi> ms: often have the issue of scope-creep so need to keep the scope bounded
14:06:30 <Zakim> -Michael_Cooper.a
14:07:08 <francois> shadi: think mentioning types of tests might be more useful than listing relevant technologies.
14:08:07 <francois> q+
14:09:44 <plh> ack francois
14:10:39 <shadi> fd: think good idea to have an IG, think the list of specifications is good for a start
14:12:07 <MikeSmith> I personally have never written up a charter like this without listing specific technologies that the group's work is intended to be related to; I think in terms of the technologies first
14:12:12 <MikeSmith> q?
14:12:44 <shadi> saz: agree with an IG as well, but currently the scope would exclude WCAG
14:13:56 <MikeSmith> we don't typically test server-side behavior; we test for what responses/information the server sends back to the client/browser
14:15:54 <shadi> plh: it would be exponentially harder to expand the scope
14:18:06 <shadi> saz: should look back at the requirements and see what overlap we have
14:18:08 <Zakim> -Mike
14:18:09 <Zakim> -Plh
14:18:09 <Zakim> -francois
14:18:17 <shadi> ...but need to address WCAG and WAI-ARIA
14:18:22 <Zakim> -Shadi
14:18:24 <Zakim> Team_(testing)13:00Z has ended
14:18:26 <Zakim> Attendees were Plh, Shadi, francois, Michael_Cooper, Mike
14:19:50 <shadi> RRSAgent, make minutes
14:19:50 <RRSAgent> I have made the request to generate http://www.w3.org/2011/04/04-testing-minutes.html shadi
14:20:06 <shadi> RRSAgent, make logs world
14:20:08 <shadi> RRSAgent, make minutes
14:20:08 <RRSAgent> I have made the request to generate http://www.w3.org/2011/04/04-testing-minutes.html shadi
14:33:30 <MikeSmith> hmm, the Web Notifications API is another case where the behavior cannot be tested within a browser
14:33:57 <MikeSmith> because the behavior is that the browser causes some platform-level notification to be generated
14:34:05 <MikeSmith> generated outside the browser
14:34:28 <plh> good point
14:35:20 <plh> s/WAI-ARIA and XQuery missing/WAI-ARIA is missing. But I wouldn't want to see XQuery listed/
14:35:33 <RRSAgent> I have made the request to generate http://www.w3.org/2011/04/04-testing-minutes.html plh
14:35:43 <plh> Meeting: Testing
14:35:47 <MikeSmith> the only thing we can test within the browser is whether it actually fires the "show" event
14:35:51 <plh> Chair: plh
14:36:09 <MikeSmith> oh, and whether it actually creates a Notification object
14:36:34 <MikeSmith> RRSAgent, make minutes
14:36:34 <RRSAgent> I have made the request to generate http://www.w3.org/2011/04/04-testing-minutes.html MikeSmith
14:36:58 <plh> I think that we want to say that the framework has to run within a browser
14:37:27 <plh> now, if someone wants to use that framework for XQuery for example, I won't prevent them
14:38:14 <plh> (but I'll think they'd be better off with a different framework)
14:44:30 <MikeSmith> yeah
14:44:42 <MikeSmith> hey, is there are test suite for Element Traversal?
14:44:57 <MikeSmith> ah yes, there is
14:52:32 <plh> yep, and it would be nice in the future to convert it
14:58:55 <shadi> RRSAgent, bye
14:58:55 <RRSAgent> I see no action items