14:03:54 <RRSAgent> RRSAgent has joined #testing
14:03:54 <RRSAgent> logging to http://www.w3.org/2011/02/28-testing-irc
14:03:56 <plh> agenda+ annotations
14:04:01 <francois> RRSAgent, make logs public
14:04:02 <plh> agemda+ timeline
14:04:10 <plh> agenda+ timeline
14:04:16 <francois> scribe: francois
14:04:23 <plh> zakim, move to next agendum
14:04:23 <Zakim> agendum 1. "wiki" taken up [from plh]
14:04:26 <francois> zakim, mute me
14:04:26 <Zakim> francois should now be muted
14:05:01 <plh> --> http://www.w3.org/wiki/TestInfra/goals Test Infrastructure
14:05:19 <francois> mike: I took material written more than one year and half ago when we had some meeting and put together some requirements
14:05:35 <francois> ... linked from that page are some requirements from other groups
14:06:00 <francois> ... goal is to get all the requirements in one place so that we know what we want to do
14:06:45 <francois> plh: I had a call on Thursday with accessibility people. We should break down the page into "what the harness should do", "how to submit the tests", etc.
14:06:56 <francois> ... Not all the WG will submit tests the same way
14:07:07 <francois> ... First priority: what the harness should do.
14:07:15 <MikeSmith> http://www.w3.org/wiki/Testing
14:07:20 <MikeSmith> http://www.w3.org/wiki/Testing/Requirements
14:07:22 <francois> mike: doug set up another page
14:07:49 <francois> ... because he did not like the name. I don't care about the name
14:08:10 <francois> ack me
14:08:36 <francois> plh: I don't care either way. Let's take a decision. Do you care?
14:08:48 <francois> [ok to flip a coin]
14:09:02 <francois> plh: ok, please proceed under /Testing, then, Mike.
14:09:33 <francois> plh: could you start classifying the topics in the page?
14:09:44 <francois> mike: yes, easy enough to do, happy to do this this week.
14:10:02 <plh> zakim, move to next agendum
14:10:02 <Zakim> agendum 2. "annotations" taken up [from plh]
14:10:18 <plh> ACTION: Mike to classify the requirements in the wiki
14:10:45 <plh> Mike?
14:10:48 <plh> can't hear you?
14:10:55 <Zakim> -??P2
14:12:43 <plh> agenda+ framework
14:12:51 <MikeSmith> MikeSmith has joined #testing
14:14:43 <MikeSmith> the fan on my laptop has quit, so my laptop is overheating now and then, to the point where it hits 85 degrees or something and then shuts itself down
14:15:09 <MikeSmith> I have a new laptop on the way but I won't get until for at least one more week
14:15:23 <plh> can you rejoin the call or should we continue on irc?
14:15:32 <Zakim> +??P2
14:16:04 <MikeSmith> Zakim, ??P2 is me
14:16:04 <Zakim> +MikeSmith; got it
14:16:12 <francois> mike: let me paste in the URL for annotations
14:16:16 <MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/
14:16:36 <francois> mike: here's the system that Philipp Taylor's set up for canvas.
14:16:58 <francois> ... not scalable to the whole spec as it does regular expression checks specific to this section.
14:17:39 <francois> ... what we could do is to scope the tests with the IDs of a specific section and then we can have it scale for finer-grained addressing.
14:18:01 <francois> ... If it works out for HTML5, it should work for everything else, since there's hardly anything bigger than the HTML5 spec.
14:18:09 <MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html
14:18:29 <MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html#testrefs.initial.colour
14:18:46 <MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/initial.colour.html
14:18:55 <francois> mike: you'll see annotations in the result specifications, attached to testable assertions.
14:19:21 <francois> plh: I thought this was hard to replicate on a larger specification?
14:19:56 <francois> mike: not really. The problem is if we don't have any way to scope a test case to a specific section of the spec. If we do, then this thing can scale.
14:21:28 <francois> mike: the point is that you have at the beginning of the test case the id of the section in the spec, and then the regular expression would apply to this section only.
14:21:36 <francois> ... If we do it that way, it would scale.
14:22:02 <francois> plh: but the specification is a moving target!
14:22:41 <francois> mike: when the prose changes, we need to come up with an error message that says "I cannot find this assertion anymore".
14:22:56 <francois> plh: so that's what you're working on?
14:22:58 <francois> mike: yes.
14:27:02 <MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/tests2dtext.yaml
14:27:36 <francois> francois: in short, I'd introduce a level of indirection with a mapping table between testable assertions, extracted by a regular expression, and an ID, that tests can then reference.
14:28:01 <francois> mike: it seems doable and is not far away from what I've been contemplating do.
14:28:12 <francois> ... I need to take a closer look at that.
14:28:23 <francois> ... I should be able to work on that later this week.
14:28:28 <plh> zakim, move to next agendum
14:28:28 <Zakim> agendum 3. "timeline" taken up [from plh]
14:28:29 <francois> plh: anything else on annotations?
14:28:51 <francois> plh: in terms of timeline, I'd like us to be done with requirements by the end of March.
14:29:13 <francois> ... I'm hoping that Shadi and Michael are going to be able to show us what they need in the coming week.
14:29:34 <francois> ... Once done with requirements, April and June should be spent implementing the bouzin.
14:29:47 <francois> s/April and June/April to June/
14:30:04 <francois> ... We may need some things running in May, for instance for Michael.
14:30:31 <francois> ... At the minimum, we should try not to change the format of the tests.
14:31:50 <francois> ... Because of the parsing tests, we'll need some way to have tests completely separated from metadata about the tests.
14:32:37 <francois> francois: will we need to setup some hybrid system? I'm all for keeping things separated, but some people may think otherwise.
14:33:08 <francois> plh: Right. About parsing tests, we may need tests that are exactly 1 byte long, and we cannot include metadata in there.
14:33:32 <francois> mike: yes, I think we already discussed and agreed about the need to have a manifest file that describes the tests.
14:33:58 <francois> plh: what I would suggest for this metadata information is to look at what the CSS working group did, because they are pretty advanced there.
14:34:01 <francois> mike: ok.
14:34:38 <francois> plh: After June, once we are done with setting up the framework, we should focus on integrating tests in there, such as HTML parsing tests.
14:34:48 <francois> ... tests done outside of W3C, for instance.
14:35:25 <francois> mike: getting the parsing tests in is definitely an important goal.
14:35:47 <francois> plh: yes, if we manage to get parsing tests in, the rest should be relatively easy to integrate.
14:36:11 <francois> ... Some tests may apply to places where JavaScript is not available and may be trick, but the rest should be fine.
14:36:18 <francois> s/trick/tricky/
14:36:48 <francois> ... On the framework front, we need to identify all the frameworks that are out there. I believe that the Wiki page already has such a list.
14:37:11 <francois> ... Someone needs to look at what Mozilla is doing and understand how Mozilla is testing their implementations.
14:37:22 <francois> ... They are probably the most advanced ones out there.
14:37:35 <francois> francois: happy to have a look
14:38:05 <MikeSmith> Ms2ger
14:38:18 <francois> ACTION: francois to see how Mozilla is testing Firefox
14:38:51 <francois> mike: you may get in touch with Ms2ger, as he is a key contributor to testing efforts.
14:39:34 <francois> plh: I think we should still have a direct look at it. Francois, would be good to spend time and see e.g. the test format, how they test their mobile implementation, etc.
14:39:45 <MikeSmith> Mark Finkle
14:40:07 <francois> mike: the guy to get in touch with to have pointers to the right things is Mark Finkle.
14:40:16 <francois> plh: is there an IRC channel for testing?
14:40:26 <MikeSmith> irc.mozilla.org, ask on #developers
14:40:42 <francois> mike: don't know, but you should be able to find the info on Mozilla's main developer irc channel.
14:41:30 <francois> plh: don't know how long it would take, probably two weeks. I think that's valuable. We need it before end of March.
14:42:00 <francois> francois: ok, will do asap.
14:42:14 <francois> plh: anything else on the testing framework?
14:42:35 <francois> ... I'll ping Michael and Shadi to make sure they get their requirements right.
14:42:41 <francois> mike: next meeting?
14:42:53 <francois> plh: how about this day and time?
14:43:34 <Zakim> -francois
14:43:37 <francois> [should work]
14:43:44 <francois> RRSAgent, draft minutes
14:43:44 <RRSAgent> I have made the request to generate http://www.w3.org/2011/02/28-testing-minutes.html francois
14:48:38 <francois> Chair: plh
14:48:46 <francois> Present: plh, francois, mike
14:49:40 <francois> Meeting: Testing team call
14:49:51 <francois> s/plh: anything else on annotations?//
14:49:55 <francois> RRSAgent, draft minutes
14:49:55 <RRSAgent> I have made the request to generate http://www.w3.org/2011/02/28-testing-minutes.html francois
14:50:38 <MichaelC> MichaelC has joined #testing
14:56:09 <Zakim> -Plh
14:56:10 <Zakim> Team_(testing)14:00Z has ended
14:56:12 <Zakim> Attendees were Plh, francois, MikeSmith
15:02:51 <francois> RRSAgent, bye
15:02:51 <RRSAgent> I see 2 open action items saved in http://www.w3.org/2011/02/28-testing-actions.rdf :
15:02:51 <RRSAgent> ACTION: Mike to classify the requirements in the wiki [1]
15:02:51 <RRSAgent>   recorded in http://www.w3.org/2011/02/28-testing-irc#T14-10-18
15:02:51 <RRSAgent> ACTION: francois to see how Mozilla is testing Firefox [2]
15:02:51 <RRSAgent>   recorded in http://www.w3.org/2011/02/28-testing-irc#T14-38-18