See also: IRC log
mike: I took material written
more than one year and half ago when we had some meeting and
put together some requirements.
... Linked from that page are some requirements from other
groups.
... Goal is to get all the requirements in one place so that we
know what we want to do
plh: I had a call on Thursday
with accessibility people. We should break down the page into
"what the harness should do", "how to submit the tests",
etc.
... Not all the WG will submit tests the same way
... First priority: what the harness should do.
<MikeSmith> http://www.w3.org/wiki/Testing
<MikeSmith> http://www.w3.org/wiki/Testing/Requirements
mike: doug set up another
page
... because he did not like the name. I don't really care about the
name
plh: I don't care either way. Let's take a decision. Do you care?
[attendees agree to flip a coin]
plh: ok, please proceed under
/Testing, then, Mike.
... could you start classifying the topics in the page?
mike: yes, easy enough to do, happy to do this this week.
<plh> ACTION: Mike to classify the requirements in the wiki [recorded in http://www.w3.org/2011/02/28-testing-minutes.html#action01]
<plh> Mike?
<plh> can't hear you?
<MikeSmith> the fan on my laptop has quit, so my laptop is overheating now and then, to the point where it hits 85 degrees or something and then shuts itself down
<MikeSmith> I have a new laptop on the way but I won't get until for at least one more week
<plh> can you rejoin the call or should we continue on irc?
mike: let me paste in the URL for annotations
<MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/
mike: here's the system that
Philipp Taylor's set up for canvas.
... not scalable to the whole spec as it does regular
expression checks specific to this section.
... what we could do is to scope the tests with the IDs of a
specific section and then we can have it scale for
finer-grained addressing.
... If it works out for HTML5, it should work for everything
else, since there's hardly anything bigger than the HTML5
spec.
<MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html
<MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/annotated-spec/canvas.html#testrefs.initial.colour
<MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/initial.colour.html
mike: you'll see annotations in the result specifications, attached to testable assertions.
plh: I thought this was hard to replicate on a larger specification?
mike: not really. The problem is
if we don't have any way to scope a test case to a specific
section of the spec. If we do, then this thing can scale.
... the point is that you have at the beginning of the test
case the id of the section in the spec, and then the regular
expression would apply to this section only.
... If we do it that way, it would scale.
plh: but the specification is a moving target!
mike: when the prose changes, we need to come up with an error message that says "I cannot find this assertion anymore".
plh: so that's what you're working on?
mike: yes.
<MikeSmith> http://test.w3.org/html/tests/submission/PhilipTaylor/tools/canvas/tests2dtext.yaml
francois: in short, I'd introduce a level of indirection with a mapping table between testable assertions, extracted by a regular expression, and an ID, that tests can then reference.
mike: it seems doable and is not
far away from what I've been contemplating do.
... I need to take a closer look at that.
... I should be able to work on that later this week.
plh: in terms of timeline, I'd
like us to be done with requirements by the end of March.
... I'm hoping that Shadi and Michael are going to be able to
show us what they need in the coming week.
... Once done with requirements, April to June should be spent
implementing the bouzin.
... We may need some things running in May, for instance for
Michael.
... At the minimum, we should try not to change the format of
the tests.
... Because of the parsing tests, we'll need some way to have
tests completely separated from metadata about the tests.
francois: will we need to setup some hybrid system? I'm all for keeping things separated, but some people may think otherwise.
plh: Right. About parsing tests, we may need tests that are exactly 1 byte long, and we cannot include metadata in there.
mike: yes, I think we already discussed and agreed about the need to have a manifest file that describes the tests.
plh: what I would suggest for this metadata information is to look at what the CSS working group did, because they are pretty advanced there.
mike: ok.
plh: After June, once we are done
with setting up the framework, we should focus on integrating
tests in there, such as HTML parsing tests.
... tests done outside of W3C, for instance.
mike: getting the parsing tests in is definitely an important goal.
plh: yes, if we manage to get
parsing tests in, the rest should be relatively easy to
integrate.
... Some tests may apply to places where JavaScript is not
available and may be tricky, but the rest should be fine.
... On the framework front, we need to identify all the
frameworks that are out there. I believe that the Wiki page
already has such a list.
... Someone needs to look at what Mozilla is doing and
understand how Mozilla is testing their implementations.
... They are probably the most advanced ones out there.
francois: happy to have a look
<MikeSmith> Ms2ger
<scribe> ACTION: francois to see how Mozilla is testing Firefox [recorded in http://www.w3.org/2011/02/28-testing-minutes.html#action02]
mike: you may get in touch with Ms2ger, as he is a key contributor to testing efforts.
plh: I think we should still have a direct look at it. Francois, would be good to spend time and see e.g. the test format, how they test their mobile implementation, etc.
<MikeSmith> Mark Finkle
mike: the guy to get in touch with to have pointers to the right things is Mark Finkle.
plh: is there an IRC channel for testing?
<MikeSmith> irc.mozilla.org, ask on #developers
mike: don't know, but you should be able to find the info on Mozilla's main developer irc channel.
plh: don't know how long it would take, probably two weeks. I think that's valuable. We need it before end of March.
francois: ok, will do asap.
plh: anything else on the testing
framework?
... I'll ping Michael and Shadi to make sure they get their
requirements right.
mike: next meeting?
plh: how about this day and time?
[should work]