See also: IRC log
<krisk> hello
<krisk> trackbot, status
<trackbot> This channel is not configured
<krisk> trackbot, prepare telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> trackbot #htmlt
<krisk> trackbot, prepare telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<krisk> trackbot, prepare telcon
<trackbot> Sorry... I don't know anything about this channel
<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)
<plh> trackbot-ng, start telcon
<trackbot> Date: 04 May 2010
<krisk> should we start the meeting?
<krisk> anyone want to scribe?
<plh> --> http://lists.w3.org/Archives/Public/public-html-testsuite/2010May/0000.html agenda
<scribe> scribenick: adrianba
<scribe> scribe: adrianba
krisk: currently zero, but we can
have some bugs once we make changes to the harness
... to support james graham's async feedback
gsnedders: don't want to start with one harness and to keep hacking another on top and one on top of that
krisk: do you have an idea what that should look like?
gsnedders: not really
<gsnedders> jgraham: ping
krisk: in the windows tests i think i did something along those lines
plh: i wonder how independent we
can be from the harness
... so that the tests can be written in a way that makes it
easy to switch to a new harness
... if we get another harness that is better
... we don't have to re-write everything
<jgraham> FWIW I think the async stuff is the stuff that makes it hard to switch harnesses too
krisk: is this about consistency so that we can depend upon certain things being in the test
gsnedders: the main thing is
about having the assert functions that check for what the test
is testing
... assertEquals, etc.
krisk: if you are too specific
about what has to be written then you lose flexibility
... but it has to say pass/fail in a consistent manner
<jgraham> Given a set of assertFunctions and two harnesses that implement them, it should be quite easy to swicth synchronous tests between the two harnesses
krisk: there will be some tests where assertEquals isn't going to work for you
gsnedders: most harnesses have
these kind of functions
... the main thing that varies is how you say what tests should
be run
krisk: nothing would prevent you from changing later - for example i changed from jsunit by replacing a lot of the include files
<jgraham> but with an async test, you are typically deep into the harness-specific behaviour to determine what you need to write, when it gets called, how you express failure, and so on
krisk: in that regard, if a
browser vendor didn't like how the tests ran they could make
updates to run in their own way
... the basic contract between the test and the harness is all
that has to be defined
... even if you had an async test you could poll and know the
test was starting to run
... and through some setTimeouts continue to check if the work
has been done
... and at some point the harness can say the work didn't
complete and move on
<jgraham> That sounds like a harness-specific design
krisk: but in the end it needs to look for a pass or test completed result in a consistent way
<jgraham> You can't just replace a test designed to work in that way with one designed to work in a different way
plh: right now, kris you provided one?
krisk: i wouldn't say a harness,
more like a loader
... could be extended to do more
plh: do we need to do more? i
suggest we don't need to
... once we have plenty of tests we can focus on that if
necessary
gsnedders: once we have plenty of
tests it will be harder to change
... for example if someone gives lots of tests for a section of
the spec
<jgraham> FWIW my point of view is that we should write a harness and provide hooks for people to get the results out for whatever purpose they have
<jgraham> e.g. customs regression-detection systems
<jgraham> *custom
plh: the reason i'm not in a
hurry to develop another test harness is to avoid duplicating
work
... would rather delay discussion on test harness until
june
... see where we are then
krisk: regardless of the harness we should define the consisten way of seeing a pass
plh: yes, we should define assertEquals, etc
krisk: i mean even like tag id
gnsedders: why do we need to define that?
krisk: a number of tests have different ways of saying the pass
<jgraham> one assert != one passing test
krisk: at the moment mostly it is reading what is on the screen
gsnedders: i sent e-mail an hour ago about getting this programatically get this out
<krisk> this post ?
<krisk> http://lists.w3.org/Archives/Public/public-html-testsuite/2010May/0005.html
<gsnedders> yeah
krisk: when you call
assertEquals, the implementation of that will go call something
else
... so you take the logging out of the test
... then you can change assertEquals to do whatever you
want
<jgraham> So one approach is that you have a callback funtion that you call when the test has a result
gsnedders: this is what most of the JS libraries do
krisk: otherwise you have a consistent place - the harness knows to go look there
gsnedders: the problem is that
you have to keep polling the DOM
... but a function means you can send it back as soon as you
get the call
... makes it easy to send back to regression tracking
systems
<jgraham> To be clear, a single assertEquals should not be considered a single result. One result may be the combination of several assertions
krisk: should the harness be in control? what happens if the test blows up?
gsnedders: the harness should
catch all exceptions
... for critical issues in a browser you won't be able to
access the DOM either
... i don't think there's a difference
krisk: it sounds like we would want to change the DOM tests JS implementation of assertEquals to fit into a harness at some point
gsnedders: yeah
krisk: sounds like we have agreement
<krisk> wiki -> http://www.w3.org/html/wg/wiki/Testing
<plh> http://www.w3.org/html/wg/wiki/Main_Page
krisk: we've covered this too
krisk: is anyone thinking about submitting more tests?
plh: one of the reasons I was
asking this - do we have anything that says which areas we are
looking for tests for
... for example, progress events for video aren't implemented
in some browsers - if we ask for video tests we might get tests
for things that aren't stable
... i remember we said that we wanted tests for window because
that was more stable
krisk: video would be one
plh: yes, as long as you don't go
too far
... for example, testing the js play function is fine
... i might start working on video tests, probably not in
may
<jgraham> Writing video tests is one way to ensure that we support async tets well :)
<jgraham> *tests
krisk: some people at microsoft have some tests that we'd like to submit soon too
plh: i don't want to duplicate
effort
... if there are people who will submit video tests soonish
gsnedders: i will when i have
time
... hopefully this month
plh: video will be important - <video> gets the most buzz
krisk: do we want to use <video> to test the async part of our requirements?
plh: we could
krisk: we should
gsnedders: okay
<plh> http://lists.w3.org/Archives/Public/public-html-testsuite/2010Apr/0017.html
plh: sent a report last week
about getting things moved to the W3C site
... after discussing with system folks they would like to avoid
putting thousands of tests into main site
... running into issues with css2.1
... will provide access to a web site by end of may
... will have to different hosts, two different domains in
fact
... so that we can do cross-origin tests if needed
... also looking at whether test harness runs on this server or
if we need it on a third server
<plh> tests.w3.org and tests.www.org
<krisk> for example....
<krisk> tests.w3.org/html5/
krisk: i'd make one primary - i think one ending in w3.org is important
plh: these sites will also host
the css2.1
... would prefer not to run php or whatever on the same
site
... prefer to run harness somewhere else
... but it's not possible to differentiate which is the harness
at the moment
... still considering how to approach the problem
... but if harness only uses js then not an issue
... only an issue if we want to run server-side php or
python
gsnedders: we will want some tests eventually for, e.g. slow loading images
plh: at some point we will need
something on the server-side but not in the immediate
future
... so we should run with this idea by end of may and this is
the main priority right now
... expect more information next week
... not going to get deployment right away because of that but
will have a better solution by the end of the month
krisk
<plh> htmt5tests.w3.org
krisk: i think this is a good
starting point
... keep it simple in the short term seems good to me
... anything else?
<krisk> When I push to HG now i get a permission denied error
<krisk> I hit the error on 2 differnet systems
<krisk> specific error
<krisk> permission denied: .hg/store/lock
gsnedders: for the video tests, we're going to have to use some video format
plh: we can use two
gsnedders: yes, we can use two
<plh> http://www.w3.org/2008/12/dfxp-testsuite/web-framework/START.html
plh: this is similar to what
happened in the timed text working group
... there is a video and it is available in multiple
formats
... it's not a big deal
... we could use the same videos - they were done for the
purpose of testing the timing
... but it doesn't contain sound
gsnedders: we have test videos too
plh: yes, if people want to
submit videos as long as they have the rights we can use
them
... and we can transform into whatever format we need to
krisk: sounds like we have
agreement that we'll have to support different formats
... and plh will help with converting
plh: we will run into some
problems for example testing the src attribute
... but we could have some JS hacks testing which the browser
supports
krisk: any other business?
... okay, meeting adjourned
This is scribe.perl Revision: 1.135 of Date: 2009/03/02 03:52:20 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/how to/getting this/ Found ScribeNick: adrianba Found Scribe: adrianba Inferring ScribeNick: adrianba Default Present: Plh, +04613479aaaa, gsnedders, adrianba, krisk Present: Plh +04613479aaaa gsnedders adrianba krisk Agenda: http://lists.w3.org/Archives/Public/public-html-testsuite/2010May/0000.html Found Date: 04 May 2010 Guessing minutes URL: http://www.w3.org/2010/05/04-htmlt-minutes.html People with action items:[End of scribe.perl diagnostic output]