trackbot-ng, start telcon

<trackbot> Sorry... I don't know anything about this channel

<trackbot> If you want to associate this channel with an existing Tracker, please say 'trackbot, associate this channel with #channel' (where #channel is the name of default channel for the group)

<krisk> OK

<krisk> sounds fine from my phone

<krisk> shall we get started?

<scribe> scribeNick: plh

Check for any bugs on approved tests (currently zero)

Kris: no bugs

Review Current Tests Posted To List For Approval

<krisk> That is good since we approved alot of tests and moved them into the approved folder

plh: what about the canvas tests?

<krisk> 1 canvas test will be updated and then moved (from MSFT)

kris: there is one canvas from Microsoft?

s/Microsoft\?/Microsoft/

Kris: I expect to have feedback for 50 tests from Philipp Taylor

plh: I thought that opera was using those tests already

Geoffrey: does it actually make sense to review them all and not just the script to generate them

<jgraham> We are using the tests, but we didn't do documented test-by-test review, so we can't trivially approve them

<krisk> we noticed a few issues -

Geoffrey: the only ones that we looked into are the ones we failed

<krisk> for example http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/context.unrecognised.nullsuffix.html

<krisk> This test is testing jscript null handling for example...

Geoffrey: why would it be? it's testing exactly what null does in this context

<jgraham> That test seems fine to me

<gsnedders> s/Goeffrey/Geoffrey/g

<krisk> so does the HTML5 spec state how the "2d\0" string passed in should be treated in the DOM?

<krisk> I recall that 2 browsers treated 2d\0 as a c string == '2d'

I get "Aborted with exception: Component returned failure code: 0x80070057 (NS_ERROR_ILLEGAL_VALUE) " in firefox

<krisk> though another set of browsers translated it to 2d0

<krisk> and another left 2d\0 asw 2d\0

<jgraham> It states that you should only return a context for the name "2d". "2d\0" isn't "2d"

<krisk> so should it not be just '2d'?

<jgraham> Whether null in strings should cause it to throw is kinda a WebIDL question

<jgraham> I think

I tried the test in opera, firefox, and chrome and they all fail...

<jgraham> The test is testing that you *don't* get a context object back in this case

<gsnedders> WebKit gives undefined and not null

<jgraham> Chromium passes for me

<jgraham> Opera is just buggy in this case

<krisk> We don't need to debug this right now - but

<krisk> this is the type of critical review we want to do for the tests

<krisk> which takes time and why I don't want to just approve them all right away

plh: so all browsers don't implement this properly. What do we do? change the browsers or change the spec?
... if the spec doesn't match reality...

<jgraham> (chromium passes for me)

<gsnedders> WebKit trunk passes

<jgraham> (the opera behaviour is insane)

Goeffrey: the sensitive thing to do is what is in the spec

<jgraham> (the firefox behaviour is defensible)

<krisk> This is a good bug (test)

<gsnedders> s/sensitie/sensible/

<jgraham> (conclusion: what's in the spec is no worse than anything else)

<krisk> but really not many web devs are going to type in '2d\0' and depend upon the error handling

<krisk> It has not happend 'yet' - but it is possible to approve a test that all current browsers fail

<krisk> So one ask may be to have browser vendors focus on tests that they fail - since this is an interoperability bug

<krisk> So I'll send a list out to the list this week, ones that are good and one that may need updates.

conclusion: spec is correct and current browser behaviors aren't

<krisk> Then the group can discuss on the list and move forward

geoffrey: we'll go with Kris's recommendations and may file bug later

<krisk> sounds like a way to move forward

Discuss the Test Runner/Harness

<krisk> Since it's actually painful to run the tests

<krisk> I submitted a way to run the tests http://test.w3.org/html/tests/harness/harness.htm

<krisk> It loads tests from a text file - list of uri's

Kris: load list of uris, build an array, keep track of passes or fails.
... then generates results

Geoffrey: is there anyway to automate this?

<krisk> I'll still need to do two workitems

Kris: still looking at it

<krisk> one to auto run tests that don't need manual verification (yuk)

<krisk> The list comes for all the tests in the approved folder

<jgraham> We seem to have a depressing number of tests that need manual verification already

<krisk> I agree

http://test.w3.org/html/tests/harness/approvedtests.txt

<krisk> The uris come from all the tests in the approved folder

<krisk> So the other workitem is to turn the approvedtests.txt to xml and same with the results

<jgraham> Maybe we should stop approving tests that require manual work unless it can be demonstrated that the feature cannot be tested in a better way

<jgraham> For many of these javascript tests or reftests could work

<krisk> I think for jscript tests that will be fine feedback

but we don't have a framework to run reftests

<jgraham> I propose we beg borrow or steal from CSSWG if we need reftests

<jgraham> I mean borrow their framework

<jgraham> if we need reftests for HTML features

<jgraham> But hopefully we will need fewer than them since we should have fewer visual things to test

<krisk> I don't think we'll end up with too many ref test for HTML

but do their framework generate test results?

<jgraham> I'm not sure. I need to talk to Tab about it

<krisk> I like the idea of reftests - though making it work across browsers, devices, operating systems, dpi, fonts is tricky

<krisk> For example border == thick

<krisk> how many pixels is thick?

more than thin :)

<krisk> true - but then how about inset?

<krisk> who's inset is correct and wrong?

<krisk> though browsers today are different - but correct

<gsnedders> Where in HTML5 is this defined? :)

<krisk> It's not - which is the point

<jgraham> HTML5 says nothing about the rendering in those cases though

<jgraham> So you can't write a test for it at all

<krisk> yep

<jgraham> (the rendering section is all non-normative)

<krisk> So then it would be easy to store the results in Hg

<gsnedders> (like, rendering it with display: none is conforming)

<krisk> Which with XHR you can look for results...0.xml, 1.xml, etc...

<krisk> I can get the UA and store this into the XML file

<krisk> Though I think we can add a 'Browser Name' value and have this added to the result

<krisk> Do you want to submit a draft of test results? with a fake browser?

<krisk> The data pivot - would be feature, test, results

<krisk> I'd keep the format simple html+css

<krisk> the results are not a test :)

<krisk> any thing else to discuss?

<krisk> see you on the list

<krisk> or on irc

[adjourned]

This is scribe.perl Revision: 1.135  of Date: 2009/03/02 03:52:20  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

FAILED: s/Microsoft\?/Microsoft/
Succeeded: s/it does make/does it actually make sense/
Succeeded: s/Goeffrey/Geoffrey/g
Succeeded: s/Goeffrey/Geoffrey/
FAILED: s/Goeffrey/Geoffrey/g
FAILED: s/sensitie/sensible/
Found ScribeNick: plh
Inferring Scribes: plh

WARNING: No "Present: ... " found!
Possibly Present: AAPI F2F Goeffrey Microsoft SW_RIF Team_ VB_VBWG Vision W3C_Global WAI_PFWG XML_ aaaa active conclusion geoffrey gsnedders jgraham kris krisk plh scribeNick trackbot
You can indicate people for the Present list like this:
        <dbooth> Present: dbooth jonathan mary
        <dbooth> Present+ amy

Agenda: http://lists.w3.org/Archives/Public/public-html-testsuite/2010Jul/0001.html

WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth

Got date from IRC log name: 13 Jul 2010
Guessing minutes URL: http://www.w3.org/2010/07/13-htmlt-minutes.html
People with action items: