SV_MEETING_TITLE -- 27 Jul 2010

<krisk> OK

<krisk> let's start then

<scribe> scribeNick: plh

bugs in approved tests

Kris: None

Review Current Tests Posted To List For Approval

<krisk> Lets move on to #2

Philip Taylor's canvas tests

<krisk> So we have Philip taylor's first batch of tests (fallback, type, size)

<krisk> I look at the first set and they seem fine except the type.delete test

http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/type.delete.html

<krisk> So the spirit of the WebIDL is to ensure the DOM objects are like JS Objects

kris: it seems that we're all wrong
... and the test is wrong

<jgraham> Hmm? I'm not sure why that's the "spirit" of WebIDL

<krisk> The test tests that you can't delete the HTMLCanvas object

<jgraham> If browsers interoperably implement one behaviour I would expect that to be retained

delete window.HTMLCanvasElement;

<krisk> but delete is supported by all JS objects

<jgraham> Any js object can be [[DontDelete]]

<jgraham> or the ES5 equivalent

http://dev.w3.org/2006/webapi/WebIDL/#delete

<jgraham> I don't see a problem with this test

kris: do we want to support for JS in 2006 or align with ES5?
... ES5 is forcing you to delete the object

<jgraham> the ES5 equivalent is [[Configurable]]=false

kris: there is no [[DontDelete]] option

<jgraham> See 8.12.7

<jgraham> Of ES5

pointer?

<jgraham> The delete operator calls that with Throws=false (in non-strict mode)

<jgraham> http://www.ecmascript.org/docs/tc39-2009-043.pdf

thanks

<krisk> see page 40

<jgraham> Yes 8.6.2 explains what [[Configurable]] means

so, on delete window.HTMLCanvasElement

one need to throw a TypeError exception

if [[Configurable]] isn't true

<jgraham> No

<jgraham> Only if Throw is true

oh yes

<jgraham> (which is isn't unless you are in strict mode)

<jgraham> (which the test case isn't)

<krisk> so I think we should align to ecma5

does the 2d spec needs to say anything about [[configurable]] ?

<jgraham> WebIDL needs to be updated to ES5

<jgraham> But it is clear how [[DontDelete]] maps in this case

I gree

agree

<krisk> OK

so, I would expect the canvas 2d spec or html5 to say that the property cannot be deleted

<krisk> so then the test should be [[configurable]] == false

there is probably a general statement in thml5 indicating that you can't delete any of the properties

<krisk> so then we just need to check that the HTML5 spec states this so that it's clear

<jgraham> I assume that WebIDL says that interface objects in generat cannot be deleted so that every spec doesn't need to say it for every interface

<jgraham> *general

oh yes it does

in section 4.5

If a host object implements an interface, then for each attribute defined on the interface, there MUST be a corresponding property on the host object:

# The name of the property is the identifier of the attribute.

# If the attribute is declared readonly, the property has attributes { DontDelete, ReadOnly }. Otherwise, the property has attributes { DontDelete }.

so ew're fine

we're

<krisk> ok then we should assume that at some date the webidl will change and be updated to use configurable and not dontdelete

<krisk> Any other feedback on the first set of tests?

Resolution: fallback, type and size canvas tests are approved

<krisk> I can move them into the approved folder

Opera's and Microsoft's getElementsByClassName tests

<krisk> I'll keep the same structure

<krisk> Seems like these should be changed so they can fit into the harness

<jgraham> Anne wrote the Opera tests. I think he plans to convert them to the new test harness and submit them

<krisk> I can do the same for the MSFT tests

<krisk> great

<jgraham> (if Anne doesn't do it soon, I will do it; it doesn't look like much work)

<krisk> Now in theory once this is done they will look like http://test.w3.org/html/tests/submission/Opera/resources/apisample.htm

<krisk> So we just need to combine this with the test runner http://test.w3.org/html/tests/harness/harness.htm

<krisk> Though since these don't need manual verification they should run automatically

<jgraham> It should be easy to combine the harness with a funner. It provides hooks to get callbacks when the tests are complete

<jgraham> *runner

<jgraham> It's the same way the output is implemented

<krisk> yep

<krisk> we can approve them once they get moved into the proper harness

<krisk> jgraham do you agree?

<jgraham> Yeah

Discuss test runner/harness

Discuss test input and test results xml formats

<krisk> currently the test runner outputs plain text

<krisk> I'd like to convert this to xml, or JSON (jgraham's feedback)

<krisk> jgraham do you really want JSON or can you live with xml?

<jgraham> I can live with XML but I would prefer not to

<jgraham> I think for key-value pairs JSON is a better format

<jgraham> That's essentially exactly what we have here

<jgraham> It is also easier to work with from code

<krisk> I think either would work just fine, though xml would seem to be easier to validate that the results don't have a type/error

<krisk> though let's just choose JSON and move on

I don't think this is going to be an issue

if you don't give me correct results, I'll simply complain :)

<jgraham> Yeah, if we need a "validator" it should be a few lines of custom code

<jgraham> in python or javascript or whatever

as long as the results go through my transformation steps, I'll be happy

<krisk> OK let's talk about the data - to make sure we all agree

so, I need a clear result for each test

either pass, or fail

<krisk> the first section of data should contain UA info

is there an other state?

like not applicable?

<jgraham> kris had "not implemeneted"

<jgraham> I don't understand the use case though

<jgraham> We don't really have optional features

for the section, I need a simple string to identify the agent

OS info might be good

<jgraham> That is typically included in the UA string, I think

<krisk> userAgent, browsername, date, submitter, Testsuitename

<krisk> so then we want userAgent, browsername, date, submitter, Testsuitename, and OS

Testsuitename?

<jgraham> why submitter?

<krisk> TestSuiteName == HTML5

<jgraham> Why testsuitename?

<jgraham> If it is a constant

<jgraham> As Philip pointed out, we probably care about the version of the testsuite that was used

<krisk> that is true it should not change - so then userAgent, browsername, date

what's the different between userAgent and browsername?

<krisk> do we agree

<krisk> some browsers for compat place stuff in the UA to make them look like other browsers

<jgraham> As long as there is a 1:1 mapping between the complete UA string and the browser, it is not a problem

<krisk> We should keep both - since browser name makes more sense that a UA

<jgraham> Well I don't mind if it is easy to extract using js

<krisk> Lets move on to the next part - data for each test

<krisk> so we have URI,Featurename, specref, result

do we have specref?

or even feature name?

<krisk> I think we just need URI, Featurename, result

<jgraham> I think URI:result is fine

<krisk> I think we should have feature name - since parts of HTML5 are much farther along and will be interoperable long before other parts

<jgraham> But maybe URI:[result, message] is better

<jgraham> We can get the feature names from the URI

agreed. if we have featurename available somewhere, I can find it based on the URI anyway

<jgraham> No need to submit it each time

<krisk> That will work, we can pull lhe feature name from the URI

was thinking that I might need to do an extra classification anyway to make the results more comprehensive

<krisk> I tihnk using the URI will work e.g http://test.w3.org/html/tests/approved/audio/audio_001.htm == audio test

so, we need pass, fail? do we want to differentiate between the fail states? (timeout, not implemented, crashed, come back later, etc.)?

<krisk> using a simple split on '/'

<jgraham> I think timeout is like fail for conformance tests

<krisk> the next part is the results part pass, fail, not implemented

<jgraham> It might be different for browser vendors so it is worth distinguishing at the harness level

<krisk> I don't want timeout

<jgraham> but not in the presentation of the results

<krisk> The reason for not implemented is that we are going to have some features not implemented

<krisk> for example SVG in HTML -

so, you'll expect the harness to recognize if the feature was implemented or not?

<jgraham> I don't really like the idea of not implemented

<krisk> if a browser doesn't implement SVG in HTML that's differnet than one that does a very poor job at implementing SVG in HTML

<jgraham> It seems likely to be inconsistent between different parts of the testsuite

<jgraham> I disagree that is different

<jgraham> From the users point of view it is feature that you cannot rely on in that browser

the boundary between not implemented and failure isn't clear to me

<krisk> so then we should just stick to pass/fail?

if a function returns a wrong result, is it a failure or not implemented?

<jgraham> I think we stick to pass/fail

<jgraham> Anything else is too complex

<krisk> Well the browser vendor that submits the result can just say it's not implemented

<krisk> surely they would know if a feature is implemented or not

<krisk> I was thinking of the case for css2.1 run-ins

yes, but one might say it's not implemented, and an other one might say it's a failure

<jgraham> My imagined process here doesn't involve people editing the reaults files by hand

I agree with sticking to fail/pass seems a lot easier

<krisk> even if you don't implement css2.1 run-ins you end up passing about 1/2 the cases

<krisk> which is a little misleading

oh, that's different indeed. so you want to be able to say "not implemented" on tests that you actually pass>?

<jgraham> If we end up with a data-presentation problem, we can compile a seperate list of features that UAs claim to not implement

<jgraham> It doesn't need to be tied to the test harness

<jgraham> and indeed I don't see how it can be (without manually editing the results files) in the case you mentioned

<krisk> Ok then lets just have the tests results be pass/fail and then we can list features that are not implemented when the data doesn't make sense

<krisk> like the css2.1 run-in case

I don't mind have a "blacklist" on the side

having

ie, if you really insist, sure I'll pretend you don't pass the tests :)

<jgraham> heh

<krisk> Ok then we have for the test results part pass, fail

<krisk> and we can add a not implemented list to the report when it becomes a problem

<krisk> The test input part - I assume this should be easy

<krisk> just uri and type

type?

<krisk> type == manual verification || trype == automatic verification

<jgraham> That seems to be a property of the testcase

<jgraham> So we don;t need to submit it with the results

<krisk> This is the input not the output

I think Kris is saying the same thing

<jgraham> OK

<krisk> At first I didn't think this woud be needed at all

<krisk> and we could just use plain text

<krisk> though when you run the tests you don't want to run a manual test, wait for a automatic test to run, run a few more manual tests, wait for another automatic test to run, etc...

<krisk> it should just let you run that manual tests as one bunch

makes sense to me

<krisk> any othe agenda items?

<krisk> I have 2

<krisk> I

<krisk> I'll send out another set of canvas tests to review (philip's tests)

<krisk> since we want to keep making progress on these tests

<krisk> also I'll be submitting some tests for http://dev.w3.org/html5/spec/Overview.html#dom-document-getselection

<krisk> so for the next meeting we should be able to approve some more tests and have further progress on the harness and test runner

<krisk> any others?

ok, souds good

sounds

- DRAFT -

SV_MEETING_TITLE

27 Jul 2010

Attendees

Contents

bugs in approved tests

Review Current Tests Posted To List For Approval

Discuss test runner/harness

Summary of Action Items

Scribe.perl diagnostic output