IRC log of htmlt on 2010-07-27

Timestamps are in UTC.

15:10:58 [RRSAgent]

RRSAgent has joined #htmlt

15:10:58 [RRSAgent]

logging to http://www.w3.org/2010/07/27-htmlt-irc

15:11:03 [krisk]

OK

15:11:06 [krisk]

let's start then

15:11:14 [plh]

Chair: Kris

15:11:18 [plh]

scribeNick: plh

15:11:52 [plh]

agenda: http://lists.w3.org/Archives/Public/public-html-testsuite/2010Jul/0009.html

15:12:02 [plh]

Topic: bugs in approved tests

15:12:04 [plh]

Kris: None

15:12:19 [plh]

Topic: Review Current Tests Posted To List For Approval

15:12:22 [krisk]

Lets move on to #2

15:12:35 [plh]

Philip Taylor's canvas tests

15:12:49 [krisk]

So we have Philip taylor's first batch of tests (fallback, type, size)

15:13:18 [krisk]

I look at the first set and they seem fine except the type.delete test

15:13:41 [plh]

http://test.w3.org/html/tests/submission/PhilipTaylor/canvas/type.delete.html

15:14:34 [krisk]

So the spirit of the WebIDL is to ensure the DOM objects are like JS Objects

15:15:14 [plh]

kris: it seems that we're all wrong

15:15:18 [plh]

... and the test is wrong

15:15:19 [jgraham]

Hmm? I'm not sure why that's the "spirit" of WebIDL

15:15:45 [krisk]

The test tests that you can't delete the HTMLCanvas object

15:15:58 [jgraham]

If browsers interoperably implement one behaviour I would expect that to be retained

15:15:58 [plh]

delete window.HTMLCanvasElement;

15:16:15 [krisk]

but delete is supported by all JS objects

15:16:44 [jgraham]

Any js object can be [[DontDelete]]

15:16:50 [jgraham]

or the ES5 equivalent

15:16:52 [plh]

http://dev.w3.org/2006/webapi/WebIDL/#delete

15:17:14 [jgraham]

I don't see a problem with this test

15:18:00 [plh]

kris: do we want to support for JS in 2006 or align with ES5?

15:18:24 [plh]

... ES5 is forcing you to delete the object

15:18:37 [jgraham]

the ES5 equivalent is [[Configurable]]=false

15:18:37 [plh]

... there is no [[DontDelete]] option

15:19:14 [jgraham]

See 8.12.7

15:19:17 [jgraham]

Of ES5

15:19:20 [plh]

pointer?

15:19:58 [jgraham]

The delete operator calls that with Throws=false (in non-strict mode)

15:20:33 [jgraham]

http://www.ecmascript.org/docs/tc39-2009-043.pdf

15:20:43 [plh]

thanks

15:20:44 [krisk]

see page 40

15:22:01 [jgraham]

Yes 8.6.2 explains what [[Configurable]] means

15:22:19 [plh]

so, on delete window.HTMLCanvasElement

15:22:25 [plh]

one need to throw a TypeError exception

15:22:31 [plh]

if [[Configurable]] isn't true

15:22:32 [jgraham]

No

15:22:40 [jgraham]

Only if Throw is true

15:22:47 [plh]

oh yes

15:23:01 [jgraham]

(which is isn't unless you are in strict mode)

15:23:10 [jgraham]

(which the test case isn't)

15:23:47 [krisk]

so I think we should align to ecma5

15:24:08 [plh]

does the 2d spec needs to say anything about [[configurable]] ?

15:24:20 [jgraham]

WebIDL needs to be updated to ES5

15:24:42 [jgraham]

But it is clear how [[DontDelete]] maps in this case

15:25:12 [plh]

I gree

15:25:14 [plh]

agree

15:26:51 [krisk]

OK

15:28:26 [plh]

so, I would expect the canvas 2d spec or html5 to say that the property cannot be deleted

15:28:37 [krisk]

so then the test should be [[configurable]] == false

15:29:41 [plh]

there is probably a general statement in thml5 indicating that you can't delete any of the properties

15:29:41 [krisk]

so then we just need to check that the HTML5 spec states this so that it's clear

15:29:54 [jgraham]

I assume that WebIDL says that interface objects in generat cannot be deleted so that every spec doesn't need to say it for every interface

15:29:59 [jgraham]

*general

15:30:29 [plh]

oh yes it does

15:30:32 [plh]

in section 4.5

15:30:40 [plh]

If a host object implements an interface, then for each attribute defined on the interface, there MUST be a corresponding property on the host object:

15:30:48 [plh]

# The name of the property is the identifier of the attribute.

15:30:48 [plh]

# If the attribute is declared readonly, the property has attributes { DontDelete, ReadOnly }. Otherwise, the property has attributes { DontDelete }.

15:31:19 [plh]

so ew're fine

15:31:21 [plh]

we're

15:31:51 [krisk]

ok then we should assume that at some date the webidl will change and be updated to use configurable and not dontdelete

15:32:05 [krisk]

Any other feedback on the first set of tests?

15:33:11 [plh]

Resolution: fallback, type and size canvas tests are approved

15:34:06 [krisk]

I can move them into the approved folder

15:34:07 [plh]

Opera's and Microsoft's getElementsByClassName tests

15:34:51 [krisk]

I'll keep the same structure

15:35:20 [krisk]

Seems like these should be changed so they can fit into the harness

15:35:31 [jgraham]

Anne wrote the Opera tests. I think he plans to convert them to the new test harness and submit them

15:35:45 [krisk]

I can do the same for the MSFT tests

15:36:24 [krisk]

great

15:36:44 [jgraham]

(if Anne doesn't do it soon, I will do it; it doesn't look like much work)

15:37:03 [krisk]

Now in theory once this is done they will look like http://test.w3.org/html/tests/submission/Opera/resources/apisample.htm

15:38:01 [krisk]

So we just need to combine this with the test runner http://test.w3.org/html/tests/harness/harness.htm

15:38:24 [krisk]

Though since these don't need manual verification they should run automatically

15:38:52 [jgraham]

It should be easy to combine the harness with a funner. It provides hooks to get callbacks when the tests are complete

15:38:56 [jgraham]

*runner

15:39:19 [jgraham]

It's the same way the output is implemented

15:39:36 [krisk]

yep

15:40:02 [krisk]

we can approve them once they get moved into the proper harness

15:40:15 [krisk]

jgraham do you agree?

15:40:19 [jgraham]

Yeah

15:40:34 [plh]

Topic: Discuss test runner/harness

15:40:44 [plh]

Discuss test input and test results xml formats

15:41:58 [krisk]

currently the test runner outputs plain text

15:42:21 [krisk]

I'd like to convert this to xml, or JSON (jgraham's feedback)

15:42:53 [krisk]

jgraham do you really want JSON or can you live with xml?

15:43:07 [jgraham]

I can live with XML but I would prefer not to

15:43:20 [jgraham]

I think for key-value pairs JSON is a better format

15:43:44 [jgraham]

That's essentially exactly what we have here

15:43:57 [jgraham]

It is also easier to work with from code

15:45:02 [krisk]

I think either would work just fine, though xml would seem to be easier to validate that the results don't have a type/error

15:45:33 [krisk]

though let's just choose JSON and move on

15:45:35 [plh]

I don't think this is going to be an issue

15:45:57 [plh]

if you don't give me correct results, I'll simply complain :)

15:46:21 [jgraham]

Yeah, if we need a "validator" it should be a few lines of custom code

15:46:36 [jgraham]

in python or javascript or whatever

15:46:45 [plh]

as long as the results go through my transformation steps, I'll be happy

15:47:14 [krisk]

OK let's talk about the data - to make sure we all agree

15:47:29 [plh]

so, I need a clear result for each test

15:47:36 [plh]

either pass, or fail

15:47:38 [krisk]

the first section of data should contain UA info

15:47:41 [plh]

is there an other state?

15:47:46 [plh]

like not applicable?

15:47:57 [jgraham]

kris had "not implemeneted"

15:48:05 [jgraham]

I don't understand the use case though

15:48:12 [jgraham]

We don't really have optional features

15:48:17 [plh]

for the section, I need a simple string to identify the agent

15:48:25 [plh]

OS info might be good

15:48:43 [jgraham]

That is typically included in the UA string, I think

15:48:44 [krisk]

userAgent, browsername, date, submitter, Testsuitename

15:49:27 [krisk]

so then we want userAgent, browsername, date, submitter, Testsuitename, and OS

15:49:34 [plh]

Testsuitename?

15:49:35 [jgraham]

why submitter?

15:49:36 [krisk]

TestSuiteName == HTML5

15:49:42 [jgraham]

Why testsuitename?

15:49:56 [jgraham]

If it is a constant

15:50:25 [jgraham]

As Philip pointed out, we probably care about the version of the testsuite that was used

15:50:25 [krisk]

that is true it should not change - so then userAgent, browsername, date

15:50:42 [plh]

what's the different between userAgent and browsername?

15:50:43 [krisk]

do we agree

15:51:06 [krisk]

some browsers for compat place stuff in the UA to make them look like other browsers

15:51:49 [jgraham]

As long as there is a 1:1 mapping between the complete UA and the browser, it is not a problem

15:52:09 [jgraham]

s/UA/UA string/

15:52:48 [krisk]

We should keep both - since browser name makes more sense that a UA

15:53:10 [jgraham]

Well I don't mind if it is easy to extract using js

15:53:10 [krisk]

Lets move on to the next part - data for each test

15:54:52 [krisk]

so we have URI,Featurename, specref, result

15:55:08 [plh]

do we have specref?

15:55:14 [plh]

or even feature name?

15:55:18 [krisk]

I think we just need URI, Featurename, result

15:55:49 [jgraham]

I think URI:result is fine

15:55:59 [krisk]

I think we should have feature name - since parts of HTML5 are much farther along and will be interoperable long before other parts

15:56:03 [jgraham]

But maybe URI:[result, message] is better

15:56:15 [jgraham]

We can get the feature names from the URI

15:56:18 [plh]

agreed. if we have featurename available somewhere, I can find it based on the URI anyway

15:56:22 [jgraham]

No need to submit it each time

15:57:13 [krisk]

That will work, we can pul lhe feature name from the URI

15:57:29 [plh]

was thinking that I might need to do an extra classification anyway to make the results more comprehensive

15:57:32 [krisk]

s/pul/pull/

15:58:39 [krisk]

I tihnk using the URI will work e.g http://test.w3.org/html/tests/approved/audio/audio_001.htm == audio test

15:59:01 [plh]

so, we need pass, fail? do we want to differentiate between the fail states? (timeout, not implemented, crashed, come back later, etc.)?

15:59:03 [krisk]

using a simple split on '/'

16:00:02 [jgraham]

I think timeout is like fail for conformance tests

16:00:23 [krisk]

the next part is the results part pass, fail, not implemented

16:00:30 [jgraham]

It might be different for browser vendors so it is worth distinguishing at the harness level

16:00:42 [krisk]

I don't want timeout

16:00:43 [jgraham]

but not in the presentation of the results

16:01:59 [krisk]

The reason for not implemented is that we are going to have some features not implemented

16:02:08 [krisk]

for example SVG in HTML -

16:02:43 [plh]

so, you'll expect the harness to recognize if the feature was implemented or not?

16:03:00 [jgraham]

I don't really like the idea of not implemented

16:03:15 [krisk]

if a browser doesn't implement SVG in HTML that's differnet than one that does a very poor job at implementing SVG in HTML

16:03:22 [jgraham]

It seems likely to be inconsistent between different parts of the testsuite

16:03:29 [jgraham]

I disagree that is different

16:03:55 [jgraham]

From the users point of view it is feature that you cannot rely on in that browser

16:05:00 [plh]

the boundary between not implemented and failure isn't clear to me

16:05:15 [krisk]

so then we should just stick to pass/fail?

16:05:18 [plh]

if a function returns a wrong result, is it a failure or not implemented?

16:05:32 [jgraham]

I think we stick to pass/fail

16:05:39 [jgraham]

Anything else is too complex

16:05:47 [krisk]

Well the browser vendor that submits the result can just say it's not implemented

16:06:15 [krisk]

surely they would know if a feature is implemented or not

16:06:47 [krisk]

I was thinking of the case for css2.1 run-ins

16:06:56 [plh]

yes, but one might say it's not implemented, and an other one might say it's a failure

16:07:03 [jgraham]

My imagined process here doesn't involve people editing the reaults files by hand

16:07:34 [plh]

I agree with sticking to fail/pass seems a lot easier

16:07:46 [krisk]

even if you don't implement css2.1 run-ins you end up passing about 1/2 the cases

16:07:57 [krisk]

which is a little misleading

16:08:31 [plh]

oh, that's different indeed. so you want to be able to say "not implemented" on tests that you actually pass>?

16:08:44 [jgraham]

If we end up with a data-presentation problem, we can compile a seperate list of features that UAs claim to not implement

16:08:55 [jgraham]

It doesn't need to be tied to the test harness

16:09:23 [jgraham]

and indeed I don't see how it can be (without manually editing the results files) in the case you mentioned

16:09:34 [krisk]

Ok then lets just have the tests results be pass/fail and then we can list features that are not implemented when the data doesn't make sense

16:09:52 [krisk]

like the css2.1 run-in case

16:09:53 [plh]

I don't mind have a "blacklist" on the side

16:09:57 [plh]

having

16:10:35 [plh]

ie, if you really insist, sure I'll pretend you don't pass the tests :)

16:10:51 [jgraham]

heh

16:11:23 [krisk]

Ok then we have for the test results part pass, fail

16:11:55 [krisk]

and we can add a not implemented list to the report when it becomes a problem

16:12:02 [plh]

ok

16:12:24 [krisk]

The test input part - I assume this should be easy

16:12:39 [krisk]

just uri and type

16:12:59 [plh]

type?

16:13:04 [krisk]

type == manual verification || trype == automatic verification

16:13:32 [jgraham]

That seems to be a property of the testcase

16:13:44 [jgraham]

So we don;t need to submit it with the results

16:14:01 [krisk]

This is the input not the output

16:14:02 [plh]

I think Kris is saying the same thing

16:14:11 [jgraham]

OK

16:14:52 [krisk]

At first I didn't think this woud be needed at all

16:15:02 [krisk]

and we could just use plain text

16:16:25 [krisk]

though when you run the tests you don't want to run a manual test, wait for a automatic test to run, run a few more manual tests, wait for another automatic test to run, etc...

16:16:45 [krisk]

it should just let you run that manual tests as one bunch

16:17:18 [plh]

makes sense to me

16:17:53 [krisk]

any othe agenda items?

16:17:58 [krisk]

I have 2

16:18:38 [krisk]

I

16:19:04 [krisk]

I'll send out another set of canvas tests to review (philip's tests)

16:19:17 [krisk]

since we want to keep making progress on these tests

16:19:27 [krisk]

also I'll be submitting some tests for http://dev.w3.org/html5/spec/Overview.html#dom-document-getselection

16:20:02 [krisk]

so for the next meeting we should be able to approve some more tests and have further progress on the harness and test runner

16:20:17 [krisk]

any others?

16:20:24 [plh]

ok, souds good

16:20:27 [plh]

sounds

16:20:36 [RRSAgent]

I have made the request to generate http://www.w3.org/2010/07/27-htmlt-minutes.html plh

16:21:04 [Zakim]

-Plh

16:21:05 [Zakim]

-krisk

16:21:05 [Zakim]

HTML_WG(HTMLT)11:00AM has ended

16:21:06 [Zakim]

Attendees were Plh, krisk