Agenda http://lists.w3.org/Archives/Public/public-html-testsuite/2010Nov/0060.html

Item #1 Bugs on approved tests

I started a thread on the list http://lists.w3.org/Archives/Public/public-html-testsuite/2010Nov/0062.html

<plh> http://www.w3.org/Bugs/Public/buglist.cgi?product=HTML%20WG&component=testsuite&resolution=---

<jgraham> Oh, telecon

This item we can't close on today - looks like their is not agreement in the group on how a page should behave when the page doesn't have the HTML5 doctype

We have a few other bugs as well

http://www.w3.org/Bugs/Public/show_bug.cgi?id=11321

<jgraham> who is "there"?

<plh> Kris will follow up with Anne on the list

<jgraham> Oh

<plh> we have also http://lists.w3.org/Archives/Public/public-html-testsuite/2010Nov/0038.html

<jgraham> OK. The spec is well defined on that point

<jgraham> But yes, we should have this discussion on-list

Seems like a bug in the spec

<plh> http://www.w3.org/Bugs/Public/show_bug.cgi?id=11236

<jgraham> Hmm?

alot of content on the web is old and would not be compatible with HTML5

<jgraham> This is not a significant problem for us at least

It's also not just a 'browser' vendor choice

alot of content authoring systems generate html

and they don't generate html5 content

<jgraham> If you think the spec should be changed that should go through the normal HTMLWG Process

IMHO - let's not discuss this here....

<jgraham> But per current spec, the test is fine afaict

The test still is 'approved'

<jgraham> Bug 11321 is something for me. Maybe we could have a different component in bugzilla for bugs in the framework rather than bugs in tests?

if we have alot of test harness bugs sure, but that seems like overkill given the number of bugs

<jgraham> Could be

The general feedback from TPAC was that the harness was a bit terse to use

<jgraham> "terse"?

cumbersome

I think we should be open to changes - in theory more folks will write tests

<jgraham> That could be true. There is the possibility of adding a simpler API, but there are tradeoffs with roubustness

james do you think you could incorporate this feedback?

<jgraham> I plan to look at it

My worry is that we continue to work on the harness and not tests

<jgraham> And encouraging people to write tests is a goal for sure

Given the turnout at TPAC (apple, boeing, Opera, Mozilla, Microsoft) was alot more than normal

<jgraham> From my point of view, the tests are something that get developed alongside other work at Opera. The harness needs special attention because it doesn't fall naturally out of implementation work

it seems appropriate to start a requirement ask with a end date

It would also let everyone have time from various particpants to send in requirements and provide feedback

<jgraham> We could do that for sure

lets do that - target a date say in a month

<jgraham> OK

Note that I have an action item from TPAC that is dependent up changes to the harness (test type javascript, manual, ref-test)

Next bug http://www.w3.org/Bugs/Public/show_bug.cgi?id=11236

seems like a legit bug

<jgraham> That action item seems like it would be affected by my proposal for per-directory metadata

<plh> http://test.w3.org/html/tests/approved/video/video_003.htm

<plh> btw, we're working in changing the domain name for test.w3.org

<plh> to avoid XSS issues

<jgraham> Yeah, the bug is legitimate. It would be good if bugzilla had a "confirmed" state

<plh> "HTML5 Media Elements: 'application/octet-stream' is not a type the UA cannot render."

well a UA may say 'maybe'

seems like we should remove the test and have it fixed

<plh> +1

let's move on

<jgraham> The relevat UA requirement seems to be """ canPlayType(type) method must return the empty string if type is a type that the user agent knows it cannot render or is the type "application/octet-stream" """ fwiw

<jgraham> *relevant

agree

Item #2 Approval for the next 25 phillip taylor tests

They seem fine to me

Though at TPAC concern was raise about the progress of getting them all approved

How about we pick a date to be all 'reviewed'

<jgraham> Yes, in general it is not clear to me how the approval process will scale with large numbers of tests

<jgraham> So you propose that we pick a date and say "and objections by this date or we use the bug system"?

<jgraham> +approve

that would be the plan

<jgraham> That wfm

what is wfm?

<jgraham> "Works for me"

<jgraham> Apologies for being obscure

ms2ger gave some feedback as well

though the attahed pdf faults?

<jgraham> That also wfm :)

<jgraham> The PDF was a derivation of the correct answers for the tests based on teh Porter Duff operators and the test input

<jgraham> I believe Philip fixed all the issues

He did

<jgraham> (basically the issues were incorrect rounding in the expected output)

<jgraham> So the PDF is interesting but not essential if you are having difficulties opening it

assume it has some value, else it would not be attached

<jgraham> Well, yes the value is understanding where the corrections came from

anyhow - lets pick end of the year as a target date to approval all of phillip taylors tests

I'll send a email to the list - after that date updates can be done via bugs

Item #3 TPAC Follow Up

before then plh said we have another set of test results from Midori

We should have a single person from each vendor handle their test results

I can take care of Microsoft

<plh> we'll need to figure what to do with all those test results

I think the question should be...

<plh> at least, the wave of results went down

Do they represent the browser vendor?

If they are member of the w3c it should be easy to check

<plh> so, are you saying we should accept results from browser vendors?

<jgraham> To be honest I am not sure who is benefiting from getting any results at all at this stage

<plh> we could, but this hasn't been clear to me in the past

<jgraham> Apart from advertising to possible contributers

Though not sure if the Midori or the maxthon

<jgraham> We *know* the testsuite is woefully incomplete

are members of the w3c - though they build a browser and want to have their results displayed

that is why it has all the warnings and disclaimers

though it's not like someone can't go run the tests themselves

<jgraham> Of course they can

this happens all the time - for example a number of sites post SVG results

the reason it's good if it's on the w3c is that then everyone gets to have a voice

<jgraham> So, just to play devil's advocate, given all the health warnings we have to put up at the moment, why not just pull the results from public display for now

<jgraham> Then there would be no concern about people misusing them

<jgraham> What would the drawbacks be?

it won't help the spec move forward

<jgraham> In what way? The spec is not blocked on a testsuite at the moment

It's good to see that a browser I never heard of participate (Midori)

<jgraham> It is good if they are contributing tests. If they are just running the (incomplete) testsuite then I see no benefits

the results are all opt-in

<jgraham> I don't understand what you mean?

If browser vendor doesn't want the results shown then they don't have to have them posted

<jgraham> Oh I see. But there is a considerable difference between no-one posting their results and one or two vendors opting out

<jgraham> Until the testsuite reaches a meaningful size

Well it seems that on about a month or so we will have more than 1000 tests

which is bigger in size than the SVG test suite and about 1/10 the size of the CSS2.1 test suite

<jgraham> I expect we will end up with 10,000-100,000 if we do a good job

<jgraham> Hopefully in the upper end of that range

It's not complete but will have more html5 tests than other resources that I am aware of

<jgraham> So we are talking about a few percent of the total number of tests

<jgraham> That we need

<jgraham> (btw I consider it a priority to get some mechanism for mapping tests to the spec so we can see where we have coverage and where we do not)

<jgraham> (Philip's canvas tests do this nicely and I think it might be possible to adapt that mechanism)

Intresitng at TPAC this was discussed - use the classname

we just need to make sure that ian doesn't change classnames

<jgraham> I don't think that is fine-grained enough

<jgraham> Or, at least, I'm not sure

<jgraham> and lots of the ids and classes in the spec are auto-generated

well I think we need something that is concrete

<jgraham> by the pre-processor and don't appear in the original document

<jgraham> Sure, my suggestion is we reuse the mechanism Philip used

<jgraham> Which is based on a regexp match of the text

<plh> using an annotated spec?

The reason we started with 'features' was that even chapters numbers were changing a year ago

I think we should choose the classname for a while

<jgraham> Also has the advantage that when something changes the regexp is unlikely to still match

<jgraham> So you get to see which tests probably became invalid

<jgraham> And you can link to an actual conformance requirement rather than a general area of the spec

I'll take a peek

<jgraham> and the code is mostly written :)

moving on...

plh can you make sure that 'new' heads are not possible for Hg?

<plh> hu... I don'

<plh> t know

at tpac this came up briefly - when people were learning how to submit tests

Jonas should be able to help

<plh> I can try to follow up with Jonas on this

I'll send an email to Jonas you and the systems folks - sound good?

<jgraham> http://mercurial.selenic.com/wiki/TipsAndTricks#Prevent_a_push_that_would_create_multiple_heads

<plh> kris, ok

jonas was also concerned about the logging and said mozilla has a tool to track changes better

<jgraham> (just need a pre-commit hook)

<plh> James, thank you for the pointer, I'll follow up with the webmaster

note I move backwards in the list #3.G -> #3.F....

<plh> kris, re tracking tool, yes I already sent an email to our team about this

<jgraham> What was the actual concern about logging?

<plh> might take a while to get deployed

that is fine

<plh> james, the names that appear at http://dvcs.w3.org/hg/html/

<plh> are set by the user, and not the server

and could be changed and don't map to the w3c account

<plh> so, we don't know for sure who did the push

<jgraham> Oh. OK. I don't have a problem with more logging of course but it seems unlikely to be an issue in practice

<plh> well, folks seemed concerned about this during TPAC

<plh> and Jonas pointed me to http://hg.mozilla.org/hgcustom/pushlog/

<jgraham> Like I said, I don't mind solving it if the oppertunity cost is low

given the number of people pushing right now it's not a problem

<plh> I won't push our system team to ct on this one fast

<plh> php and w3ctest is more important imho

though if we get more participants it can be a problem (at least some people that have worked on larger project have ran into this problem)

<jgraham> Yeah

agree

<jgraham> PHP and w3test are essential

<plh> agreed

<jgraham> (or s/PHP/Python :)

so how is the XSS and PHP stuff coming along?

<plh> w3test should be done pretty fast

I recall we agreed to have the name be test-w3.org

<plh> we bought the domain name already

for sure it should be registered and have the w3c be the owner

what is the name?

<plh> w3c-test.org

will we still have the test.w3c-test.org and test2.w3c-test.org?

<plh> yes

james not sure how much you talked to anne post TPAC

<plh> for php, I know that our system folks looked into that and are coming up with a solution

though it seems that we needs a few specific PHP pages

<plh> dunno about the ETA at this point

the group didn't want to create alot of PHP pages

<jgraham> I hve talked to Anne a bit

in fact it may not be php - could be any generic cgi script

<plh> http://www.w3.org/2008/webapps/wiki/Testing_Requirements

so basically we don't want to enable anyone to just push a new php file to the server

<jgraham> Yes, there are reasonable security concerns

the php file would be a manually added to the web servers unlike the rest of the content

any other agenda items?

<plh> I'm guessing they'll create /scripts or something like that

<jgraham> It would be useful if it was still VCS backed somehow

<plh> and move the php/python/whatever into /scripts as they are approving them

<plh> james, we can still maintain them under /html

It could be maybe a seperate project that only get proped to the web server upon demand

<jgraham> Right. That seems like it should work

<plh> it's just that we'll only execute them when they're place under /scripts

<plh> placed

<jgraham> They are managed in hg but the server only executes them in a special directory which they only reach after review for security problems

<plh> yep

<plh> as long as the review process isn't long, we should be fine

let's adjurn

This is scribe.perl Revision: 1.135  of Date: 2009/03/02 03:52:20  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/their/there/
Succeeded: s/how/who/
No ScribeNick specified.  Guessing ScribeNick: krisk
Inferring Scribes: krisk

WARNING: No "Topic:" lines found.


WARNING: No "Present: ... " found!
Possibly Present: jgraham plh
You can indicate people for the Present list like this:
        <dbooth> Present: dbooth jonathan mary
        <dbooth> Present+ amy


WARNING: No meeting title found!
You should specify the meeting title like this:
<dbooth> Meeting: Weekly Baking Club Meeting


WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth

Got date from IRC log name: 16 Nov 2010
Guessing minutes URL: http://www.w3.org/2010/11/16-htmlt-minutes.html
People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: No "Topic: ..." lines found!  
Resulting HTML may have an empty (invalid) <ol>...</ol>.

Explanation: "Topic: ..." lines are used to indicate the start of 
new discussion topics or agenda items, such as:
<dbooth> Topic: Review of Amy's report