16:00:28 RRSAgent has joined #htmlt
16:00:28 logging to http://www.w3.org/2013/03/12-htmlt-irc
16:01:27 we can wait a few to see if Ms2Ger attends
16:09:37 OK lets get rolling!
16:10:57 First agenda item - change the meeting time to start one hour later so that it doesn't conflict with other task force meetings
16:11:41 I was asked if the testing task force meeting could start one hour later
16:12:28 Would this be a problem for folks? Especially people from Europe?
16:13:19 Effectively starting in April we would meet from 17:00->18:00 GMT
16:13:47 In Redmond Washington/Califormia the meeting would be at 9am
16:13:49 Well
16:13:56 On the east coast it would be noon
16:13:57 Thtat would be worse for me
16:14:02 But not impossibly bad
16:14:47 We could do it one hour earlier as well...
16:14:51 Redmond/Cali would be 9am standard, but 10am daylight saving time
16:15:51 We typically stay at the same time all year - except the few times during the year when europe/usa times don't leap forward back at the same time
16:16:26 Like right now - the meeting is the same time as it has been in europe but in the USA it's starting an hour later due to the time changes
16:17:19 Then in april the Europe springs ahead and then people in the USA have the meeting start at 8am like normal
16:18:09 IMHO this oddity only is a 2-3 weeks of the year and I'm flexible who meeting times actually change
16:18:39 jgraham - is the time change something that would be easier if we start this in a month?
16:19:29 Well my point is that 17:00 local time is easier than 18:00
16:19:37 So if we preserve that, it's better
16:19:46 But 18:00 isn't a disaster
16:20:34 others? tobie/robin?
16:21:36 Let's move on...happy to re-discuss later if needed...
16:22:59 Looks like a test the web forward event will be happing in April (not 100% confirmed but highly likey)
16:23:37 Part of this event I'd like to give focus to areas in the HTML5 spec that we need to have coverage
16:25:10 The last meeting we discussed about classifying all the sections in the spec
16:26:51 Which I *think* we had consensus - but looking at the threads on the list it seems like we didn't have complete consensus :)
16:27:13 IMHO I think with a few tweaks we can all agree that this would be a good idea
16:27:39 For example mdyck asked if we can use caniuse
16:28:48 Which has some data but other data on caniuse is a bit misleading since includes features not in the spec and is missing alot of sections in the spec
16:30:13 mdyck did you see my response on the list?
16:30:26 yup
16:31:44 the fact that caniuse deals with some things that aren't part of html5 spec or aren't normative, that's presumably not a problem,
16:32:28 OK sounds like we are in agreement that some data on caniuse can help classify sections in the HTML5 spec
16:32:34 because we'd need to map their features to html5 spec sections, and we can just map such features to not-a-section or some such
16:32:35 however
16:33:28 the lack of similar granularity is probably more of a problem
16:33:59 i.e., a caniuse feature might map to multiple spec sections, what do we do then?
16:34:13 We started the testing a few years ago specifically with 'features' rather than sections of the spec for two reasons
16:35:16 One browser vendors were shippping features and while browser vendors were shipping features they created tests to validate their implementation
16:35:28 Two the spec spec had a lot of churn...
16:36:24 Now right the spec has alot less churn and as the spec moves to REC we need push on spec coverage
16:36:53 I'm not conserned about how a feature on caniuse maps to multiple sections
16:37:29 Rather I'm more conserned about sections in the spec that have normative requirements and either poor interop or lack of implementations.
16:39:45 For example the 'the-input-byte-stream' section of the HTML5 spec is quite important but this is not listed on caniuse at all and we don't have a good set of tests in the suite
16:39:45 well, yes, but then the question was, where do we get data about poor interop / lack of impl? (and one potential answer was caniuse.com, but then you have to ask if caniuse's data can answer the original question)
16:41:54 Well Robins cool coverage report (http://w3c-test.org/html-testsuite/master/tools/coverage/) has listed 403 sections
16:42:25 okay, so the input-byte-stream example indicates that caniuse can't give us full info about lack of interop
16:43:22 I don't have a specific focus other than taking the 403 sections listed in the coverage report and getting this number small where possible.
16:43:56 Right, caniuse.com doesn't tell us everything
16:44:02 For example the 'introduction' section has no normative requirements - so now we have 402 possible sections that we need more tests for, etc..
16:44:21 It helps us a bit with some sections
16:44:36 Which is better than nothing (which, approximately, is the other option)
16:45:42 How about this...
16:45:54 Basically the problem is this:
16:46:06 We have a spec with inconsistent test coverage
16:46:13 correct
16:47:00 Tests are the most objective measure we have of interoperability
16:47:15 So for sections where we have good test coverage things are good
16:47:22 For the regions without test coverage we want a heuristic measure of interoperability to help us prioritise writing tests
16:47:33 yes!
16:47:49 Any such measure is necessarily incomplete
16:48:12 So basically we should scrape together as much partial data from different sources as we can get our hands on
16:48:47 I'm open to that...
16:50:09 As a first step how about this...
16:50:45 I'll send a list of sections (starting at the top of the spec) and classify them in the 5 ways suggested on the list
16:51:03 Which includes MikeSmith's suggestion
16:51:30 "has conformance requirements, has tests, known interop issues".
16:52:12 technically, that's a subset of class C.
16:52:48 I'll start with a summary of the 'classifications'...
16:53:13 so if you want disjoint classes, you would want to redefine C
16:53:25 I think there are some sections that we know are interop disasters even where we don't have lots of submitted tests. So just asking people to nominate parts of the spec where they thing interop is poor might be a helpful data source
16:53:57 Though in terms of progress classifying sections of the spec that have no normative requirements (aka introduction)...
16:54:16 and classifying sections in the spec that have normative requirements and no tests...
16:54:31 would be very valueable...
16:54:55 Right, but darobin's tool can help with that
16:54:55 robin's report already does that
16:55:09 We can change our mind with more data for the other classifications...
16:55:12 A) should be easy
16:55:47 C) is easy depending on what you mean by "tests"
16:56:04 Distinguishing B) and D) is hard
16:56:15 But really there are a bunch more subclasses
16:56:47 e.g. C1) Has conformance requirements, has tests, tests show interop issues
16:57:04 e.g. C2) Has conformance requirements, has tests, tests show no interop issues, interop issues exist
16:57:20 Open to other/additional classifications...
16:57:35 e.g. C3) Has conformance requirements, has tests, no known interop issues but test coverage too low
16:57:38 etc.
16:58:13 (don't use those verbaitam, my point is just that these are other points in the space of has tests/has interop issues not covered by the classifications given)
16:59:43 (Where did A, B, C, and D come from?)
17:00:12 (Feb 27 meeting)
17:00:33 (soirry, Feb 25)
17:00:57 see -> http://lists.w3.org/Archives/Public/public-html-testsuite/2013Mar/0003.html
17:01:00 A) no requirements
17:01:00 B) has conformance requirements, no tests, known interop issues (for example the-input-byte-stream)
17:01:04 C) has conformance requirements, has tests (canvas)
17:01:06 D) has conf reqs, no tests, no known interop issues
17:01:42 (gack, Feb 26: irc log is http://www.w3.org/2013/02/26-htmlt-irc )
17:02:00 I'll send this out to the list and we can hash out classifications a bit more..
17:02:17 Well the thing is, it's pretty arbitary
17:02:27 For each feature there is a 2D space
17:02:57 Something like interop problems against percentage of imeplemtation covered by tests
17:03:33 Assuming some mappoing of interop problems to a point in the space
17:04:16 It's going to be a start to the problem and not an end..
17:04:21 We want each section to move toward the lower right of the diagram (high interop, complete test coverage)
17:04:51 (hmm, my axes might be wrong there)
17:04:57 In theory over time the data will get better with either more tests, feedback, etc..
17:05:27 For most sections at the moment we don't even know where we are in the space
17:05:49 Knowning we don't know where we are is a sign of progress!
17:06:08 Maybe I should start sending out ascii art :)
17:06:28 OK I have to move on and the meeting is over...
17:06:36 I have one more thing to add
17:06:41 On a different topic
17:06:42 I look forward to some good discussion on the list
17:06:47 Yes
17:07:18 I set up a copy of the "opera critic" code review system on http://critic.hoppipolla.co.uk and connected it to the html-testsuite and testharness.js github repos
17:07:22 This is experimental
17:07:40 But when you make a pull request it will open a review request there
17:07:59 And imo, that is a much better UI for code review than the github one
17:08:08 So it's something we could experiment with
17:08:35 neat - maybe this can get moved to a w3c server?
17:08:48 Yeah, I would be super-happy for that to happen :)
17:08:57 At the moment it is still a work in progress
17:09:02 OK meeting adjourned!
17:09:12 RRSAgent, make logs public
17:09:12 I need to add a couple more features and get the code approved upstream
17:09:26 (the github integration I mean)
17:09:56 tobie you should note this if you have not already given your testing work in the w3c
17:10:05 rrsagent, generate minute
17:10:05 I'm logging. I don't understand 'generate minute', krisk. Try /msg RRSAgent help
17:10:10 rrsagent, generate minutes
17:10:10 I have made the request to generate http://www.w3.org/2013/03/12-htmlt-minutes.html krisk
17:10:28 krisk: note what?
17:10:35 tobie: About Critic
17:10:40 But I think you already know
17:11:00 jgraham and I don't agree about this at all.
17:11:14 We're planning on a sword fight at tpac
17:11:55 As a followup for darobin and Lachy?
17:12:53 you're already fighting darobin and lachy about this?
17:14:16 jgraham: I question the value of having a dedicated code review tool atm.
17:15:04 As it significantly steepens the learning curve.
17:15:11 tobie: Well I plan to use it for testharness.js reviews at least. And anything that lessens the considerable pain of code review seems worthwhile to me. This lessens the pain.
17:16:08 I'd like to understand what GitHub is missing that would make reviewing testharness.js diffs simpler.
17:17:44 Also code review is a process that involves two parties: the reviewers and the author. And I'm not sure everyone will agree that learning a new tool lessens the pain.
17:18:23 Well for the author it is nearly trasparent. I mean you just keep pushing fixes to your branch like you are used to
17:18:53 Where will you go to see inline comments?
17:19:06 Critic. Or your inbox.
17:19:12 There you go.
17:19:24 Sure, you have to know to do that.
17:19:29 Departure from the model everyone on GitHub is used too.
17:19:44 Also, CSS WG will argue they rather use shepherd.
17:19:56 So now you have three systems to learn instead of one.
17:20:05 It can add a comemnt at the top of the review saying "code review at {critic url}"
17:20:21 (it doesn't yet, but that doesn't seem hard to implement)
17:20:29 yay. that's going to solve the cognitive burden.
17:20:34 :P
17:20:58 Well it really isn't that hard to use. Ms2ger and I already did one review this way
17:22:10 And it doesn't have to be used if the pull request author isn't comfortable with it. But 80% of pull requests come from 20% of contributers and for that 20% there is great value in having better tools
17:22:12 Peter Linss says exactly the same about Shepherd.
17:22:27 Sure.
17:22:29 that's a chicken and egg problem.
17:22:42 Anyway my point is that I plan to keep using it
17:22:52 Because I hate doing code review on github
17:22:56 It really really sucks
17:23:00 I figured.
17:23:10 And I disagree re GH tooling.
17:23:35 No side-by-side diffs, no multiline comments, no notes, worse whitespace viewing, etc.
17:23:47 It's like being asked to use wordpad when you're used to vim or something
17:24:32 (no automatic addressing of issues, no recording progress)
17:27:20 (no filters to automatically assign reviews to the right people)
17:27:47 The main difference is that the editor you choose to work with doesn't force other people working on the same project to use the same editor as you.
17:28:08 A review tool does.
17:29:06 Anyway, it's OK that we disagree. :)
17:29:51 Should we do swords or pistols?
17:29:56 Heh :)
17:30:23 running out
17:30:32 it's my wife's birthday, today.
17:30:39 Ms2ger has joined #HTMLT
17:30:57 Well for comparison, I would actually consider reviewing Aryeh's changes from http://critic.hoppipolla.co.uk/r/5 but not so much from https://github.com/w3c/html-testsuite/pull/1
17:31:10 tobie: Cool, hope she has a nice birthday :)
17:31:29 RRSAgent, pointer?
17:31:29 See http://www.w3.org/2013/03/12-htmlt-irc#T17-31-29