IRC log of testing on 2011-11-02

Timestamps are in UTC.

18:21:50 [RRSAgent]
RRSAgent has joined #testing
18:21:50 [RRSAgent]
logging to
18:22:03 [ArtB]
Chair: Wilhelm
18:22:23 [ArtB]
Meeting: #testing breakout @ TPAC 2011
18:22:31 [ArtB]
Chair: James
18:22:43 [ArtB]
Date: 2 November 2011
18:22:55 [cyril]
cyril has joined #testing
18:23:01 [ArtB]
Scribe: ArtB
18:23:28 [ArtB]
Agenda: TPAC Testing - a Practical Sesion
18:23:36 [ArtB]
RRSAgent, make log Public
18:24:09 [Youngwan]
Youngwan has joined #testing
18:24:21 [ArtB]
Present: Wilhelm, PeterH, DavidB, JamesG, PLH, Kris, ArtB, Laszlo, EdO
18:24:44 [ArtB]
18:24:52 [ArtB]
RRSAgent, make minutes
18:24:52 [RRSAgent]
I have made the request to generate ArtB
18:25:16 [ArtB]
RRSAgent, make log Public
18:25:27 [ArtB]
JG: welcome everyone
18:25:34 [ArtB]
Present+ MarkV
18:25:49 [ArtB]
Present: Joh
18:26:12 [ArtB]
Present: Wilhelm, PeterH, DavidB, JamesG, PLH, Kris, ArtB, Laszlo, EdO
18:26:42 [ArtB]
Present+ Mark, MikeSmith, JohnJ
18:26:52 [ArtB]
JG: start with current state
18:26:56 [ArtB]
… in the various WGs
18:27:03 [ArtB]
… I know about WebApps and HTML WGs
18:27:06 [shepazu]
shepazu has joined #testing
18:27:12 [ArtB]
… want to look at testing formats
18:27:13 [MikeSmith]
MikeSmith has joined #testing
18:27:21 [ArtB]
… and the procs for gathering tests
18:27:34 [ArtB]
… Would like to know about areas for improvement
18:27:47 [ArtB]
KK: how many are familiar with Hg/Mercurial?
18:27:52 [ArtB]
… and submitting tests?
18:28:13 [ArtB]
… And how to create W3C tests?
18:28:20 [ArtB]
… [ Not very many ]
18:28:31 [ArtB]
… Can talk about how to push test cases to W3C
18:28:38 [ArtB]
… We don't @ Msft use Hg
18:28:43 [ArtB]
… but it's pretty easy
18:29:00 [ArtB]
… Can search for Mercurial
18:29:13 [JohnJansen]
18:29:15 [JanL]
JanL has joined #testing
18:29:17 [krisk]
krisk has joined #testing
18:29:28 [ArtB]
18:29:47 [TabAtkins_]
TabAtkins_ has joined #testing
18:29:51 [ArtB]
KK: the HTML WG has a wiki of resources
18:30:10 [ArtB]
… we use public-html-testsuite
18:30:11 [stearns]
18:30:52 [Youngsun]
Youngsun has joined #testing
18:31:12 [ArtB]
… HTML wiki:
18:31:34 [plh_]
plh_ has joined #testing
18:31:38 [ArtB]
… after getting Mercurial
18:31:47 [ArtB]
… must create a ~/.hgrc file
18:31:51 [jgraham]
18:31:54 [ArtB]
… or ini file for other OS's
18:32:15 [ArtB]
… May need to mess with proxy settings if behind a f/w
18:32:37 [jdurand]
jdurand has joined #testing
18:33:02 [ArtB]
… after install, Hg should be in $PATH
18:33:08 [ArtB]
… the verbs are pull and push
18:33:18 [ArtB]
… to get a local copy, use pull
18:34:01 [krisk]
18:34:07 [jgraham]
18:34:12 [ArtB]
AB: mirror is:
18:34:33 [ArtB]
KK: the Hg root has tests and specs
18:34:39 [ArtB]
… so not just about tests
18:34:54 [hober]
hober has joined #testing
18:35:07 [ArtB]
… after a test is `push`ed to server, it is stored in a src control system
18:35:20 [ArtB]
… can do some complex stuff on backend e.g. PHP
18:35:24 [plh_]
--> Test infrastructure goals
18:35:27 [ArtB]
… Several WGs are using it
18:35:44 [ArtB]
… there is a `resources` folder
18:35:50 [ArtB]
… it includes some sample tests
18:35:54 [jgraham]
18:36:29 [ArtB]
JG: there are some sample HTML5 API tests
18:36:45 [ArtB]
… e.g.
18:37:07 [ArtB]
… designed to integrate with internal testing resources/frameworks/tools
18:37:19 [ArtB]
… can make them talk to each other a bit
18:38:13 [ArtB]
KK: expect people to pull tests and run them internally
18:38:29 [ArtB]
… W3C server can't support lots of browser vendors running their tests on resources
18:39:03 [ArtB]
Q: is it possible to select specific test cases for a spec?
18:39:16 [ArtB]
… so not clear if I can find specific test cases
18:39:39 [ArtB]
JG: for HTML test suite, the directory names give a hint
18:39:56 [ArtB]
… the CSS WG is working on something more sophisticated
18:40:08 [ArtB]
PL: we are working on a bug tracker for the tests
18:40:22 [ArtB]
… want to support test case management
18:40:29 [ArtB]
… includiing metadata
18:40:42 [ArtB]
… want to link test cases to specific parts of the spec
18:40:48 [ArtB]
… it is up and running now
18:40:57 [ArtB]
… but still needs some work
18:41:09 [ArtB]
Q: how about optional vs. mandatory parts of the spec?
18:41:18 [ArtB]
PL: we want to support that
18:41:24 [matt]
matt has joined #testing
18:41:42 [ArtB]
… we need to markup the spec to facilitiate test cases pointing to specific parts of the spec
18:42:17 [ArtB]
… If want to browse the spec, want to be able to see what tests are available
18:42:28 [ArtB]
… We also use an annotation system
18:42:48 [ArtB]
… so that results from tests runs are available to someone browsing the spec
18:43:10 [ArtB]
Mark: is there any type of overall plan?
18:43:20 [ArtB]
… and some data about coverage?
18:43:36 [ArtB]
… Also, is there some convergence for tools?
18:43:49 [ArtB]
JG: yes, there is work toward tool convergence
18:43:57 [ArtB]
… at least at the harness level
18:44:21 [ArtB]
JG: re the 1st question, that's pretty complicated
18:44:56 [ArtB]
Mark: want to understand the plan for ex, HTML5
18:45:12 [ArtB]
KK: we try to facilitate, but no hard rules
18:45:23 [ArtB]
… it's up to the WG participants re what will actually get done
18:45:36 [ArtB]
… We have a structure to enable lots and lots of tests
18:46:08 [ArtB]
… We have some features that have lots (1,000s) of tests and other features with more like 10's of tests
18:46:29 [ArtB]
Mark: for CSS 2.1, is test suite done?
18:46:43 [ArtB]
PL: one reason for the tests is to get to REC
18:46:59 [ArtB]
… the other reason is to determine if everyone impleemtns the spec correctly
18:47:17 [ArtB]
… we try to use a s/w development process
18:47:37 [ArtB]
Mark: are there specific milestones?
18:47:48 [ArtB]
PL: not really, we continue to improve
18:48:02 [ArtB]
DB: as bugs are found, new tests are added
18:48:26 [ArtB]
PL: as we hit milestones, we snapshot the test suite
18:48:37 [ArtB]
… and add new tests if we need to
18:49:11 [ArtB]
KK: we need to get a handle on how close the test suite is to the impls
18:49:17 [ArtB]
… and that can be hard to determine
18:49:34 [ArtB]
… F.ex, web workers is already supported in several browsers
18:49:55 [ArtB]
… and mostly interoperable so there is a question about how much testing effort should be done
18:50:06 [plh_]
--> q?
18:50:10 [plh_]
18:50:29 [ArtB]
… The work done depends on the WG participants
18:51:03 [jdurand]
18:51:12 [ArtB]
PL: when a spec is early, it's a bit iffy to write tests
18:51:19 [ArtB]
… but it would be useful too
18:51:35 [ArtB]
JG: eventually must write tests so it makes sense to start early
18:51:55 [ArtB]
LG: agree converging test frameworks is important
18:52:10 [ArtB]
… need to eliminate outside dependencies
18:52:26 [ArtB]
KK: yes, we ran into that with some old DOM tests (from NIST)
18:52:38 [Linuz]
Linuz has joined #testing
18:52:46 [ArtB]
JG: we have a framework now
18:52:52 [ArtB]
… that is getting convergence
18:53:02 [ArtB]
… it is platform independent
18:53:10 [ArtB]
… it does depend on JavaScript
18:53:16 [ArtB]
… but no DOM dependencies
18:53:52 [ArtB]
KK: we need to be careful about adding features to the framework
18:54:29 [ArtB]
AB: how many WGs are using testharness?
18:55:15 [ArtB]
PLH: HTML, WebApps, WebPerformance, WebEvents,
18:55:35 [ArtB]
John: it was relatively easy for some tests
18:55:52 [ArtB]
LG: I think DAPI agreed to use testharness
18:56:06 [ArtB]
… but they don't have many tests yet
18:56:15 [plh_]
s/ WebEvents,//
18:56:27 [ArtB]
KK: I think we need to make sure WGs use testharness
18:56:57 [ArtB]
Q2?: what about rendering tools?
18:57:07 [ArtB]
JG: there are RefTests and manual tests
18:57:17 [ArtB]
… testharness is a JS api
18:57:27 [ArtB]
… reports results via a callback
18:58:13 [ArtB]
PLH: we have a framework on the test server to collect test results (HTML WG)
18:58:38 [ArtB]
PL: we can import results from other formats
18:58:50 [ArtB]
… the format can be XML or whatever
18:58:58 [ArtB]
… we also have an XHR API
18:59:31 [ArtB]
Q2?: need a good format
18:59:41 [ArtB]
John: testharness is really for JS APIs
18:59:55 [ArtB]
… there are challenges for visual matches
19:00:13 [ArtB]
KK: yes, e.g. font variability
19:00:23 [ArtB]
RRSAgent, make minutes
19:00:23 [RRSAgent]
I have made the request to generate ArtB
19:01:17 [Ralph_]
Ralph_ has joined #testing
19:01:19 [ArtB]
19:01:40 [kensaku]
kensaku has joined #testing
19:01:45 [ArtB]
19:02:41 [ArtB]
KK: if there are any questions, please send them to public-test-infra
19:02:48 [ArtB]
… that's the list for the Testing IG
19:03:47 [ArtB]
[ James gives a quick tutorial on how to create a test with testharness.js ]
19:04:46 [ArtB]
Present+ JacqueD, JanL
19:04:59 [ArtB]
Present+ MattW
19:05:19 [ArtB]
Present+ CyrilC
19:06:24 [ArtB]
Present+ Russell
19:06:48 [ArtB]
Russ: what about images?
19:06:55 [ArtB]
JG: most use RefTests
19:07:16 [ArtB]
… and then can compare rendering
19:07:51 [ArtB]
KK: the biggest problem with rendering is fonts
19:08:52 [ArtB]
… RefTests give expected output
19:09:23 [ArtB]
… tests with "ref" in the test file name
19:09:49 [ArtB]
… at runtime, if they render the same, the test passes
19:10:28 [ArtB]
DB: a test in RefTest is an assertion of pixel equality
19:11:13 [ArtB]
… want to exclude external factors like fonts, margins, etc.
19:11:35 [ArtB]
RRSAgent, make minutes
19:11:35 [RRSAgent]
I have made the request to generate ArtB
19:12:15 [ArtB]
Chair: James
19:12:20 [ArtB]
RRSAgent, make minutes
19:12:20 [RRSAgent]
I have made the request to generate ArtB
19:13:39 [ArtB]
Lunch Call
19:52:25 [myakura]
myakura has joined #testing
19:56:36 [stearns]
stearns has joined #testing
19:58:52 [TabAtkins_]
TabAtkins_ has joined #testing
19:59:32 [kensaku]
kensaku has joined #testing
20:08:56 [ctalbert]
ctalbert has joined #testing
20:13:03 [MikeSmith]
MikeSmith has joined #testing
20:13:49 [krisk]
krisk has joined #testing
20:14:31 [kensaku]
kensaku has joined #testing
20:18:04 [matt]
matt has left #testing
20:20:26 [myakura]
myakura has joined #testing
20:22:25 [krisk]
krisk has joined #testing
20:23:58 [cyril]
cyril has joined #testing
20:25:56 [TabAtkins_]
TabAtkins_ has joined #testing
20:29:22 [krisk]
krisk has joined #testing
20:29:45 [arronei]
arronei has joined #testing
20:30:27 [MikeSmith]
MikeSmith has joined #testing
20:34:13 [krisk]
krisk has joined #testing
20:34:33 [plinss]
plinss has joined #testing
20:35:27 [krisk]
krisk has joined #testing
20:40:27 [shepazu]
shepazu has joined #testing
20:40:42 [MichaelC]
MichaelC has joined #testing
20:43:05 [francois]
francois has joined #testing
20:43:21 [abarsto]
abarsto has joined #testing
20:45:36 [myakura]
myakura has joined #testing
20:48:19 [kensaku]
kensaku has joined #testing
20:51:53 [stearns]
stearns has joined #testing
20:54:07 [cyril_]
cyril_ has joined #testing
21:02:50 [kensaku]
kensaku has joined #testing
21:14:55 [shepazu]
shepazu has joined #testing
21:24:24 [kensaku_]
kensaku_ has joined #testing
21:34:01 [krisk]
krisk has joined #testing
21:35:14 [krisk]
krisk has left #testing
21:35:15 [kensaku]
kensaku has joined #testing
21:37:41 [plinss]
plinss has joined #testing
21:38:41 [arronei]
arronei has joined #testing
21:39:00 [abarsto]
abarsto has joined #testing
21:39:40 [ArtB]
ArtB has left #testing
21:41:03 [plinss__]
plinss__ has joined #testing
21:41:49 [cyril]
cyril has joined #testing
21:42:37 [plinss]
plinss has joined #testing
21:43:25 [francois]
francois has joined #testing
21:44:20 [MichaelC]
MichaelC has joined #testing
21:44:33 [MikeSmith]
MikeSmith has joined #testing
21:58:15 [shepazu]
shepazu has joined #testing
22:22:08 [MikeSmith]
MikeSmith has joined #testing
22:31:55 [kensaku]
kensaku has joined #testing
22:33:07 [TabAtkins_]
TabAtkins_ has joined #testing
22:36:31 [plinss]
plinss has joined #testing
22:38:50 [arronei]
arronei has joined #testing
22:39:05 [TabAtkins_]
TabAtkins_ has joined #testing
22:39:07 [MichaelC]
MichaelC has joined #testing
22:46:07 [myakura]
myakura has joined #testing
23:31:58 [myakura]
myakura has left #testing
23:43:30 [francois]
francois has joined #testing
23:43:37 [arronei]
arronei has joined #testing
23:45:11 [TabAtkins_]
TabAtkins_ has joined #testing
23:49:08 [plh]
plh has joined #testing
23:50:33 [MichaelC]
MichaelC has joined #testing
23:54:23 [kensaku]
kensaku has joined #testing