18:21:50 <RRSAgent> RRSAgent has joined #testing
18:21:50 <RRSAgent> logging to http://www.w3.org/2011/11/02-testing-irc
18:22:03 <ArtB> Chair: Wilhelm
18:22:23 <ArtB> Meeting: #testing breakout @ TPAC 2011
18:22:31 <ArtB> Chair: James
18:22:43 <ArtB> Date: 2 November 2011
18:22:55 <cyril> cyril has joined #testing
18:23:01 <ArtB> Scribe: ArtB
18:23:28 <ArtB> Agenda: TPAC Testing - a Practical Sesion
18:23:36 <ArtB> RRSAgent, make log Public
18:24:09 <Youngwan> Youngwan has joined #testing
18:24:21 <ArtB> Present: Wilhelm, PeterH, DavidB, JamesG, PLH, Kris, ArtB, Laszlo, EdO
18:24:44 <ArtB> s/PeterH/PeterL/
18:24:52 <ArtB> RRSAgent, make minutes
18:24:52 <RRSAgent> I have made the request to generate http://www.w3.org/2011/11/02-testing-minutes.html ArtB
18:25:16 <ArtB> RRSAgent, make log Public
18:25:27 <ArtB> JG: welcome everyone
18:25:34 <ArtB> Present+ MarkV
18:25:49 <ArtB> Present: Joh
18:26:12 <ArtB> Present: Wilhelm, PeterH, DavidB, JamesG, PLH, Kris, ArtB, Laszlo, EdO
18:26:42 <ArtB> Present+ Mark, MikeSmith, JohnJ
18:26:52 <ArtB> JG: start with current state
18:26:56 <ArtB> … in the various WGs
18:27:03 <ArtB> … I know about WebApps and HTML WGs
18:27:06 <shepazu> shepazu has joined #testing
18:27:12 <ArtB> … want to look at testing formats
18:27:13 <MikeSmith> MikeSmith has joined #testing
18:27:21 <ArtB> … and the procs for gathering tests
18:27:34 <ArtB> … Would like to know about areas for improvement
18:27:47 <ArtB> KK: how many are familiar with Hg/Mercurial?
18:27:52 <ArtB> … and submitting tests?
18:28:13 <ArtB> … And how to create W3C tests?
18:28:20 <ArtB> … [ Not very many ]
18:28:31 <ArtB> … Can talk about how to push test cases to W3C
18:28:38 <ArtB> … We don't @ Msft use Hg
18:28:43 <ArtB> … but it's pretty easy
18:29:00 <ArtB> … Can search for Mercurial
18:29:13 <JohnJansen> http://mercurial.selenic.com/
18:29:15 <JanL> JanL has joined #testing
18:29:17 <krisk> krisk has joined #testing
18:29:28 <ArtB> AB: http://www.w3.org/2008/webapps/wiki/WorkMode#CVS_and_Mercurial
18:29:47 <TabAtkins_> TabAtkins_ has joined #testing
18:29:51 <ArtB> KK: the HTML WG has a wiki of resources
18:30:10 <ArtB> … we use public-html-testsuite
18:30:11 <stearns> http://wiki.csswg.org/test/mercurial
18:30:52 <Youngsun> Youngsun has joined #testing
18:31:12 <ArtB> … HTML wiki: http://www.w3.org/html/wg/wiki/Testing
18:31:34 <plh_> plh_ has joined #testing
18:31:38 <ArtB> … after getting Mercurial
18:31:47 <ArtB> … must create a ~/.hgrc file
18:31:51 <jgraham> http://www.w3.org/html/wg/wiki/Testing
18:31:54 <ArtB> … or ini file for other OS's
18:32:15 <ArtB> … May need to mess with proxy settings if behind a f/w
18:32:37 <jdurand> jdurand has joined #testing
18:33:02 <ArtB> … after install, Hg should be in $PATH
18:33:08 <ArtB> … the verbs are pull and push
18:33:18 <ArtB> … to get a local copy, use pull
18:34:01 <krisk> http://www.w3.org/html/wg/wiki/Testing/Submission/
18:34:07 <jgraham> http://dvcs.w3.org/hg/
18:34:12 <ArtB> AB: mirror is: http://w3c-test.org//
18:34:33 <ArtB> KK: the Hg root has tests and specs
18:34:39 <ArtB> … so not just about tests
18:34:54 <hober> hober has joined #testing
18:35:07 <ArtB> … after a test is `push`ed to server, it is stored in a src control system
18:35:20 <ArtB> … can do some complex stuff on backend e.g. PHP
18:35:24 <plh_> --> http://www.w3.org/wiki/TestInfra/goals Test infrastructure goals
18:35:27 <ArtB> … Several WGs are using it
18:35:44 <ArtB> … there is a `resources` folder
18:35:50 <ArtB> … it includes some sample tests
18:35:54 <jgraham> http://w3c-test.org/resources/
18:36:29 <ArtB> JG: there are some sample HTML5 API tests
18:36:45 <ArtB> … e.g. http://w3c-test.org//resources/apisample.htm
18:37:07 <ArtB> … designed to integrate with internal testing resources/frameworks/tools
18:37:19 <ArtB> … can make them talk to each other a bit
18:38:13 <ArtB> KK: expect people to pull tests and run them internally
18:38:29 <ArtB> … W3C server can't support lots of browser vendors running their tests on w3.org resources
18:39:03 <ArtB> Q: is it possible to select specific test cases for a spec?
18:39:16 <ArtB> … so not clear if I can find specific test cases
18:39:39 <ArtB> JG: for HTML test suite, the directory names give a hint
18:39:56 <ArtB> … the CSS WG is working on something more sophisticated
18:40:08 <ArtB> PL: we are working on a bug tracker for the tests
18:40:22 <ArtB> … want to support test case management
18:40:29 <ArtB> … includiing metadata
18:40:42 <ArtB> … want to link test cases to specific parts of the spec
18:40:48 <ArtB> … it is up and running now
18:40:57 <ArtB> … but still needs some work
18:41:09 <ArtB> Q: how about optional vs. mandatory parts of the spec?
18:41:18 <ArtB> PL: we want to support that
18:41:24 <matt> matt has joined #testing
18:41:42 <ArtB> … we need to markup the spec to facilitiate test cases pointing to specific parts of the spec
18:42:17 <ArtB> … If want to browse the spec, want to be able to see what tests are available
18:42:28 <ArtB> … We also use an annotation system
18:42:48 <ArtB> … so that results from tests runs are available to someone browsing the spec
18:43:10 <ArtB> Mark: is there any type of overall plan?
18:43:20 <ArtB> … and some data about coverage?
18:43:36 <ArtB> … Also, is there some convergence for tools?
18:43:49 <ArtB> JG: yes, there is work toward tool convergence
18:43:57 <ArtB> … at least at the harness level
18:44:21 <ArtB> JG: re the 1st question, that's pretty complicated
18:44:56 <ArtB> Mark: want to understand the plan for ex, HTML5
18:45:12 <ArtB> KK: we try to facilitate, but no hard rules
18:45:23 <ArtB> … it's up to the WG participants re what will actually get done
18:45:36 <ArtB> … We have a structure to enable lots and lots of tests
18:46:08 <ArtB> … We have some features that have lots (1,000s) of tests and other features with more like 10's of tests
18:46:29 <ArtB> Mark: for CSS 2.1, is test suite done?
18:46:43 <ArtB> PL: one reason for the tests is to get to REC
18:46:59 <ArtB> … the other reason is to determine if everyone impleemtns the spec correctly
18:47:17 <ArtB> … we try to use a s/w development process
18:47:37 <ArtB> Mark: are there specific milestones?
18:47:48 <ArtB> PL: not really, we continue to improve
18:48:02 <ArtB> DB: as bugs are found, new tests are added
18:48:26 <ArtB> PL: as we hit milestones, we snapshot the test suite
18:48:37 <ArtB> … and add new tests if we need to
18:49:11 <ArtB> KK: we need to get a handle on how close the test suite is to the impls
18:49:17 <ArtB> … and that can be hard to determine
18:49:34 <ArtB> … F.ex, web workers is already supported in several browsers
18:49:55 <ArtB> … and mostly interoperable so there is a question about how much testing effort should be done
18:50:06 <plh_> --> q?
18:50:10 <plh_> q?
18:50:29 <ArtB> … The work done depends on the WG participants
18:51:03 <jdurand> q?
18:51:12 <ArtB> PL: when a spec is early, it's a bit iffy to write tests
18:51:19 <ArtB> … but it would be useful too
18:51:35 <ArtB> JG: eventually must write tests so it makes sense to start early
18:51:55 <ArtB> LG: agree converging test frameworks is important
18:52:10 <ArtB> … need to eliminate outside dependencies
18:52:26 <ArtB> KK: yes, we ran into that with some old DOM tests (from NIST)
18:52:38 <Linuz> Linuz has joined #testing
18:52:46 <ArtB> JG: we have a framework now 
18:52:52 <ArtB> … that is getting convergence
18:53:02 <ArtB> … it is platform independent
18:53:10 <ArtB> … it does depend on JavaScript
18:53:16 <ArtB> … but no DOM dependencies
18:53:52 <ArtB> KK: we need to be careful about adding features to the framework
18:54:29 <ArtB> AB: how many WGs are using testharness?
18:55:15 <ArtB> PLH: HTML, WebApps, WebPerformance, WebEvents, 
18:55:35 <ArtB> John: it was relatively easy for some tests
18:55:52 <ArtB> LG: I think DAPI agreed to use testharness
18:56:06 <ArtB> … but they don't have many tests yet
18:56:15 <plh_> s/ WebEvents,//
18:56:27 <ArtB> KK: I think we need to make sure WGs use testharness
18:56:57 <ArtB> Q2?: what about rendering tools?
18:57:07 <ArtB> JG: there are RefTests and manual tests
18:57:17 <ArtB> … testharness is a JS api
18:57:27 <ArtB> … reports results via a callback
18:58:13 <ArtB> PLH: we have a framework on the test server to collect test results (HTML WG)
18:58:38 <ArtB> PL: we can import results from other formats
18:58:50 <ArtB> … the format can be XML or whatever
18:58:58 <ArtB> … we also have an XHR API
18:59:31 <ArtB> Q2?: need a good format
18:59:41 <ArtB> John: testharness is really for JS APIs
18:59:55 <ArtB> … there are challenges for visual matches
19:00:13 <ArtB> KK: yes, e.g. font variability
19:00:23 <ArtB> RRSAgent, make minutes
19:00:23 <RRSAgent> I have made the request to generate http://www.w3.org/2011/11/02-testing-minutes.html ArtB
19:01:17 <Ralph_> Ralph_ has joined #testing
19:01:19 <ArtB> s/Q?/Jan/g
19:01:40 <kensaku> kensaku has joined #testing
19:01:45 <ArtB> s/Q2?/JD/g
19:02:41 <ArtB> KK: if there are any questions, please send them to public-test-infra
19:02:48 <ArtB> … that's the list for the Testing IG
19:03:47 <ArtB> [ James gives a quick tutorial on how to create a test with testharness.js ]
19:04:46 <ArtB> Present+ JacqueD, JanL
19:04:59 <ArtB> Present+ MattW
19:05:19 <ArtB> Present+ CyrilC
19:06:24 <ArtB> Present+ Russell
19:06:48 <ArtB> Russ: what about images?
19:06:55 <ArtB> JG: most use RefTests
19:07:16 <ArtB> … and then can compare rendering
19:07:51 <ArtB> KK: the biggest problem with rendering is fonts
19:08:52 <ArtB> … RefTests give expected output
19:09:23 <ArtB> … tests with "ref" in the test file name
19:09:49 <ArtB> … at runtime, if they render the same, the test passes
19:10:28 <ArtB> DB: a test in RefTest is an assertion of pixel equality
19:11:13 <ArtB> … want to exclude external factors like fonts, margins, etc.
19:11:35 <ArtB> RRSAgent, make minutes
19:11:35 <RRSAgent> I have made the request to generate http://www.w3.org/2011/11/02-testing-minutes.html ArtB
19:12:15 <ArtB> Chair: James
19:12:20 <ArtB> RRSAgent, make minutes
19:12:20 <RRSAgent> I have made the request to generate http://www.w3.org/2011/11/02-testing-minutes.html ArtB
19:13:39 <ArtB> Lunch Call
19:52:25 <myakura> myakura has joined #testing
19:56:36 <stearns> stearns has joined #testing
19:58:52 <TabAtkins_> TabAtkins_ has joined #testing
19:59:32 <kensaku> kensaku has joined #testing
20:08:56 <ctalbert> ctalbert has joined #testing
20:13:03 <MikeSmith> MikeSmith has joined #testing
20:13:49 <krisk> krisk has joined #testing
20:14:31 <kensaku> kensaku has joined #testing
20:18:04 <matt> matt has left #testing
20:20:26 <myakura> myakura has joined #testing
20:22:25 <krisk> krisk has joined #testing
20:23:58 <cyril> cyril has joined #testing
20:25:56 <TabAtkins_> TabAtkins_ has joined #testing
20:29:22 <krisk> krisk has joined #testing
20:29:45 <arronei> arronei has joined #testing
20:30:27 <MikeSmith> MikeSmith has joined #testing
20:34:13 <krisk> krisk has joined #testing
20:34:33 <plinss> plinss has joined #testing
20:35:27 <krisk> krisk has joined #testing
20:40:27 <shepazu> shepazu has joined #testing
20:40:42 <MichaelC> MichaelC has joined #testing
20:43:05 <francois> francois has joined #testing
20:43:21 <abarsto> abarsto has joined #testing
20:45:36 <myakura> myakura has joined #testing
20:48:19 <kensaku> kensaku has joined #testing
20:51:53 <stearns> stearns has joined #testing
20:54:07 <cyril_> cyril_ has joined #testing
21:02:50 <kensaku> kensaku has joined #testing
21:14:55 <shepazu> shepazu has joined #testing
21:24:24 <kensaku_> kensaku_ has joined #testing
21:34:01 <krisk> krisk has joined #testing
21:35:14 <krisk> krisk has left #testing
21:35:15 <kensaku> kensaku has joined #testing
21:37:41 <plinss> plinss has joined #testing
21:38:41 <arronei> arronei has joined #testing
21:39:00 <abarsto> abarsto has joined #testing
21:39:40 <ArtB> ArtB has left #testing
21:41:03 <plinss__> plinss__ has joined #testing
21:41:49 <cyril> cyril has joined #testing
21:42:37 <plinss> plinss has joined #testing
21:43:25 <francois> francois has joined #testing
21:44:20 <MichaelC> MichaelC has joined #testing
21:44:33 <MikeSmith> MikeSmith has joined #testing
21:58:15 <shepazu> shepazu has joined #testing
22:22:08 <MikeSmith> MikeSmith has joined #testing
22:31:55 <kensaku> kensaku has joined #testing
22:33:07 <TabAtkins_> TabAtkins_ has joined #testing
22:36:31 <plinss> plinss has joined #testing
22:38:50 <arronei> arronei has joined #testing
22:39:05 <TabAtkins_> TabAtkins_ has joined #testing
22:39:07 <MichaelC> MichaelC has joined #testing
22:46:07 <myakura> myakura has joined #testing
23:31:58 <myakura> myakura has left #testing
23:43:30 <francois> francois has joined #testing
23:43:37 <arronei> arronei has joined #testing
23:45:11 <TabAtkins_> TabAtkins_ has joined #testing
23:49:08 <plh> plh has joined #testing
23:50:33 <MichaelC> MichaelC has joined #testing
23:54:23 <kensaku> kensaku has joined #testing