04:34:50 RRSAgent has joined #testing 04:34:50 logging to http://www.w3.org/2015/10/28-testing-irc 04:34:59 jgraham: We have two places where we collect tests 04:35:13 jgraham: We have web platform tests, github repo, can submit tests using normal github workflow 04:35:18 jgraham: Has tests for most specs apart from CSS 04:35:24 jgraham: CSS has a separate repo for historical repo 04:35:33 jgraham: It is actually an hg repo, but has a mirror in github 04:35:40 jgraham: There are slightly different requirements for each set of tests 04:35:52 jgraham: We have a site ttwf.org which attemts, oocasionally inaccurately, how you write and submit tests 04:35:59 jgraham: Kinds of things we can test atm : 04:36:06 s/ttwf.org/testthewebforward.org/ 04:36:16 jgraham: 1. Things you can access through JS DOM Apis, using testharness.js 04:36:25 jgraham: which gives you a way to write JS test 04:36:34 jgraham: 2. reftests, which are for things that depend on layout/rendering 04:36:59 jgraham: You create two versions of a document, one that uses feature you're testing, and another that's supposed to have identical rendering, but using simpler technologies (specifically, not the feature being tested) 04:37:04 jgraham: We also accept manual tests as well 04:37:33 jgraham: There's a design to add some sort of automation to that, for things that can currently only be done with manual tests or only browser-internal testing APIAs 04:38:11 Florian: In Opera we distinguished between tests requiring interaction, and others which you couldn't create a reference file, but could for subsequent runs after a pass, compare by screenshot 04:38:20 e.g. for gradients or something like that 04:38:24 dom has joined #testing 04:38:48 SimonSapin: Gecko has fuzzy reftests, where only a few pixels off. Need to specify how many pixels, and how much off 04:38:56 (Presto-based Opera, that was) 04:39:08 SimonSapin has joined #testing 04:39:19 jgraham: Those are impl detaisl at opera 04:39:33 jgraham: One of the goals that I have at last, is to allow as many tests as possible to be run in continuous integration system 04:39:33 yosuke has joined #testing 04:39:46 jgraham: Even with all of Opera's infrastructure, couldn't run those in continuous integration 04:39:47 shoko has joined #testing 04:39:52 Florian: yes 04:39:53 rrsaget, draft minutes 04:39:57 gsnedders: Sort of 04:40:00 fwtnb has joined #testing 04:40:06 jgraham: You can do it if you're running handful of interations a day 04:40:25 jgraham: If you're running 100s of integrations a day, if they all requiring manual fiddling with tests 04:40:57 jgraham: Even with Opera's case, slight tweaks would need re-evaluation, ended up with thousands of possible renderings, 04:41:05 gsnedders: I don't think anyone has a desire to do this 04:41:15 jgraham: Other thing is, in terms of where we are wrt running the tests 04:41:26 jgraham: At Mozilla we run almost all the web platform tests in automation for each commit 04:41:43 jgraham: Have an open source impl called wptrunner, which is somewhat browser-independent 04:41:51 jgraham: There's a pull request to get edge support 04:41:55 jgraham: Also have it running in servo 04:42:04 jgraham: Also in Servo, run a subset of the CSS tests using same test runner 04:42:18 jgraham: That's the State of the Union 04:42:37 jyasskin: What proportion of tests is Chrome actually running? 04:42:43 jgraham: Hoping someon can say 04:42:52 jgraham: Chrome has a heavyweight import process 04:42:59 jgraham: As a result, nobody actually imports stuff 04:43:09 jgraham: So Chrome is only running tests they upstreamed, but not taking bugfixes down 04:43:25 ojan: Really adhoc, so ppl who are motivated wrt a test suite will pull new versions 04:43:38 ?: We made some effort to run wptserv so we can run the tests as written 04:43:56 ?: That work is 80% done, but had problems finding people to do the last 20% 04:44:04 s/?/mikewest/ 04:44:05 ?: Once that's done, will be able to just run all of the tests 04:44:20 ojan: Are you working on this? 04:44:28 mikewest: Intern did most of it 04:44:34 gsnedders: So, CSS test suite 04:44:39 gsnedders: Has basically same types of tests 04:44:42 gsnedders: As wpt 04:44:56 gsnedders: But also has more metadata, such that it's possible to find out which ones can be screenshotted 04:45:21 gsnedders: Ideally we want to get rid all the screenshot compared tests 04:45:29 fantasai: Can get rid of all of them, but not quite all 04:45:43 fantasai: Some can't be turned into reftests 04:46:27 fantasai: E.g. can't test for underlines 04:46:43 fantasai: Can test that it's not not underlined, but underlining thickness and position isn't defined per spec 04:47:00 q+ 04:47:24 JohnJansen: But converting the reftestable tests is a big undertaking 04:47:31 gsnedders: We should have all new tests with references 04:47:35 Florian: When possible 04:47:45 Florian has joined #testing 04:47:52 rniwa: That's only for things that need visual comparison 04:47:56 q+ jyasskin 04:48:14 kawai has joined #testing 04:48:16 rniwa: Should be JS test if possible 04:48:43 thx gsnedders 04:48:59 JohnJansen: CSSWG resolved on Js test first, reftest second, manual test third 04:49:09 fantasai: No, resolved on reftest or JS test (automatable) over manual 04:49:11 ack r12a 04:49:21 fantasai: Didn't want to prefer JS over reftest because CSS has non-JS implementations 04:49:24 ... 04:49:31 r12a: ... 04:49:37 r12a: Can we go over objectives for this session? 04:49:52 the first was about viewing test results 04:49:54 jgraham: In regard to first thing, there is a tool that was built for visualizing which tests pass 04:49:59 jgraham: show you tests that pass 04:50:07 jgraham: but that doesn't work with the output from wptrunner 04:50:23 jgraham: There's another in-browser runner that people were using for getting specs to CR 04:50:28 jgraham: rather than continuous integration systems 04:50:35 jgraham: That will output in ways that can be read by this tool 04:50:38 plh has joined #testing 04:50:51 jgraham: For general web platform tests in wptrunenr, don't have a way to visualize 04:51:11 jgraham: CSS has other systems that allows running tests online and store test results including UA data 04:51:24 jgraham: Shoudld allow slurping test results, and display those 04:51:30 jgraham: And display results fro 200,000 tests 04:51:39 gsnedders: Should integrate with continuous integration systems 04:51:58 ack jyasskin 04:52:18 jyasskin: Totally unrelated to what's discussed so far. How do we do WebBluetooth or USB testing or geolocation 04:52:29 jyasskin: To test those, you have to tell the browser how to respond to those tests 04:52:34 jyasskin: We don't currently have a way to do that 04:52:40 jyasskin: This group should tell us how 04:52:45 rniwa: What are you saying? 04:53:06 jyasskin: For web bluetooth, the implementation we have right now doesn't go end to end. Tests platform-independent part of crhom 04:53:23 jyasskin: As part of the test, need to make the API call that shows dialog, tell browser how to respond to dialog 04:53:34 jyasskin: Spec says there should be some prompt, or that the user grants permission in some way 04:53:37 jyasskin: Has a space for UI to appear 04:53:48 jyasskin: You don't get the response until permission granted 04:54:00 jyasskin: Want to configure a fake device that will respond in certain ways to bluetooth radio 04:54:15 jyasskin: Then want to assert that those responses make it to the browser and .. 04:54:22 Florian: Or geolocation, pretend to be somewhere 04:54:29 rniwa: if you're mocking it, don't know if it actually works or not 04:54:33 jyasskin: we test half of the thing 04:54:47 jyasskin: We can use same functions to configure a physical device that actually tests the whole stack. 04:54:56 jyasskin: Would like tests to work for both 04:55:03 jyasskin: Produce same results 04:55:16 rniwa: Suppose bluetooth device, need to have a very specific deice for very speciif cresult 04:55:26 jyasskin: There ar test devices that allow responding in specific ways 04:55:33 rniwa: bluetoothe Q tool 04:55:34 jyasskin: yes 04:55:48 jyasskin: want to run in a lab, but want to run tests in multipe browsers, and be able to run test if you don't have the lab 04:56:06 jyasskin: I have a spec for this set of testing functions, but happens to be what we write in chrome, it's terrible names, no consensus on this 04:56:17 rniwa: Seems like a prime candidate for webdriver 04:56:25 rniwa: Seems like the right place to do this kind of stuff 04:56:30 jyasskin: Talked to them yesterday 04:56:41 jyasskin: For first part, controlling dialog, yes, but 2nd part maybe not 04:57:10 fantasai: We're off-topic. Going back to r12a's 2nd question, clarifying the topic 04:58:16 Sorry for asking the off-topic question. :) 04:58:34 fantasai: the topic is actually primarily getting everything synchronied up and getting everyone running it up so we don't have duplicate tests being written and then can spend more reasources on writing different tests 04:59:44 [...] 05:00:08 jgraham: Going back to synchronization 05:00:14 q+ 05:00:19 jgraham: Situation we have at Mozilla atm is worth talking about, because better than anyone else has 05:00:39 jgraham: For web platform tests, we have a script that allows us to.. it pulls in the upstream repository and replaces our copy with the upstream copy 05:00:45 jgraham: Which would be fine, and what we had at the beginning 05:00:53 jgraham: But didn't allow devs an easy workflow to submit tests 05:01:06 jgraham: So what we did then is we added functionality that allows devs to land patches on a local copy of the test 05:01:17 jgraham: And before we do a pull, those patches get upstreameed 05:01:35 jgraham: web platform test review policy is that as long as review was public, it's accepted as a review 05:01:54 jgraham: We upstream the patches, and then pull down the changes from the w3c master 05:02:14 jgraham: We then have to update metadata about which changed tests pass/fail 05:02:23 jgraham: That takes about a day, mostly automatic and just waiting 05:02:33 jgraham: With CSS, the tests that we run aren't the tests that we submit 05:02:48 jgraham: We submit source files, but run built tests 05:02:55 jgraham: So our devs can't patch the tests 05:02:57 q+ 05:03:01 SimonSapin: CSS tests have a build system 05:03:06 jgraham: The build system changes the tests 05:03:20 r12a: Do we still need that? 05:03:32 r12a: It was introduced years ago when we had to deal with XHTML-only implementations 05:03:43 gsnedders: Part of it was to get of CR 05:04:31 fantasai: Was not just getting out of CR, but also that we wanted the tests to be able to run in more CSS implementations than just browsers 05:05:16 fantasai: we can probably change the build system at this point o that the HTML copy of the tests is just a pass through, we do need to parse it to extract the mtadata but we can probably change that 05:06:29 ... 05:06:46 gsnedders: Need that, also need accepting review from other organizations 05:06:55 s/gsnedders/jgraham/ 05:07:17 .... 05:07:35 rniwa: I have oposite problem, get reviews where need to change the test, and dont' have time to go back and fix the test. 05:08:01 Florian: Depending on who's reviewing the test, can get comments on "would be better to fix this, but not necessary" vs. "this test is incorrect" 05:08:04 ack gsnedders 05:08:15 q- 05:08:20 ?: Same way that browsers reviewing patches, also need to review tests 05:09:18 fantasai: I've noticed in a lot of cases, tests aren't reviewed, just "yay, you have tests, good good check it in" 05:09:31 q? 05:09:39 jgraham: Once we have tests running everywhere, breakage on a different impl will highlight test errors 05:10:00 gsnedders: better to run it, because ppl runing tests will notice it 05:10:15 Florian: If the tests fail when they should not, will catch it. If the tests pass when they should not, will not catch it. 05:10:31 == about:blank about:blank is a great test right? 05:10:33 Florian: e.g. test that written to not fail 05:12:33 fantasai: I think it's fine to go with this approach for browser vendors in our community, just need to be clear that W3C tests might not be correct, need to read spec when implementing 05:13:01 fantasai: I'm concerned that people will try to fix implementation to match the tests instead of reporting errors in the tests 05:13:24 fantasai: Esp. implementers in China, Japan, places that don't speak English and aren't as wlel integrated into this community 05:13:36 gsnedders: What do people need to get this to work/ 05:13:58 jgraham: automate more tests 05:14:34 ... 05:15:48 jgraham: Need to make it easy for people to fix tests locally and upstream the fixes 05:15:53 [discussion about out-of-date documentation] 05:16:16 JohnJansen: Having documentation all in one place makes it easy to integrate tests 05:16:23 JohnJansen: We import tests ever week, run them every day 05:16:40 JohnJansen: Our automation for CSS tests is screenshot based, very problematic, want to switch to reftests 05:16:49 JohnJansen: It's hard for us 05:17:02 JohnJansen: There's tribal knowledge necesary to contribute to CSSWG tests 05:17:15 gsnedders: So need to make it easier to contribute tests 05:19:12 rniwa: Make it easier for browser vendors to sync 05:19:18 rniwa: Need to import all tests automatically 05:19:22 rniwa: Not set by set 05:20:26 fantasai: Mozilla has a directory that gets automatically synced, but nobody puts tests into that directory for some reason 05:21:00 ?: For blink, isn't about import directories, but about it being on a different server 05:21:07 ?: Because of special headers etc. 05:21:16 s/?/mikewest/ 05:21:27 s/mikewest/mkwst/ 05:21:30 mikewest: Setting up a server is historically difficult 05:22:19 jgraham: The difficulty at Mozilla has been that people are more familiar with existing tools, so use Mozilla-specific tools instead of wpt 05:22:29 jgraham: Or they need browser-specific APis 05:23:03 mkwst: Some layout tests we use browser-speciic stuff, could possibly be done with DOM but very tricky 05:23:16 JohnJansen: Blinks' layout tests require internal calls 05:23:30 mkwst: We have things we wnat to do to set up browser in certain ways, need to use internal APIs 05:23:40 jyasskin: No consensus on standardizing these APis 05:23:51 jgraham: Historically ppl hesitant to standardize these APIs 05:23:58 rniwa: Click event, etc. that coudl be done through webdriver 05:24:01 rniwa: halfway there 05:24:08 rniwa: can add more of that stuff 05:24:17 rniwa: tricky ones, e.g. geolocation 05:24:26 rniwa: I'd imagine that feature would be very useful for random websites, too 05:24:34 rniwa: Let's say your'e trying to show UI based on location 05:24:49 rniwa: if you could tell browser to pretend it's in Japan could be useful 05:24:55 q+ 05:24:57 jgraham: Class of features would be useful for adding to web driver 05:25:10 jgraham: There's also a class of really internal stuff, that nobody wants to expose to authors 05:25:20 jgraham: Like "now trigger a garbage collection", which would be a disaster to have on the web 05:25:39 jgraham: That's the thing where ppl go, well since I need to use this interal API in 1/10 tests, will write all the tests using that API 05:25:51 mkwst: Also test that for convenience, will print things from browser internals 05:25:57 mkwst: For resource loading e.g. 05:26:02 mkwst: We wouldn't want that on the web 05:26:07 mkwst: There are class of thigns we want to test 05:26:11 mkwst: Hard to test with content-level apis 05:28:07 fantasai: For tests where we have the ability, we need to address people writing tests in the wrong format without any real excuse 05:28:10 fantasai: So how do we do that? 05:28:17 [discussion of review policies requiring tests] 05:28:25 s/tests/tests to be in the right format/ 05:28:39 jgraham: For Servo, policy is if you can write a test that we can upstream, do it. 05:28:54 jgraham: If you can't, use the same harness if policy, but put it into a different directory 05:29:05 jgraham: But Servo doesn't have a history, devs haven't learned 05:29:18 jgraham: Problem with Google, and Mozilla etc. then have 500 engineers, whov'e been doign something different for 10 years 05:30:15 fantasai: If we can get just the reviewers to switch over and enforce that switch on the patches they review, then we can make that shift happen 05:30:30 jgraham: Need to make it easy enough for the reviewers to do that, so if it's hard currently need to fix that. 05:30:43 Florian: We also have presto-testo repo with lots of tests 05:30:47 jgraham: There has been some effort 05:30:52 Florian: There's still 80,000 files in it 05:30:58 gsnedders: Not much interesting 05:31:32 gsnedders, jgraham: Let's keep talking about this 05:31:37 jgraham: feel free to chat with us 05:31:44 jgraham: Discussion is on public-test-infra@w3.org 05:31:51 jgraham: CSS also has public-css-testsuite@w3.org 05:31:56 gsnedders: Relatively low traffic atm 05:32:59 jyasskin has joined #testing 05:33:42 1. Change the build system at CSSWG 05:33:43 2. Fixing ttwf documentation to make CSSWG testing info findable and up-to-date 05:33:43 3. Automate more tests 05:33:43 4. Infrastructure for automating manual tests in cross-browser way 05:33:43 5. Get browser vendors to agree that all new tests should be wpt/testharness/reftest format 05:33:50 rniwa: testharness is verbose 05:33:55 jgraham: It's a lot better now 05:34:40 FWIW, Blink folks _do_ write tests in testharness, but so many things are impossible there. 05:35:31 need to reduce required metadata in tests 05:36:30 r12a: People leaving out assertions is problematic, can't tell what's being tested 05:37:43 RRSAgent: stop 07:10:42 plh has joined #testing 07:10:43 Meeting: Web Platform Testing 07:10:49 RRSAgent: off 07:14:55 RRSAgent has joined #testing 07:14:55 logging to http://www.w3.org/2015/10/28-testing-irc 07:15:15 Chair: gsnedders 07:15:59 ACTION: Change the build system at CSSWG 07:16:06 ACTION: Fixing ttwf documentation to make CSSWG testing info findable and up-to-date 07:16:14 ACTION: Automate more tests 07:16:23 ACTION: Infrastructure for automating manual tests in cross-browser way 07:16:39 ACTION: Get browser vendors to agree that all new tests should be wpt (testharness/reftest) format 07:16:46 ACTION: Reduce metadata requirements in CSSTS 07:16:55 RRSAgent: make the minutes 07:16:55 I have made the request to generate http://www.w3.org/2015/10/28-testing-minutes.html gsnedders 07:18:40 chitra has joined #testing 07:19:08 Present: Florian JohnJansen SimonSapin dom fantasai gsnedders jgraham jyasskin kawai mkwst ojan r12a rniwa shoko yosuke 07:19:32 RRSAgent: make the minutes 07:19:32 I have made the request to generate http://www.w3.org/2015/10/28-testing-minutes.html gsnedders 07:19:54 RRSAgent: quit 07:19:54 I'm logging. I don't understand 'quit', gsnedders. Try /msg RRSAgent help 07:20:04 RRSAgent: off