TPAC 2018 Plenary Day web-platform-tests session -- 24 Oct 2018

<gsnedders> did anyone go to the CSS WG + testing discussion session earlier?

<boaz> I did

<boaz> i have notes I can share

<gsnedders> boaz: drop me an email?

<gsnedders> or publicly, or whatever

<boaz> yikes

<boaz> ack

<astearns> the raw minutes start here: https://log.csswg.org/irc.w3.org/css/2018-10-24/#e1101428

<gsnedders> s|s///G||

<foolip> slides are at https://docs.google.com/presentation/d/1j74NaHk7B9ZzRMMUFSQMrp1WWkiBAPivA66jVVptbrI/edit?usp=sharing

<inserted> Scribe: gsnedders

foolip gives a presentation

[foolip and lukebjerring do so, summarising the history of web-platform-tests, reading from the above linked slides]

cbiesinger: The Edge results are over a month old?

lukebjerring: we'll get to that

[more slides]

lukebjerring: ... Edge is currently waiting on JohnJansen providing more up to date results
... main thing we want to cover today are contributor pain points

Q&A

<inserted> Scribe: foolip

surma: talked to Jake A about an idea:
... we could provide a glitch as a starting point
... templates as an entry point, then maybe tell people where to put it into the test suite, lower the barrier

fremy: I have https://wptest.center/#/new
... if you know the web platform you probably have a fork of wpt, but a lot of the time people might not want to spend time creating PRs and all that
... would be nice to make it easier to create PRs just from files, without a clone
... for a single test, that automation would be great
... let's say you have a test case, then we'd like to see a smoother path
... especially for newcomers
... lots of folders... not clear where to go. would be nice with automation.

zcorpan_: surma also said you should be able to run the tests without a local setup
... I think we should support that

surma: glitch is the new hotness like jsfiddle. but we can give it a docker container

<AutomatedTester> RRSAgent: make minutes

surma: there's a collaboration feature, seems like a good tool we could utilize
... there will be some tests we can't develop there, but the majority
... it's a good start for testharness.js or reftests
... and you can immediately see the results, play around, then PR

zcorpan_: web-platform-tests.live has what's on master, not what's on master

marcosc: bless() will be hard to do when running tests manually

estes: [foolip couldn't scribe]

<Hexcles> foolip: for bless() specifically, we will be able to run it manually with jugglinmike's PR

marcosc: so you converted the tests from WebKit?

estes: yes, we have window.internals that we used for web payments
... I copied manual tests for wpt, but don't know how to upstream them.
... every browser has a different UI, so we'd need something different

jgraham: the spec would have to say what the API should be for testing

<jose-cdbtr> join #tpac-chat

jgraham: not sensible for everyone to have their own internal API that works different
... ideally a WebDriver API so that authors can test it, but if that doesn't make sense a test-only DOM API would be good
... but the spec needs to say what the APIs do

mgiuca: I think this applies to a lot of new APIs
... do you mean the Web IDL should have a [ForTesting] thing?

jgraham: yes, there's some discussion around this
... WebKit folks said it's best if it all hangs off a single objecting

mgiuca: but I'd like to be able to test this in Chrome stable

jgraham: it would depend on browser policy, could have a pref for testing that can be enabled
... WebKit doesn't want to compile it into the binary, so those tests will fail on wpt.fyi unless we run webkit test runner

<Hexcles> foolip: no established pattern/recommendation yet

foolip: if there's a candidate for exploring this, please speak up

marcosc: there hasn't been much interest in payments yet

foolip: but people have shared the tests
... and done the work to make them run

<gsnedders> foolip: [ForTesting] would be a reasonable thing to have in WebIDL, though it might not be ideal

mgiuca: maybe we could to web share (target)

bz: gecko has internal testing things

<mgiuca> +Present mgiuca

gsnedders: there's concern from Blink about the binary size cost of testing APIs

foolip: Blink concern has been with very large APIs like USB and Bluetooth

bz: there's also security concerns, some of these we really don't want to expose to userse

fremy: probably don't want a place to allow arbitrary payments :)

foolip: soliciting questions from newcomers here
... we know PR review latency is a problem

gsnedders: what can we do?

marcosc: I think we should limit the number of reviewers
... if you're listed and don't review within 2 days, that's bad

lukebjerring: we don't know what the policy change should be, but we'd like to do round robin style
... from a list of people, you only assign one person
... if that person is assigned, they should either delegate or take care of it
... removes the diffusion of responsibility
... we're hoping that will help

fremy: some people take vacation

marcosc: people should say they're on vacation. this is our job

jgraham: we could have an indicator

<jib> jib: q+

fremy: I'm assigned to a lot of reviews
... I don't see any email
... I just use notifications

https://web-platform-tests.org/reviewing-tests/email.html

foolip: redirect your GitHub email to work and filter per that documentation
... we'll try round robin first and some manual poking

zcorpan_: another idea was notifying people in their own review systems
... could help with people who don't pay attention to Github

jgraham: that might work for gecko and chromium, but not microsoft and apple

zcorpan_: we can talk to them

jib: I still write PRs and am not sure who can review

lukebjerring: with the round robin change it was suggested we also list all the reviewers

<jib> jib is Jan-Ivar, Mozilla

lukebjerring: for notifying people in their own review system, that would be on a case-by-case basis

fremy: CSS is interesting
... two people are editors for the vast majority of the spec
... they'll have to either write or review tests
... we have a limited set of people, bottlenecks

foolip: on reviews, anyone who's in the reviewers team can review

lukebjerring: apart from reviews, any other reasons people aren't excited to use GitHub?

<Hexcles> foolip: one big problem is showing test results (regressions) in PRs

foolip: surprised that nobody has brought up the difficulty of understanding test results on PRs

fremy: the checkbox only tells you if it's stable
... didn't think there would be a better way

bz: as a browser engineer, I write tests and run on my browsers
... fixing that bug, I might as well check it into gecko and get it upstreamed that way
... it would still be useful to run them cross-browser
... but can't run Edge locally

foolip: if we have results and regressions on PRs, then we could use the in-flight PRs to get cross-browser results

jgraham: we don't have in-flight PRs for gecko yet
... haven't had that because of too many review systems, but maybe we can do that now

bz: most useful would be the ability to submit a WIP test and get results

jgraham: that would probably be possible
... you could do something with branches perhaps

foolip: I think that could work wit a special branch name pattern

jib: curious about how people feel that no test no commit policy is working

foolip: how have you found it in WebRTC?

jib: maybe more software rules

foolip: would you like a status check on spec PRs which fails if you don't link to tests?

jib: maybe
... also for new things you might not write tests

<jib> jib: ...until you have an implementation to write it with

foolip: for new features, I would not recommend strictly requiring tests
... makes sense to file an issue and have the first implementer write the tests

marcosc: we've written tests before implementation

foolip: and did that work out, or lots of broken tests?

marcosc: some bugs, sure

foolip: but not enough to make it a net negative to write tests up front?

marcosc: no. and spec quality shot up
... when removing things in payments, we add it to historical.html test
... the whole cycle really works, for us it works really well

jib: are there limits for what we can add web-platform-tests for?

foolip: there are tentative tests https://web-platform-tests.org/writing-tests/file-names.html

zcorpan_: if there's just a single vendor interested, maybe they shouldn't be part of the test suite

mgiuca: I disagree with that, if you're writing a spec you'd like others to implement the tests will be helpful
... to not have to go digging around

jgraham: 2 cases
... 1. nobody is interested yet but you think they will be
... 2. using a webkit-internal API

mgiuca: I mean (1)

zcorpan_: I mean if some browser vendor is actively opposed

marcosc: that's happened, with performance.memory
... that could have had tests

<mgiuca> foolip: There's a risk if the tests live in one vendor repo that they may not be interested inupstreaming later.

<mgiuca> ... Little risk in keeping it in a shared repo. You can disable them until needed.

<gsnedders> RRSAgent: off

- DRAFT -

TPAC 2018 Plenary Day web-platform-tests session

24 Oct 2018

Attendees

Contents

foolip gives a presentation

Q&A

Summary of Action Items

Summary of Resolutions

Scribe.perl diagnostic output