<gsnedders> did anyone go to the CSS WG + testing discussion session earlier?
<boaz> I did
<boaz> i have notes I can share
<gsnedders> boaz: drop me an email?
<gsnedders> or publicly, or whatever
<boaz> yikes
<boaz> ack
<astearns> the raw minutes start here: https://log.csswg.org/irc.w3.org/css/2018-10-24/#e1101428
<gsnedders> s|s///G||
<foolip> slides are at https://docs.google.com/presentation/d/1j74NaHk7B9ZzRMMUFSQMrp1WWkiBAPivA66jVVptbrI/edit?usp=sharing
<inserted> Scribe: gsnedders
[foolip and lukebjerring do so, summarising the history of web-platform-tests, reading from the above linked slides]
cbiesinger: The Edge results are over a month old?
lukebjerring: we'll get to that
[more slides]
lukebjerring: ... Edge is
currently waiting on JohnJansen providing more up to date
results
... main thing we want to cover today are contributor pain
points
<inserted> Scribe: foolip
surma: talked to Jake A about an
idea:
... we could provide a glitch as a starting point
... templates as an entry point, then maybe tell people where
to put it into the test suite, lower the barrier
fremy: I have https://wptest.center/#/new
... if you know the web platform you probably have a fork of
wpt, but a lot of the time people might not want to spend time
creating PRs and all that
... would be nice to make it easier to create PRs just from
files, without a clone
... for a single test, that automation would be great
... let's say you have a test case, then we'd like to see a
smoother path
... especially for newcomers
... lots of folders... not clear where to go. would be nice
with automation.
zcorpan_: surma also said you
should be able to run the tests without a local setup
... I think we should support that
surma: glitch is the new hotness like jsfiddle. but we can give it a docker container
<AutomatedTester> RRSAgent: make minutes
surma: there's a collaboration
feature, seems like a good tool we could utilize
... there will be some tests we can't develop there, but the
majority
... it's a good start for testharness.js or reftests
... and you can immediately see the results, play around, then
PR
zcorpan_: web-platform-tests.live has what's on master, not what's on master
marcosc: bless() will be hard to do when running tests manually
estes: [foolip couldn't scribe]
<Hexcles> foolip: for bless() specifically, we will be able to run it manually with jugglinmike's PR
marcosc: so you converted the tests from WebKit?
estes: yes, we have
window.internals that we used for web payments
... I copied manual tests for wpt, but don't know how to
upstream them.
... every browser has a different UI, so we'd need something
different
jgraham: the spec would have to say what the API should be for testing
<jose-cdbtr> join #tpac-chat
jgraham: not sensible for
everyone to have their own internal API that works
different
... ideally a WebDriver API so that authors can test it, but if
that doesn't make sense a test-only DOM API would be good
... but the spec needs to say what the APIs do
mgiuca: I think this applies to a
lot of new APIs
... do you mean the Web IDL should have a [ForTesting]
thing?
jgraham: yes, there's some
discussion around this
... WebKit folks said it's best if it all hangs off a single
objecting
mgiuca: but I'd like to be able to test this in Chrome stable
jgraham: it would depend on
browser policy, could have a pref for testing that can be
enabled
... WebKit doesn't want to compile it into the binary, so those
tests will fail on wpt.fyi unless we run webkit test runner
<Hexcles> foolip: no established pattern/recommendation yet
foolip: if there's a candidate for exploring this, please speak up
marcosc: there hasn't been much interest in payments yet
foolip: but people have shared
the tests
... and done the work to make them run
<gsnedders> foolip: [ForTesting] would be a reasonable thing to have in WebIDL, though it might not be ideal
mgiuca: maybe we could to web share (target)
bz: gecko has internal testing things
<mgiuca> +Present mgiuca
gsnedders: there's concern from Blink about the binary size cost of testing APIs
foolip: Blink concern has been with very large APIs like USB and Bluetooth
bz: there's also security concerns, some of these we really don't want to expose to userse
fremy: probably don't want a place to allow arbitrary payments :)
foolip: soliciting questions from
newcomers here
... we know PR review latency is a problem
gsnedders: what can we do?
marcosc: I think we should limit
the number of reviewers
... if you're listed and don't review within 2 days, that's
bad
lukebjerring: we don't know what
the policy change should be, but we'd like to do round robin
style
... from a list of people, you only assign one person
... if that person is assigned, they should either delegate or
take care of it
... removes the diffusion of responsibility
... we're hoping that will help
fremy: some people take vacation
marcosc: people should say they're on vacation. this is our job
jgraham: we could have an indicator
<jib> jib: q+
fremy: I'm assigned to a lot of
reviews
... I don't see any email
... I just use notifications
https://web-platform-tests.org/reviewing-tests/email.html
foolip: redirect your GitHub
email to work and filter per that documentation
... we'll try round robin first and some manual poking
zcorpan_: another idea was
notifying people in their own review systems
... could help with people who don't pay attention to
Github
jgraham: that might work for gecko and chromium, but not microsoft and apple
zcorpan_: we can talk to them
jib: I still write PRs and am not sure who can review
lukebjerring: with the round robin change it was suggested we also list all the reviewers
<jib> jib is Jan-Ivar, Mozilla
lukebjerring: for notifying people in their own review system, that would be on a case-by-case basis
fremy: CSS is interesting
... two people are editors for the vast majority of the
spec
... they'll have to either write or review tests
... we have a limited set of people, bottlenecks
foolip: on reviews, anyone who's in the reviewers team can review
lukebjerring: apart from reviews, any other reasons people aren't excited to use GitHub?
<Hexcles> foolip: one big problem is showing test results (regressions) in PRs
foolip: surprised that nobody has brought up the difficulty of understanding test results on PRs
fremy: the checkbox only tells
you if it's stable
... didn't think there would be a better way
bz: as a browser engineer, I
write tests and run on my browsers
... fixing that bug, I might as well check it into gecko and
get it upstreamed that way
... it would still be useful to run them cross-browser
... but can't run Edge locally
foolip: if we have results and regressions on PRs, then we could use the in-flight PRs to get cross-browser results
jgraham: we don't have in-flight
PRs for gecko yet
... haven't had that because of too many review systems, but
maybe we can do that now
bz: most useful would be the ability to submit a WIP test and get results
jgraham: that would probably be
possible
... you could do something with branches perhaps
foolip: I think that could work wit a special branch name pattern
jib: curious about how people feel that no test no commit policy is working
foolip: how have you found it in WebRTC?
jib: maybe more software rules
foolip: would you like a status check on spec PRs which fails if you don't link to tests?
jib: maybe
... also for new things you might not write tests
<jib> jib: ...until you have an implementation to write it with
foolip: for new features, I would
not recommend strictly requiring tests
... makes sense to file an issue and have the first implementer
write the tests
marcosc: we've written tests before implementation
foolip: and did that work out, or lots of broken tests?
marcosc: some bugs, sure
foolip: but not enough to make it a net negative to write tests up front?
marcosc: no. and spec quality
shot up
... when removing things in payments, we add it to
historical.html test
... the whole cycle really works, for us it works really
well
jib: are there limits for what we can add web-platform-tests for?
foolip: there are tentative tests https://web-platform-tests.org/writing-tests/file-names.html
zcorpan_: if there's just a single vendor interested, maybe they shouldn't be part of the test suite
mgiuca: I disagree with that, if
you're writing a spec you'd like others to implement the tests
will be helpful
... to not have to go digging around
jgraham: 2 cases
... 1. nobody is interested yet but you think they will
be
... 2. using a webkit-internal API
mgiuca: I mean (1)
zcorpan_: I mean if some browser vendor is actively opposed
marcosc: that's happened, with
performance.memory
... that could have had tests
<mgiuca> foolip: There's a risk if the tests live in one vendor repo that they may not be interested inupstreaming later.
<mgiuca> ... Little risk in keeping it in a shared repo. You can disable them until needed.
<gsnedders> RRSAgent: off
This is scribe.perl Revision: 1.154 of Date: 2018/09/25 16:35:56 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00) Succeeded: s/<gitbot>//G Succeeded: s/gitbot>//G Succeeded: s/gsnedders, r? https://github.com/web-platform-tests/wpt/pull/13624// FAILED: s|s///G|| Succeeded: s|/github.com/web-platform-tests/wpt/pull/13624///github.com/web-platform-tests/wpt/pull/13624|| Succeeded: s| (me@gsnedders.com)|| Succeeded: i/Topic/Scribe: gsnedders Succeeded: s/foolip does/foolip and lukebjerring do/ Succeeded: i/surma/Topic: Q&A Succeeded: i/Jake A/Scribe: foolip Succeeded: s/size/binary size cost/ Present: estes WARNING: Fewer than 3 people found for Present list! Found Scribe: gsnedders Inferring ScribeNick: gsnedders Found Scribe: foolip Inferring ScribeNick: foolip Scribes: gsnedders, foolip ScribeNicks: gsnedders, foolip WARNING: No meeting chair found! You should specify the meeting chair like this: <dbooth> Chair: dbooth WARNING: No date found! Assuming today. (Hint: Specify the W3C IRC log URL, and the date will be determined from that.) Or specify the date like this: <dbooth> Date: 12 Sep 2002 People with action items: WARNING: Input appears to use implicit continuation lines. You may need the "-implicitContinuations" option. WARNING: IRC log location not specified! (You can ignore this warning if you do not want the generated minutes to contain a link to the original IRC log.)[End of scribe.perl diagnostic output]