See also: IRC log
<MikeSmith> scribe: boazsender
<scribe> scribe: boazsender
miketaylr from moz on web compat
mikesmith from w3c, no testing wg, but mike is responsible for testing for w3c
<MikeSmith> scribenick: boazsender
<rbyers> https://www.chromium.org/blink/platform-predictability
Dominic from google working on blink
<MikeSmith> platform predictability
rick byers from google working on web predictibility initiative
Navid at google working on blink
john jansen from microsoft working on platform testing, layout, f12 and a11y, and OS testing.
Geoffrey on contract with Google trying to sort out the huge mess of testing
<MikeSmith> frank olivier
<JohnJansen> s/jahnsen/jansen
frank from microsoft edge team
<MikeSmith> s/jeffrey/geoffrey
<rbyers> s/jeffrey Jeffrey on contract with Google/Jeffrey on contract with Google/
shane working on a11y, aria
<zcorpan> Simon Pieters, Opera
simon from opera, edits specs html cssom, and writes web platform tests
james from mozilla, responsile for the most hated parts of the testing infra (testharness.js, wpt serve)
boaz sender from bocoup, test 262, v8 conformance
John: update on wpt at msft
john jansen: our team runs our internal test runner. we have a test runner, an analyzer and a documenter
the test runner runs about I think 50 different OS test suites
when we identify one we want to run, we fork it, bring it into our share and update it weekly
anything that uses testharness.js we can run for free
we run them, get the results, parse them out and moved into the analyzer where we store the results on in the DB
then a bot spins up, asks what branch/build are you run on, is it the same as yesterday
we have an algorithm for flagging flakey tests based on day to day performance to be evaluated
initially we ran the entire WPT repo without virtualizing the servers, so it would take 6 hrs, we'd run out of diskspace
it became very intensive, so we broke them out into multiple virtual test suites
we also run our tests in chrome for interop.
so, if we have a failure, we run it in chrome and if it also fails, then we deprioritize it
scribe: if it fails in all browsers, we say 'maybe ths is a conversation to have at the w3c'
and then we have a report
q: what is your process for new tests, asks rick from google
john jansen: when we decide to implement a feature...
our team is structured such that we have leads who are responsible for new features
we start by taking tests for that feature and running it another browser that has implemented it
ideally, as we implement, we write tests as we work on the feature, and the idea is that they will be contributed to WPT
but its hard, because its so much easier to write tests for internal tool chain and not WPT
at blink, test harness js is a huge step up
<scribe> new q from blink: if we're talking about implementing a feature, is the quality of WPT tests for the feature a factor
Jansen: no, customer feedback is the bigger driver
never seen a dev lead say no to a feature bc the test suite sucks
byers: can you share your test data
jansen: yea
... im a strong advocate for sharing info
still need to work through the noise of the data
we have a lot of data for 'x test case failing in y browser'
but we dont have the analysis of does the test case match the spec
also, test coverage across directories
simon: the useful data would be which tests are flaky
james graham: i can give you that list... its the lists of tests disabled in gecko
jansen: does mozilla publicly show the disabled tests
ok, now, heres what we do at mozilla
at moz for gecko, we run the entire test suite apart from the disabled tests in CI on every commit... ish
for tests that almost never fail, we run them on 1 in 10 commits
scribe: this is used for regression detection
we can see expected data files for x fails in y environment
we update our tests ocassionally
once every week or couple of weeks I kickoff the update process
I pull down the latest tests, run it in latests, use that to update the expectation files
when we downstream stuff we also upstream stuff, so it is currently possible to add tests as though it was internal, and it gets upstreamed
having that has significantly improved test contribution
we dont have any reporting about which tests are we failing
so ive been getting haranged by boris to report on which passing tests have started to fail
at the moment its up to individual developers to look at tests and keep them up to date
scribe: and it happens sometimes, but never for legacy features
or if they are I dont know, and theres no structured way to go about it
rick now on blink:
we've long run a subset of the tests
we dont yet have an automated upstreaming process
every week, or more, we do an import of a test, disable failing tests, and then those just form a list of tests failing
we are now running about 1800 of about 9k of the test files on every commit
otherwise it is ad hoc
*hoc
some teams have put a lot of effort into going back and looking at the tests for their features
some teams are writing tests as they go on new features (pointers, storage)
but its tough because there are multiple copies
scribe: we dont use WPT server
for all of our tests, like moz, we have expected results
we dont run in any other browsers
we run in a cut down browser called "content shell" with a bunch of flags turned on
so im curious, does edge test the bleading edge of chome or stable?
Jansen: stable
and teams, yeah so, my team was writing the service worker tests, and we started using testharness.js from the get go and it was great
web componants is doing that too
big problem is the massive legacy of tests we have (~30k files ??) that use our legacy runner
but the biggest problem is upstreaming our tests
having a separate repo is tough
rick again: many of the pointer events are manual, because there is no way to gen synthetic events
so we have a -manual convention in the file name to find them
<dcooney> Blink's flakiness data for imported test suites: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eimported%2F
we're working on a tool to make upstreaming easier, so that a blink dev can write tests with their product feature, get code review, and land it and then auto open a pull request automatically
james: we auto merge the branch... because ppl complain that PRs sit there for ever
q: how do you auto run manual tests
rick: for manual tests we have
special internal tools to automate them
... but eventually we want to move them to chrome driver, and
eventually webdriver
... *maybe*
... theres always going to be things that we cant get into
webdriver, or will take a long time
like in chrome, we have window.internals that lets you mod geo location, for example
also, we want to get to the point where features must have interop documentation, and that you need a test suite and info about how your tests are doing in other browsers, and that we could regect features that dont have that
right now, engineers just put their hand over their heart and say they did it
and we dont ask
because its too hard
jansen: if youre implementing tests for feature foo, and we are, youre not publishing your code upstream
rick: no, we submit PRs on commit
so canary features have tests
at apple we pull other browsers' tests to look at them
james: im not concerned about test duplications
hows the temprature for everyone?
*caucophony of temp convo*
rick: we're trying to figure out where is the biggest bang for the buck?
adding 5k tests is non trivial (because we have a bots running on all commits), so we want to add the good ones first
james: I failed to mention that
servo also uses these tests almost exlusively
... so they are writing tests for bits of the platform that are
behind the curve
mikesmith: that is pretty significant given that all the other browsers are so mature
\0/
sam from apple:
the webkit testing strategy has been historically one of regression testing. we write tests not to check for correctness, but more to check if something has gone wrong
samweinig: but it turns out correctness tests also help with regressions, so we've started importing other tests
we've started importing some WPT tests
we dont do this on any kind of regular basis, but people in that area tend to import as they work
we currently only run js and ref (automatic) tests
we skip manual tests
we use the test harness as much as we can
we've been trying to go back and fix dom/html errors in the past year
its been very fruitful for finding areas where the specs were wrong, or have changed, etc
in terms of upstreaming, we havent done that, but I think we should
im going to go back and see if people are interested in it
we have problems with perf
our tests are run on devs machines as they work, and then when you post a patch for review, our bots run them immediately and then again when the commit lands
in terms of what you all are talking about non standard way of interacting with the browser (eg webdriver type stuff) we have a few things we've done, like event sender
but as the browser has evovled we've moved away from that to a more async model (called something like async io), so now what we have is a pure js environment running in a separate process, so we take a sting of javascript that we can send to the ui process,
so if we were to standardize something looking to implement somehting like mouse events or scroll gestures, I think doing it in a way where you can separate the execution of the tests from the ui, and order the timing of them, that would be good.
dominicc: maybe we could do that with network stuff
jgraham: wpt serve is supposed to do that
samweinig: for us at apple its more about the ui thread
for what google mentioned with th geolocation api, or other promts for users... because there is a nead for the test harness to mach geolocation, maybe as this goes forward maybe we should look at what the entry points are so we can have unified interfaces for testing these things
<miketaylr> s/jamesS:/jgraham:/
many of our internal testing interfaces dont necissarily make sense for standards though
scribe: they test really internal
state
... that are not interoperable
<gsnedders> RRSAgent: make logs
with that in mind, it might be interesting for someone to go through in bits and see what all the browsers expose to their test harnesses, and see what we can standardize
certainly the prompts/alerts will be there
ok, so thats what webkit does
coffee break
<gsnedders> <br time=15m>
rick: I think we want to improve developers lives
James: here are the 2 infrastructure things that should be done next:
1) some sharable way of writing the manual tests in a cross browser way. the plan for that would be to write a web driver api in js, that is then injected into WPT repo somewhere, and then somewhere in JS you could write a test wich can call out to a web driver server, which does web driver stuff, finishes, and resolves a promise on the client.
<rbyers> boaz: defining the api is main thing, figure out when to go to the network etc. is separate
2) the second thing I think we want to work on is some way of getting insight into which tests are passing in specific implementations and which aren't
<zcorpan> $ find . -name "*-manual.html" | wc -l
<zcorpan> 333
<gsnedders> zcorpan: that's a sensible approach to get a number
<MikeSmith> it is not just about how many manual tests we have, because the problem is there are many requirements we have no tests for because nobody has or wants to write manual tests for them
<gsnedders> zcorpan: 888 manual tests
<gsnedders> zcorpan: many aren't HTML
the concern with number 2, is with vanity metrics
this should not be a marketing report
<zcorpan> i get 892 for "*-manual.*"
<gsnedders> zcorpan: I'm looking at what the manifest has in it
the great thing about WPT is that it's massive, crowd sourced, and its hard to game.
^ jansen says
Sam: the worry is that some browser is going to be most compatible, and will be "winning"
James: if theres a tool that allows you to show the latest results on a test directory resolution, then we dont telecast a competitive vanity metric
dcoony agrees that the front end shouldn't roll up the test results
dcooney: what I want at my desk, is here is the thing that will increase compat the most today.
a second vector for this central system of test passing is API consumption telemetry
dcooney: if we want to reduce developer pain, moving testing into a client side js web driver lib would help this.
<Zakim> MikeSmith, you wanted to comment
james: it should be easier to get new features into webdriver now
MikeSmith: we've been talking about the manual testing issue, we know how to solve it, but we dont have the budget to pay people to do it at the W3C, so the next step is solving this.
<scribe> ACTION: mike to follow up with interested parties to fund work on webdriver tests [recorded in http://www.w3.org/2016/09/22-testing-minutes.html#action01]
sam: it seems like theres basic
work we could do to try this
... it seems like theres basic work we could do to try this and
make a stawman.
MikeSmith: I admire that as a
strategy, but I think a more systematic strategy would work
better.
... I think return of investment should be high here.
Rick: I think we should invest in the central testing dashboard first.
<gsnedders> reftest: 483
<gsnedders> stub: 50
<gsnedders> wdspec: 2
<gsnedders> testharness: 7177
<gsnedders> manual: 888
<ShaneM> note that there is "manual" and manual. Some are semi-automated...
Rick: we have a team at google thinking of prototyping a dashboard like this.
<rbyers> https://docs.google.com/document/d/1ehJAwL03NM-ptSbw7rDUo_GkiaZBrxanMrvjjqtuqis/edit
<Zakim> david_wood, you wanted to suggest the relation to bug reports.
Domic: web devs would submit tests if there was a jsfiddle like interface
Domenic wants to write a tool to crawl metadata of tests and append results to margins of specs
james: we've tried this and it didint work
<Ms2ger> It works somewhat on https://xhr.spec.whatwg.org/
<Ms2ger> But it's a lot of work to maintain
<Zakim> ShaneM, you wanted to talk about metadata
james: maybe the api for this should be from the IDL side in the specs
<MikeSmith> Ms2ger, what works on https://xhr.spec.whatwg.org/ I don’t see anything in the margins or links to tests
<Ms2ger> Click on "Add links to tests from requirements (beta)"
<Zakim> gsnedders, you wanted to say something about the CSS tests harness dying
<Ms2ger> There's also some prior art from Philip`
<gsnedders> Ms2ger: any link anywhere?
dcooney: i have an issue when a spec has an algorithm with 20 steps or something, and want to know where the test is for step n.
<zcorpan_> https://lists.w3.org/Archives/Public/public-test-infra/
<Zakim> MikeSmith, you wanted to suggest we follow up the conversation on public-test-infra but from the high-level goals that rbyers described
<scribe> ACTION: MikeSmith to send dashboard outline to public-test-infra for feedback [recorded in http://www.w3.org/2016/09/22-testing-minutes.html#action02]
rick: proposing a new "Defacto" directory for tests for features with no specs, eg: all of hit-testing
james: no, i'd want a hit-testing dir
there is precedence for tests without specs
undecided
breaking for lunch
<zcorpan> http://test.csswg.org/suites/geometry-1_dev/nightly-unstable/html/chapter-6.htm
<miketaylr> HELLO
<miketaylr> thx
<JohnJansen> http://www.w3.org/2016/09/22-testing-minutes.html
<gsnedders> csswg-test:
<gsnedders> testharness: 439
<gsnedders> reftest: 10851
<gsnedders> wdspec: 0
<gsnedders> manual: 2240
<gsnedders> stub: 0
<gsnedders> visual: 3535
<MikeSmith> https://csswg-test.org/
<zcorpan> scribenick: zcorpan
<MikeSmith> including https://csswg-test.org/submissions/ where PR branches are mirrored
<JohnJansen> C:\Windows\System32\drivers\etc
gsnedders: the basic status quo
is that most of the policies that made it hard to run the css
testsuite in the same way as wpt
... are now gone
... most of the policies are now the same between the two
... some things are still different because the css build
system
... we have agreement to get rid of the build system
... the biggest thing that ppl are waiting on are a replacement
of the css test harness which shows results across
browsers
... most of the work has been about wpt-tools like manifest
generation
... to create an accurate list of test in csswg-test repo
... that's still a PR, not finished. 99.9% done, 99.9% left to
do
boazsender: what's the PR?
<gsnedders> https://github.com/w3c/wpt-tools/pull/90
gsnedders: the comments aren't
quite up to date
... the current css testsuite build system doesn't build the
test unless it has a <link rel=help> for something that
sheperd knows about
... lots of such tests
... once that PR is merged it should be able to make manifests
in the same way for the two repos
... and running the server and the lint tool
... all of the tooling is likely currently broken except the
build system
... the build system is the only thing that currently
works
... the big thing is reaching the point of where css is ready
to get rid of the build system
... so having results displayed for CR exit criteria
... once that is done it should be easy to merge the two
repos
JohnJansen: do they have the directory structure?
gsnedders: yes
JohnJansen: we make each folder
into a virtual server
... we don't want to have too many tests in one folder
gsnedders: only top-level?
JohnJansen: arbitrary
gsnedders: the output of the build system is not great
boazsender: the source files are separate
gsnedders: yeah
boazsender: how is html different? oh subdirectories
<ShaneM> I think it would be nice if there weren't more than two levels of depth just for test selection reasons (when using the manual interface)
JohnJansen: css puts them all into one dir because that's the tool output
boazsender: like visual tests, is box blue?
<ShaneM> that sounds relatively easy to fix FWIW
<gitbot> [13wpt-tools] 15Ms2ger created 06blacklist-slash (+1 new commit): 02https://github.com/w3c/wpt-tools/commit/aa41b36c734b
<gitbot> 13wpt-tools/06blacklist-slash 14aa41b36 15Geoffrey Sneddon: Remove the odd "/" special-case in is_blacklsited
gsnedders: number number visual tests vs reftests
<gsnedders> testharness: 439
<gsnedders> reftest: 10851
<gsnedders> wdspec: 0
<gsnedders> manual: 2240
<gsnedders> stub: 0
<gsnedders> visual: 3535
<gitbot> [13wpt-tools] 15Ms2ger 04force-pushed 06blacklist-slash from 14aa41b36 to 14af15ae4: 02https://github.com/w3c/wpt-tools/commits/blacklist-slash
<gitbot> 13wpt-tools/06blacklist-slash 14af15ae4 15Geoffrey Sneddon: Remove the odd "/" special-case in is_blacklisted
JohnJansen: my point is that if you get a failure you have to run it manually
<gitbot> [13wpt-tools] 15Ms2ger opened pull request #119: Remove the odd "/" special-case in is_blacklisted (06master...06blacklist-slash) 02https://github.com/w3c/wpt-tools/pull/119
boazsender: who runs them?
<jgraham> RRSAgent: make minutes
JohnJansen: it takes 2 days
... converted many of them to reftests
boazsender: what's the difference between manual and visual?
gsnedders: manual needs user interaction. visual is screenshottable
<gsnedders> Ms2ger: yes
gsnedders: multiple visual tests with same expected rendering can be compared together
boazsender: can they be automated?
JohnJansen: we have a bunch
automated internally
... a tool that was created before webdriver existed
... a lot of tests are about scrolling etc
... a lot of the tests are not best practices today
gsnedders: 2 things. a fair
number can be converted to reftests, which is better for
everyone
... visual tests get invalidated because font rendering changed
etc :-(((
... webkit and blink have infrastructure for visual but don't
like it
dcooney: blink does not like pixel test results. prefer reftests
boazsender: there are duplicate tests for some of these stuff in css, webkit, html, and so on
gsnedders: yeah
boazsender: webdriver... could it make sense to start over?
JohnJansen: almost
... the webkit layout tests can't be shared
... have proprietary ???
boazsender: if the browsers moved to webdriver we could replace these tests with new/rewritten tests
gsnedders: servo is implementing stuff and working on testing with wpt
<gitbot> [13wpt-tools] 15Ms2ger closed pull request #119: Remove the odd "/" special-case in is_blacklisted (06master...06blacklist-slash) 02https://github.com/w3c/wpt-tools/pull/119
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06blacklist-slash at 14af15ae4: 02https://github.com/w3c/wpt-tools/commit/af15ae4
jgraham: using wpt
infrastructure, it's good enough that it runs
... trying to create tests. historically a problem with
csswg-test is upstreaming is difficult because metadata
requirements, weird review system etc
... we couldn't implement the system in servo in a good
way
... once they have the same rules as wpt it will be much
simpler to upstream
... rewriting everything is a vast amount of work
... everything that is manual...
... manual tests need to be rewritten to use webdriver
typically
... for visual tests, need to get rid of those
... could write infrastructure for screenshot tests but it
would be crap because it's always crap
<gitbot> [13wpt-tools] 15Ms2ger created 06blacklist-cert (+1 new commit): 02https://github.com/w3c/wpt-tools/commit/ebfd192b8151
<gitbot> 13wpt-tools/06blacklist-cert 14ebfd192 15Geoffrey Sneddon: Make /_certs/ correctly blacklisted
jgraham: don't run anything
manual
... individual developers may do but not Test Days running
though css tests, nobody thinks that's useful
JohnJansen: also css3 stuff
gsnedders: tends to be far less shallow than 2.1
<gitbot> [13wpt-tools] 15Ms2ger opened pull request #120: Make /_certs/ correctly blacklisted (06master...06blacklist-cert) 02https://github.com/w3c/wpt-tools/pull/120
boazsender: csswg chair asked if we could write testharness.js tests for css
<ShaneM> jgraham: I understand. and I agree. But during test development or when re-running tests to ensure something is fixed...
boazsender: so they want to move in this direction
gsnedders: i'm well aware
... convincing them to do the sensible thing
jgraham: so status...
gsnedders: visual tests can split
into 2 categories
... first should be converted to reftests
... second don't have guaranteed rendering per spec, so can't
create a reference for them
... dashed borders for instance
... firefox is not consistent across display type
... display:table-cell is different from display:block
boazsender: on purpose?
gsnedders: dunno, legacy maybe
miketaylr: my blog relies on that
<boazsender> lol
gsnedders: we should convince csswg to define how to render these things
dcooney: should be a small set of things
gsnedders: dashed borders, dotted borders, outline, some other border styles... not much else
dcooney: could be impl specific, like UA style
zcorpan: HTML spec has UA
styleshset required
... it's normative for browsers
gsnedders: some things are not expressible in CSS
dcooney: may not be a big issue
gsnedders: it's not. there are
some visual tests we can't get rid of in a cross-browser way,
but can get rid of the majority
... big thing is test results which blocks getting rid of the
build system and hence moving to web-platform-tests
<gsnedders> http://test.csswg.org/harness/
gsnedders:
test.csswg.org/harness
... click on review results
<gsnedders> e.g., http://test.csswg.org/harness/results/compositing-1_dev/grouped/
boazsender: can be improved upon ^_^
JohnJansen: it's slow
gsnedders: they want some visualization like this of results
boazsender: how do you flip the switch?
gsnedders: not a problem
jgraham: the build system does a couple of things
<gitbot> [13wpt-tools] 15Ms2ger closed pull request #120: Make /_certs/ correctly blacklisted (06master...06blacklist-cert) 02https://github.com/w3c/wpt-tools/pull/120
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06blacklist-cert at 14ebfd192: 02https://github.com/w3c/wpt-tools/commit/ebfd192
jgraham: goes through and
generates metadata that can be consumed by things
... also write test in one format and generate
HTML/XHTML/Print/something else versions
boazsender: are people using the different formats now?
jgraham: seemed like a good idea at the time
gsnedders: back in the day print UAs didn't support HTML
boazsender: that can go away?
jgraham: it has to go away
... we won't import different versions
... so anything to discuss about that? (the merge)
... if someone(TM) will build the dash board
gsnedders: me
boazsender: this could be separate
JohnJansen: this was crowdsourced
because nobody would run the tests except My Guy
... he didn't want to share the results
<gsnedders> http://test.csswg.org/harness/details/compositing-1_dev/blending_in_a_group_with_opacity/engine/gecko/
JohnJansen: someone ran this in
Edge, somebody said it passed, somebody else said it
failed...
... ppl don't agree on a test passing or failing
zcorpan: had this problem at opera as well for visual tests
gsnedders: some of the differences in results can be explained by platform differences
boazsender: the visualization is difficult
JohnJansen: is anybody against you doing the port?
gsnedders: no
... may be discussion about some particular feature, but in
principle no
... want to do this next quarter
jgraham: is there a fallback position? merging the repos but allow existing toolings to continue working?
gsnedders: every file name is
unique...
... harder one is support file has to be in a support
directory...
jgraham: can deal with that in the manifest
gsnedders: more difficult cases with tests referring to files elsewhere
jgraham: how do they enforce that now?
gsnedders: HAHAHA
... lints for css directory, would you rather merge sooner and
then work on the build system?
jgraham: yeah, my preference
gsnedders: ok
jgraham: there's so much benefit
to merging
... if you wait on the dashboard then we're blocked for months
for it to be perfect with this and that feature
... if you just do it and then work on replacement for the
dashboard, we're not delayed
... could do one-off mapping from old path to new path
etc
... these tests are the worst. this one requires knowledge of
PorterDuff operations -_-
JohnJansen: they're all terrible
<jgraham> http://test.csswg.org/harness/test/compositing-1_dev/single/line-with-svg-background/format/html5/
;_;
boazsender: where's the pass condition?
jgraham: view source
gsnedders: this one is from test the web forward
<gitbot> [13web-platform-tests] 15sideshowbarker closed pull request #3758: Add EventHandler stub to interfaces.html (06master...06patch-2) 02https://github.com/w3c/web-platform-tests/pull/3758
<br>
<gitbot> [13wpt-tools] 15Ms2ger created 06blacklist-resources (+1 new commit): 02https://github.com/w3c/wpt-tools/commit/06477b63a6cd
<gitbot> 13wpt-tools/06blacklist-resources 1406477b6 15Geoffrey Sneddon: Blacklist /support/ and /resources/ anywhere in the URL
<gitbot> [13wpt-tools] 15Ms2ger opened pull request #121: Blacklist /support/ and /resources/ anywhere in the URL (06master...06blacklist-resources) 02https://github.com/w3c/wpt-tools/pull/121
<gitbot> [13wpt-tools] 15Ms2ger closed pull request #121: Blacklist /support/ and /resources/ anywhere in the URL (06master...06blacklist-resources) 02https://github.com/w3c/wpt-tools/pull/121
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06blacklist-resources at 1406477b6: 02https://github.com/w3c/wpt-tools/commit/06477b6
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06wptserve at 1403ebf8d: 02https://github.com/w3c/wpt-tools/commit/03ebf8d
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06fix_localpaths at 14ec892f5: 02https://github.com/w3c/wpt-tools/commit/ec892f5
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06lint_json_output at 14b2c4e28: 02https://github.com/w3c/wpt-tools/commit/b2c4e28
<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06py3-lint2 at 14814311e: 02https://github.com/w3c/wpt-tools/commit/814311e
<gitbot> [13web-platform-tests] 15mwatson2 opened pull request #3798: [eme] Add test for expiration attribute value (06master...06eme-jwt) 02https://github.com/w3c/web-platform-tests/pull/3798
nikos: there was an old plan nfor
developing a SVG testsuite
... we're going to host in wpt but still use sheperd with
metadata etc
... we were planning on importing SVG 1.0 tests
... lots of text has changed between the versions
... looking for feedback and suggestsions what we should do
<JohnJansen> clarification: svg looking for advice
nikos: we've gotten advice on
github
... the old plan was old-school
<nikos> https://github.com/w3c/svgwg/issues/269
nikos: Shepherd is out
obviously
... we have a requirement that the testsuite needs to be useful
for non-browser implementations
... reftests be separate from script tests
gsnedders: that can be done via manifest
nikos: ok as long as we can separate for one impl that's fine
jgraham: the idea for html is
that we would take every section in the spec up to 3 levels
deep and create a directory
... and then fill in tests where we didn't have coverage
... didn't really help enough
... having some structure is good
... using IDs in the spec is of less value
nikos: things move around...
jgraham: not always
understandable names
... pick a structure that makes sense to you according to the
stucture of the spec
... but dont' worry too muhc about it
... don't dump all tests into one dir
zcorpan: if you don't want to use <link rel=help> then the last dir should match up to an id in the spec
nikos: reftests it's obvious what passes and what fails
jgraham: write something that's
easy to write a ref for
... if you're testing text rendering it doesn't have to be a
square...
... generally do something sensething
zcorpan: you want to share reference as much as possible
nikos: idea to use PNG e.g. for a rectangle
jgraham: should be a visual test, but minimize such tests as much as possible
JohnJansen: the fundamental stuff you don't need to test it, you'll find out if it's broken anyway
nikos: approving tests in the repo, can people be given access to do that?
gsnedders: file PRs, if they don't do stupid things we give them push access
nikos: do SVG WG people approve SVG tests?
gsnedders: vauge rules. if you
think you're competent to review, go ahead and do it
... PRs lie around for months...
... until you get into complex SVG stuff people will be able to
review
JohnJansen: I though more people would be interested in reviewing
nikos: trying to recruit people
into doing testing
... people dropped off when we got closer to CR
... any recommendations?
JohnJansen: be open if you're being told you're doing it wrong
nikos: i'm trying to gather information...
JohnJansen: we've discovererd the
hard way if you hire people to write tests you get low quality
tests, which is more costly in the long run than just writing
good tests yourself from the start
... test the web forward is an example, we got people in a room
to write tests...
MikeSmith: culture for
recommendations, we spend time in IRC channel and do very
little other interaction
... if someone will contribute, join IRC
... we ahve good time coverage across timezones
<MikeSmith> zcorpan: a lot of things can be tested with testharness.js that you might not think is suitable
<MikeSmith> ... like, a lot of rendering stuff
<MikeSmith> ... if you prioritize UAs that support CSS and Javascript
<MikeSmith> nikos: Inkscape... do not not to the details
<scribe> scribenick: MikeSmith
boazsender: bet it’s not JavaScript
nikos: it’s not yeah
JohnJansen: in some cases you
might want to run all your tests in your desktop browser and
mobile
... and you may find that you have never found a bug that was
in one place and not the other
... so the question is if Inkscape is different enough to merit
running the tests in it vs running just a browsers
nikos: it is different enough
zcorpan: so, what kind of tests would work in Inkscape
nikos: reftests
... I need to talk Tav (Inkspace rep) to found out more
This is scribe.perl Revision: 1.144 of Date: 2015/11/17 08:39:34 Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/ Guessing input format: RRSAgent_Text_Format (score 1.00) Succeeded: s/diminic/Dominic/ Succeeded: s/nab/Navid/ Succeeded: s/jahnsen/jansen/ FAILED: s/jahnsen/jansen/ Succeeded: s/jeffrey/Geoffrey/ FAILED: s/jeffrey/geoffrey/ FAILED: s/jeffrey from google/Jeffrey on contract with Google/ Succeeded: s/from google/Jeffrey on contract with Google/ Succeeded: s/html css/html cssom/ Succeeded: s/Geoffrey from google/Geoffrey on contract with Google/ Succeeded: i/miketaylr from moz/Topic: Intros Succeeded: i/update on wpt at msft/Topic: Update on wpt at Microsoft Succeeded: s/hox/hoc/ Succeeded: i/what we do at mozilla/Topic: Update on wpt at Mozilla Succeeded: i/rick now on blink/Topic: Update on wpt and Blink Succeeded: i/sam from apple/Topic: Update on wpt and WebKit Succeeded: s/but it/samweinig: but it/ Succeeded: s/maybe/dominicc: maybe/ Succeeded: s/for us/samweinig: for us/ FAILED: s/jamesS:/jgraham:/ Succeeded: s/jameS:/jgraham:/ Succeeded: s/stragey/strategy/ Succeeded: s/dominic/Domenic/ Succeeded: s/Topic: CSS WG merger into WPT// Succeeded: s/paltform/platform/ Succeeded: s/teh current/the current/ Succeeded: s/all the tooling should work/all of the tooling is likely currently broken except the build system/ Succeeded: s/proparitary/proprietary/ Succeeded: s/tends to be shallow/tends to be far less shallow than 2.1/ Succeeded: s/pla/plan/ Found Scribe: boazsender Inferring ScribeNick: boazsender Found Scribe: boazsender Inferring ScribeNick: boazsender Found ScribeNick: boazsender Found ScribeNick: zcorpan Found ScribeNick: MikeSmith ScribeNicks: boazsender, zcorpan, MikeSmith Present: Boaz DomenicDenicola DominicCooney FrankOlivier JohnJansen MikeSmith MikeTaylor NavidZolghadr SamWeinig SimonPieters TessOConnor gsnedders jgraham rbyers nikos WARNING: No meeting title found! You should specify the meeting title like this: <dbooth> Meeting: Weekly Baking Club Meeting WARNING: No meeting chair found! You should specify the meeting chair like this: <dbooth> Chair: dbooth Got date from IRC log name: 22 Sep 2016 Guessing minutes URL: http://www.w3.org/2016/09/22-testing-minutes.html People with action items: mike mikesmith[End of scribe.perl diagnostic output]