W3C

- DRAFT -

SV_MEETING_TITLE

22 Sep 2016

See also: IRC log

Attendees

Present
Boaz, DomenicDenicola, DominicCooney, FrankOlivier, JohnJansen, MikeSmith, MikeTaylor, NavidZolghadr, SamWeinig, SimonPieters, TessOConnor, gsnedders, jgraham, rbyers, nikos
Regrets
Chair
SV_MEETING_CHAIR
Scribe
boazsender

Contents


<MikeSmith> scribe: boazsender

<scribe> scribe: boazsender

Intros

miketaylr from moz on web compat

mikesmith from w3c, no testing wg, but mike is responsible for testing for w3c

<MikeSmith> scribenick: boazsender

<rbyers> https://www.chromium.org/blink/platform-predictability

Dominic from google working on blink

<MikeSmith> platform predictability

rick byers from google working on web predictibility initiative

Navid at google working on blink

john jansen from microsoft working on platform testing, layout, f12 and a11y, and OS testing.

Geoffrey on contract with Google trying to sort out the huge mess of testing

<MikeSmith> frank olivier

<JohnJansen> s/jahnsen/jansen

frank from microsoft edge team

<MikeSmith> s/jeffrey/geoffrey

<rbyers> s/jeffrey Jeffrey on contract with Google/Jeffrey on contract with Google/

shane working on a11y, aria

<zcorpan> Simon Pieters, Opera

simon from opera, edits specs html cssom, and writes web platform tests

james from mozilla, responsile for the most hated parts of the testing infra (testharness.js, wpt serve)

boaz sender from bocoup, test 262, v8 conformance

Update on wpt at Microsoft

John: update on wpt at msft

john jansen: our team runs our internal test runner. we have a test runner, an analyzer and a documenter

the test runner runs about I think 50 different OS test suites

when we identify one we want to run, we fork it, bring it into our share and update it weekly

anything that uses testharness.js we can run for free

we run them, get the results, parse them out and moved into the analyzer where we store the results on in the DB

then a bot spins up, asks what branch/build are you run on, is it the same as yesterday

we have an algorithm for flagging flakey tests based on day to day performance to be evaluated

initially we ran the entire WPT repo without virtualizing the servers, so it would take 6 hrs, we'd run out of diskspace

it became very intensive, so we broke them out into multiple virtual test suites

we also run our tests in chrome for interop.

so, if we have a failure, we run it in chrome and if it also fails, then we deprioritize it

scribe: if it fails in all browsers, we say 'maybe ths is a conversation to have at the w3c'

and then we have a report

q: what is your process for new tests, asks rick from google

john jansen: when we decide to implement a feature...

our team is structured such that we have leads who are responsible for new features

we start by taking tests for that feature and running it another browser that has implemented it

ideally, as we implement, we write tests as we work on the feature, and the idea is that they will be contributed to WPT

but its hard, because its so much easier to write tests for internal tool chain and not WPT

at blink, test harness js is a huge step up

<scribe> new q from blink: if we're talking about implementing a feature, is the quality of WPT tests for the feature a factor

Jansen: no, customer feedback is the bigger driver

never seen a dev lead say no to a feature bc the test suite sucks

byers: can you share your test data

jansen: yea
... im a strong advocate for sharing info

still need to work through the noise of the data

we have a lot of data for 'x test case failing in y browser'

but we dont have the analysis of does the test case match the spec

also, test coverage across directories

simon: the useful data would be which tests are flaky

james graham: i can give you that list... its the lists of tests disabled in gecko

jansen: does mozilla publicly show the disabled tests

Update on wpt at Mozilla

ok, now, heres what we do at mozilla

at moz for gecko, we run the entire test suite apart from the disabled tests in CI on every commit... ish

for tests that almost never fail, we run them on 1 in 10 commits

scribe: this is used for regression detection

we can see expected data files for x fails in y environment

we update our tests ocassionally

once every week or couple of weeks I kickoff the update process

I pull down the latest tests, run it in latests, use that to update the expectation files

when we downstream stuff we also upstream stuff, so it is currently possible to add tests as though it was internal, and it gets upstreamed

having that has significantly improved test contribution

we dont have any reporting about which tests are we failing

so ive been getting haranged by boris to report on which passing tests have started to fail

at the moment its up to individual developers to look at tests and keep them up to date

scribe: and it happens sometimes, but never for legacy features

or if they are I dont know, and theres no structured way to go about it

Update on wpt and Blink

rick now on blink:

we've long run a subset of the tests

we dont yet have an automated upstreaming process

every week, or more, we do an import of a test, disable failing tests, and then those just form a list of tests failing

we are now running about 1800 of about 9k of the test files on every commit

otherwise it is ad hoc

*hoc

some teams have put a lot of effort into going back and looking at the tests for their features

some teams are writing tests as they go on new features (pointers, storage)

but its tough because there are multiple copies

scribe: we dont use WPT server

for all of our tests, like moz, we have expected results

we dont run in any other browsers

we run in a cut down browser called "content shell" with a bunch of flags turned on

so im curious, does edge test the bleading edge of chome or stable?

Jansen: stable

and teams, yeah so, my team was writing the service worker tests, and we started using testharness.js from the get go and it was great

web componants is doing that too

big problem is the massive legacy of tests we have (~30k files ??) that use our legacy runner

but the biggest problem is upstreaming our tests

having a separate repo is tough

rick again: many of the pointer events are manual, because there is no way to gen synthetic events

so we have a -manual convention in the file name to find them

<dcooney> Blink's flakiness data for imported test suites: http://test-results.appspot.com/dashboards/flakiness_dashboard.html#tests=%5Eimported%2F

we're working on a tool to make upstreaming easier, so that a blink dev can write tests with their product feature, get code review, and land it and then auto open a pull request automatically

james: we auto merge the branch... because ppl complain that PRs sit there for ever

q: how do you auto run manual tests

rick: for manual tests we have special internal tools to automate them
... but eventually we want to move them to chrome driver, and eventually webdriver
... *maybe*
... theres always going to be things that we cant get into webdriver, or will take a long time

like in chrome, we have window.internals that lets you mod geo location, for example

also, we want to get to the point where features must have interop documentation, and that you need a test suite and info about how your tests are doing in other browsers, and that we could regect features that dont have that

right now, engineers just put their hand over their heart and say they did it

and we dont ask

because its too hard

jansen: if youre implementing tests for feature foo, and we are, youre not publishing your code upstream

rick: no, we submit PRs on commit

so canary features have tests

at apple we pull other browsers' tests to look at them

james: im not concerned about test duplications

hows the temprature for everyone?

*caucophony of temp convo*

rick: we're trying to figure out where is the biggest bang for the buck?

adding 5k tests is non trivial (because we have a bots running on all commits), so we want to add the good ones first

james: I failed to mention that servo also uses these tests almost exlusively
... so they are writing tests for bits of the platform that are behind the curve

mikesmith: that is pretty significant given that all the other browsers are so mature

\0/

Update on wpt and WebKit

sam from apple:

the webkit testing strategy has been historically one of regression testing. we write tests not to check for correctness, but more to check if something has gone wrong

samweinig: but it turns out correctness tests also help with regressions, so we've started importing other tests

we've started importing some WPT tests

we dont do this on any kind of regular basis, but people in that area tend to import as they work

we currently only run js and ref (automatic) tests

we skip manual tests

we use the test harness as much as we can

we've been trying to go back and fix dom/html errors in the past year

its been very fruitful for finding areas where the specs were wrong, or have changed, etc

in terms of upstreaming, we havent done that, but I think we should

im going to go back and see if people are interested in it

we have problems with perf

our tests are run on devs machines as they work, and then when you post a patch for review, our bots run them immediately and then again when the commit lands

in terms of what you all are talking about non standard way of interacting with the browser (eg webdriver type stuff) we have a few things we've done, like event sender

but as the browser has evovled we've moved away from that to a more async model (called something like async io), so now what we have is a pure js environment running in a separate process, so we take a sting of javascript that we can send to the ui process,

so if we were to standardize something looking to implement somehting like mouse events or scroll gestures, I think doing it in a way where you can separate the execution of the tests from the ui, and order the timing of them, that would be good.

dominicc: maybe we could do that with network stuff

jgraham: wpt serve is supposed to do that

samweinig: for us at apple its more about the ui thread

for what google mentioned with th geolocation api, or other promts for users... because there is a nead for the test harness to mach geolocation, maybe as this goes forward maybe we should look at what the entry points are so we can have unified interfaces for testing these things

<miketaylr> s/jamesS:/jgraham:/

many of our internal testing interfaces dont necissarily make sense for standards though

scribe: they test really internal state
... that are not interoperable

<gsnedders> RRSAgent: make logs

with that in mind, it might be interesting for someone to go through in bits and see what all the browsers expose to their test harnesses, and see what we can standardize

certainly the prompts/alerts will be there

ok, so thats what webkit does

coffee break

<gsnedders> <br time=15m>

what should we invest in?

rick: I think we want to improve developers lives

James: here are the 2 infrastructure things that should be done next:

1) some sharable way of writing the manual tests in a cross browser way. the plan for that would be to write a web driver api in js, that is then injected into WPT repo somewhere, and then somewhere in JS you could write a test wich can call out to a web driver server, which does web driver stuff, finishes, and resolves a promise on the client.

<rbyers> boaz: defining the api is main thing, figure out when to go to the network etc. is separate

2) the second thing I think we want to work on is some way of getting insight into which tests are passing in specific implementations and which aren't

<zcorpan> $ find . -name "*-manual.html" | wc -l

<zcorpan> 333

<gsnedders> zcorpan: that's a sensible approach to get a number

<MikeSmith> it is not just about how many manual tests we have, because the problem is there are many requirements we have no tests for because nobody has or wants to write manual tests for them

<gsnedders> zcorpan: 888 manual tests

<gsnedders> zcorpan: many aren't HTML

the concern with number 2, is with vanity metrics

this should not be a marketing report

<zcorpan> i get 892 for "*-manual.*"

<gsnedders> zcorpan: I'm looking at what the manifest has in it

the great thing about WPT is that it's massive, crowd sourced, and its hard to game.

^ jansen says

Sam: the worry is that some browser is going to be most compatible, and will be "winning"

James: if theres a tool that allows you to show the latest results on a test directory resolution, then we dont telecast a competitive vanity metric

dcoony agrees that the front end shouldn't roll up the test results

dcooney: what I want at my desk, is here is the thing that will increase compat the most today.

a second vector for this central system of test passing is API consumption telemetry

dcooney: if we want to reduce developer pain, moving testing into a client side js web driver lib would help this.

<Zakim> MikeSmith, you wanted to comment

james: it should be easier to get new features into webdriver now

MikeSmith: we've been talking about the manual testing issue, we know how to solve it, but we dont have the budget to pay people to do it at the W3C, so the next step is solving this.

<scribe> ACTION: mike to follow up with interested parties to fund work on webdriver tests [recorded in http://www.w3.org/2016/09/22-testing-minutes.html#action01]

sam: it seems like theres basic work we could do to try this
... it seems like theres basic work we could do to try this and make a stawman.

MikeSmith: I admire that as a strategy, but I think a more systematic strategy would work better.
... I think return of investment should be high here.

Rick: I think we should invest in the central testing dashboard first.

<gsnedders> reftest: 483

<gsnedders> stub: 50

<gsnedders> wdspec: 2

<gsnedders> testharness: 7177

<gsnedders> manual: 888

<ShaneM> note that there is "manual" and manual. Some are semi-automated...

Rick: we have a team at google thinking of prototyping a dashboard like this.

MVP reqs for the central WPT dashboard

<rbyers> https://docs.google.com/document/d/1ehJAwL03NM-ptSbw7rDUo_GkiaZBrxanMrvjjqtuqis/edit

<Zakim> david_wood, you wanted to suggest the relation to bug reports.

Domic: web devs would submit tests if there was a jsfiddle like interface

Domenic wants to write a tool to crawl metadata of tests and append results to margins of specs

james: we've tried this and it didint work

<Ms2ger> It works somewhat on https://xhr.spec.whatwg.org/

<Ms2ger> But it's a lot of work to maintain

<Zakim> ShaneM, you wanted to talk about metadata

james: maybe the api for this should be from the IDL side in the specs

<MikeSmith> Ms2ger, what works on https://xhr.spec.whatwg.org/ I don’t see anything in the margins or links to tests

<Ms2ger> Click on "Add links to tests from requirements (beta)"

<Zakim> gsnedders, you wanted to say something about the CSS tests harness dying

<Ms2ger> There's also some prior art from Philip`

<gsnedders> Ms2ger: any link anywhere?

dcooney: i have an issue when a spec has an algorithm with 20 steps or something, and want to know where the test is for step n.

<zcorpan_> https://lists.w3.org/Archives/Public/public-test-infra/

<Zakim> MikeSmith, you wanted to suggest we follow up the conversation on public-test-infra but from the high-level goals that rbyers described

<scribe> ACTION: MikeSmith to send dashboard outline to public-test-infra for feedback [recorded in http://www.w3.org/2016/09/22-testing-minutes.html#action02]

rick: proposing a new "Defacto" directory for tests for features with no specs, eg: all of hit-testing

james: no, i'd want a hit-testing dir

there is precedence for tests without specs

undecided

breaking for lunch

<zcorpan> http://test.csswg.org/suites/geometry-1_dev/nightly-unstable/html/chapter-6.htm

<miketaylr> HELLO

<miketaylr> thx

<JohnJansen> http://www.w3.org/2016/09/22-testing-minutes.html

<gsnedders> csswg-test:

<gsnedders> testharness: 439

<gsnedders> reftest: 10851

<gsnedders> wdspec: 0

<gsnedders> manual: 2240

<gsnedders> stub: 0

<gsnedders> visual: 3535

<MikeSmith> https://csswg-test.org/

<zcorpan> scribenick: zcorpan

<MikeSmith> including https://csswg-test.org/submissions/ where PR branches are mirrored

<JohnJansen> C:\Windows\System32\drivers\etc

merging csswg-test and web-platform-tests

gsnedders: the basic status quo is that most of the policies that made it hard to run the css testsuite in the same way as wpt
... are now gone
... most of the policies are now the same between the two
... some things are still different because the css build system
... we have agreement to get rid of the build system
... the biggest thing that ppl are waiting on are a replacement of the css test harness which shows results across browsers
... most of the work has been about wpt-tools like manifest generation
... to create an accurate list of test in csswg-test repo
... that's still a PR, not finished. 99.9% done, 99.9% left to do

boazsender: what's the PR?

<gsnedders> https://github.com/w3c/wpt-tools/pull/90

gsnedders: the comments aren't quite up to date
... the current css testsuite build system doesn't build the test unless it has a <link rel=help> for something that sheperd knows about
... lots of such tests
... once that PR is merged it should be able to make manifests in the same way for the two repos
... and running the server and the lint tool
... all of the tooling is likely currently broken except the build system
... the build system is the only thing that currently works
... the big thing is reaching the point of where css is ready to get rid of the build system
... so having results displayed for CR exit criteria
... once that is done it should be easy to merge the two repos

JohnJansen: do they have the directory structure?

gsnedders: yes

JohnJansen: we make each folder into a virtual server
... we don't want to have too many tests in one folder

gsnedders: only top-level?

JohnJansen: arbitrary

gsnedders: the output of the build system is not great

boazsender: the source files are separate

gsnedders: yeah

boazsender: how is html different? oh subdirectories

<ShaneM> I think it would be nice if there weren't more than two levels of depth just for test selection reasons (when using the manual interface)

JohnJansen: css puts them all into one dir because that's the tool output

boazsender: like visual tests, is box blue?

<ShaneM> that sounds relatively easy to fix FWIW

<gitbot> [13wpt-tools] 15Ms2ger created 06blacklist-slash (+1 new commit): 02https://github.com/w3c/wpt-tools/commit/aa41b36c734b

<gitbot> 13wpt-tools/06blacklist-slash 14aa41b36 15Geoffrey Sneddon: Remove the odd "/" special-case in is_blacklsited

gsnedders: number number visual tests vs reftests

<gsnedders> testharness: 439

<gsnedders> reftest: 10851

<gsnedders> wdspec: 0

<gsnedders> manual: 2240

<gsnedders> stub: 0

<gsnedders> visual: 3535

<gitbot> [13wpt-tools] 15Ms2ger 04force-pushed 06blacklist-slash from 14aa41b36 to 14af15ae4: 02https://github.com/w3c/wpt-tools/commits/blacklist-slash

<gitbot> 13wpt-tools/06blacklist-slash 14af15ae4 15Geoffrey Sneddon: Remove the odd "/" special-case in is_blacklisted

JohnJansen: my point is that if you get a failure you have to run it manually

<gitbot> [13wpt-tools] 15Ms2ger opened pull request #119: Remove the odd "/" special-case in is_blacklisted (06master...06blacklist-slash) 02https://github.com/w3c/wpt-tools/pull/119

boazsender: who runs them?

<jgraham> RRSAgent: make minutes

JohnJansen: it takes 2 days
... converted many of them to reftests

boazsender: what's the difference between manual and visual?

gsnedders: manual needs user interaction. visual is screenshottable

<gsnedders> Ms2ger: yes

gsnedders: multiple visual tests with same expected rendering can be compared together

boazsender: can they be automated?

JohnJansen: we have a bunch automated internally
... a tool that was created before webdriver existed
... a lot of tests are about scrolling etc
... a lot of the tests are not best practices today

gsnedders: 2 things. a fair number can be converted to reftests, which is better for everyone
... visual tests get invalidated because font rendering changed etc :-(((
... webkit and blink have infrastructure for visual but don't like it

dcooney: blink does not like pixel test results. prefer reftests

boazsender: there are duplicate tests for some of these stuff in css, webkit, html, and so on

gsnedders: yeah

boazsender: webdriver... could it make sense to start over?

JohnJansen: almost
... the webkit layout tests can't be shared
... have proprietary ???

boazsender: if the browsers moved to webdriver we could replace these tests with new/rewritten tests

gsnedders: servo is implementing stuff and working on testing with wpt

<gitbot> [13wpt-tools] 15Ms2ger closed pull request #119: Remove the odd "/" special-case in is_blacklisted (06master...06blacklist-slash) 02https://github.com/w3c/wpt-tools/pull/119

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06blacklist-slash at 14af15ae4: 02https://github.com/w3c/wpt-tools/commit/af15ae4

jgraham: using wpt infrastructure, it's good enough that it runs
... trying to create tests. historically a problem with csswg-test is upstreaming is difficult because metadata requirements, weird review system etc
... we couldn't implement the system in servo in a good way
... once they have the same rules as wpt it will be much simpler to upstream
... rewriting everything is a vast amount of work
... everything that is manual...
... manual tests need to be rewritten to use webdriver typically
... for visual tests, need to get rid of those
... could write infrastructure for screenshot tests but it would be crap because it's always crap

<gitbot> [13wpt-tools] 15Ms2ger created 06blacklist-cert (+1 new commit): 02https://github.com/w3c/wpt-tools/commit/ebfd192b8151

<gitbot> 13wpt-tools/06blacklist-cert 14ebfd192 15Geoffrey Sneddon: Make /_certs/ correctly blacklisted

jgraham: don't run anything manual
... individual developers may do but not Test Days running though css tests, nobody thinks that's useful

JohnJansen: also css3 stuff

gsnedders: tends to be far less shallow than 2.1

<gitbot> [13wpt-tools] 15Ms2ger opened pull request #120: Make /_certs/ correctly blacklisted (06master...06blacklist-cert) 02https://github.com/w3c/wpt-tools/pull/120

boazsender: csswg chair asked if we could write testharness.js tests for css

<ShaneM> jgraham: I understand. and I agree. But during test development or when re-running tests to ensure something is fixed...

boazsender: so they want to move in this direction

gsnedders: i'm well aware
... convincing them to do the sensible thing

jgraham: so status...

gsnedders: visual tests can split into 2 categories
... first should be converted to reftests
... second don't have guaranteed rendering per spec, so can't create a reference for them
... dashed borders for instance
... firefox is not consistent across display type
... display:table-cell is different from display:block

boazsender: on purpose?

gsnedders: dunno, legacy maybe

miketaylr: my blog relies on that

<boazsender> lol

gsnedders: we should convince csswg to define how to render these things

dcooney: should be a small set of things

gsnedders: dashed borders, dotted borders, outline, some other border styles... not much else

dcooney: could be impl specific, like UA style

zcorpan: HTML spec has UA styleshset required
... it's normative for browsers

gsnedders: some things are not expressible in CSS

dcooney: may not be a big issue

gsnedders: it's not. there are some visual tests we can't get rid of in a cross-browser way, but can get rid of the majority
... big thing is test results which blocks getting rid of the build system and hence moving to web-platform-tests

<gsnedders> http://test.csswg.org/harness/

gsnedders: test.csswg.org/harness
... click on review results

<gsnedders> e.g., http://test.csswg.org/harness/results/compositing-1_dev/grouped/

boazsender: can be improved upon ^_^

JohnJansen: it's slow

gsnedders: they want some visualization like this of results

boazsender: how do you flip the switch?

gsnedders: not a problem

jgraham: the build system does a couple of things

<gitbot> [13wpt-tools] 15Ms2ger closed pull request #120: Make /_certs/ correctly blacklisted (06master...06blacklist-cert) 02https://github.com/w3c/wpt-tools/pull/120

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06blacklist-cert at 14ebfd192: 02https://github.com/w3c/wpt-tools/commit/ebfd192

jgraham: goes through and generates metadata that can be consumed by things
... also write test in one format and generate HTML/XHTML/Print/something else versions

boazsender: are people using the different formats now?

jgraham: seemed like a good idea at the time

gsnedders: back in the day print UAs didn't support HTML

boazsender: that can go away?

jgraham: it has to go away
... we won't import different versions
... so anything to discuss about that? (the merge)
... if someone(TM) will build the dash board

gsnedders: me

boazsender: this could be separate

JohnJansen: this was crowdsourced because nobody would run the tests except My Guy
... he didn't want to share the results

<gsnedders> http://test.csswg.org/harness/details/compositing-1_dev/blending_in_a_group_with_opacity/engine/gecko/

JohnJansen: someone ran this in Edge, somebody said it passed, somebody else said it failed...
... ppl don't agree on a test passing or failing

zcorpan: had this problem at opera as well for visual tests

gsnedders: some of the differences in results can be explained by platform differences

boazsender: the visualization is difficult

JohnJansen: is anybody against you doing the port?

gsnedders: no
... may be discussion about some particular feature, but in principle no
... want to do this next quarter

jgraham: is there a fallback position? merging the repos but allow existing toolings to continue working?

gsnedders: every file name is unique...
... harder one is support file has to be in a support directory...

jgraham: can deal with that in the manifest

gsnedders: more difficult cases with tests referring to files elsewhere

jgraham: how do they enforce that now?

gsnedders: HAHAHA
... lints for css directory, would you rather merge sooner and then work on the build system?

jgraham: yeah, my preference

gsnedders: ok

jgraham: there's so much benefit to merging
... if you wait on the dashboard then we're blocked for months for it to be perfect with this and that feature
... if you just do it and then work on replacement for the dashboard, we're not delayed
... could do one-off mapping from old path to new path etc
... these tests are the worst. this one requires knowledge of PorterDuff operations -_-

JohnJansen: they're all terrible

<jgraham> http://test.csswg.org/harness/test/compositing-1_dev/single/line-with-svg-background/format/html5/

;_;

boazsender: where's the pass condition?

jgraham: view source

gsnedders: this one is from test the web forward

<gitbot> [13web-platform-tests] 15sideshowbarker closed pull request #3758: Add EventHandler stub to interfaces.html (06master...06patch-2) 02https://github.com/w3c/web-platform-tests/pull/3758

<br>

<gitbot> [13wpt-tools] 15Ms2ger created 06blacklist-resources (+1 new commit): 02https://github.com/w3c/wpt-tools/commit/06477b63a6cd

<gitbot> 13wpt-tools/06blacklist-resources 1406477b6 15Geoffrey Sneddon: Blacklist /support/ and /resources/ anywhere in the URL

<gitbot> [13wpt-tools] 15Ms2ger opened pull request #121: Blacklist /support/ and /resources/ anywhere in the URL (06master...06blacklist-resources) 02https://github.com/w3c/wpt-tools/pull/121

<gitbot> [13wpt-tools] 15Ms2ger closed pull request #121: Blacklist /support/ and /resources/ anywhere in the URL (06master...06blacklist-resources) 02https://github.com/w3c/wpt-tools/pull/121

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06blacklist-resources at 1406477b6: 02https://github.com/w3c/wpt-tools/commit/06477b6

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06wptserve at 1403ebf8d: 02https://github.com/w3c/wpt-tools/commit/03ebf8d

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06fix_localpaths at 14ec892f5: 02https://github.com/w3c/wpt-tools/commit/ec892f5

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06lint_json_output at 14b2c4e28: 02https://github.com/w3c/wpt-tools/commit/b2c4e28

<gitbot> [13wpt-tools] 15Ms2ger 04deleted 06py3-lint2 at 14814311e: 02https://github.com/w3c/wpt-tools/commit/814311e

<gitbot> [13web-platform-tests] 15mwatson2 opened pull request #3798: [eme] Add test for expiration attribute value (06master...06eme-jwt) 02https://github.com/w3c/web-platform-tests/pull/3798

SVG

nikos: there was an old plan nfor developing a SVG testsuite
... we're going to host in wpt but still use sheperd with metadata etc
... we were planning on importing SVG 1.0 tests
... lots of text has changed between the versions
... looking for feedback and suggestsions what we should do

<JohnJansen> clarification: svg looking for advice

nikos: we've gotten advice on github
... the old plan was old-school

<nikos> https://github.com/w3c/svgwg/issues/269

nikos: Shepherd is out obviously
... we have a requirement that the testsuite needs to be useful for non-browser implementations
... reftests be separate from script tests

gsnedders: that can be done via manifest

nikos: ok as long as we can separate for one impl that's fine

jgraham: the idea for html is that we would take every section in the spec up to 3 levels deep and create a directory
... and then fill in tests where we didn't have coverage
... didn't really help enough
... having some structure is good
... using IDs in the spec is of less value

nikos: things move around...

jgraham: not always understandable names
... pick a structure that makes sense to you according to the stucture of the spec
... but dont' worry too muhc about it
... don't dump all tests into one dir

zcorpan: if you don't want to use <link rel=help> then the last dir should match up to an id in the spec

nikos: reftests it's obvious what passes and what fails

jgraham: write something that's easy to write a ref for
... if you're testing text rendering it doesn't have to be a square...
... generally do something sensething

zcorpan: you want to share reference as much as possible

nikos: idea to use PNG e.g. for a rectangle

jgraham: should be a visual test, but minimize such tests as much as possible

JohnJansen: the fundamental stuff you don't need to test it, you'll find out if it's broken anyway

nikos: approving tests in the repo, can people be given access to do that?

gsnedders: file PRs, if they don't do stupid things we give them push access

nikos: do SVG WG people approve SVG tests?

gsnedders: vauge rules. if you think you're competent to review, go ahead and do it
... PRs lie around for months...
... until you get into complex SVG stuff people will be able to review

JohnJansen: I though more people would be interested in reviewing

nikos: trying to recruit people into doing testing
... people dropped off when we got closer to CR
... any recommendations?

JohnJansen: be open if you're being told you're doing it wrong

nikos: i'm trying to gather information...

JohnJansen: we've discovererd the hard way if you hire people to write tests you get low quality tests, which is more costly in the long run than just writing good tests yourself from the start
... test the web forward is an example, we got people in a room to write tests...

MikeSmith: culture for recommendations, we spend time in IRC channel and do very little other interaction
... if someone will contribute, join IRC
... we ahve good time coverage across timezones

<MikeSmith> zcorpan: a lot of things can be tested with testharness.js that you might not think is suitable

<MikeSmith> ... like, a lot of rendering stuff

<MikeSmith> ... if you prioritize UAs that support CSS and Javascript

<MikeSmith> nikos: Inkscape... do not not to the details

<scribe> scribenick: MikeSmith

boazsender: bet it’s not JavaScript

nikos: it’s not yeah

JohnJansen: in some cases you might want to run all your tests in your desktop browser and mobile
... and you may find that you have never found a bug that was in one place and not the other
... so the question is if Inkscape is different enough to merit running the tests in it vs running just a browsers

nikos: it is different enough

zcorpan: so, what kind of tests would work in Inkscape

nikos: reftests
... I need to talk Tav (Inkspace rep) to found out more

Summary of Action Items

[NEW] ACTION: mike to follow up with interested parties to fund work on webdriver tests [recorded in http://www.w3.org/2016/09/22-testing-minutes.html#action01]
[NEW] ACTION: MikeSmith to send dashboard outline to public-test-infra for feedback [recorded in http://www.w3.org/2016/09/22-testing-minutes.html#action02]
 

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.144 (CVS log)
$Date: 2016/09/22 14:54:00 $

Scribe.perl diagnostic output

[Delete this section before finalizing the minutes.]
This is scribe.perl Revision: 1.144  of Date: 2015/11/17 08:39:34  
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: RRSAgent_Text_Format (score 1.00)

Succeeded: s/diminic/Dominic/
Succeeded: s/nab/Navid/
Succeeded: s/jahnsen/jansen/
FAILED: s/jahnsen/jansen/
Succeeded: s/jeffrey/Geoffrey/
FAILED: s/jeffrey/geoffrey/
FAILED: s/jeffrey from google/Jeffrey on contract with Google/
Succeeded: s/from google/Jeffrey on contract with Google/
Succeeded: s/html css/html cssom/
Succeeded: s/Geoffrey from google/Geoffrey on contract with Google/
Succeeded: i/miketaylr from moz/Topic: Intros
Succeeded: i/update on wpt at msft/Topic: Update on wpt at Microsoft
Succeeded: s/hox/hoc/
Succeeded: i/what we do at mozilla/Topic: Update on wpt at Mozilla
Succeeded: i/rick now on blink/Topic: Update on wpt and Blink
Succeeded: i/sam from apple/Topic: Update on wpt and WebKit
Succeeded: s/but it/samweinig: but it/
Succeeded: s/maybe/dominicc: maybe/
Succeeded: s/for us/samweinig: for us/
FAILED: s/jamesS:/jgraham:/
Succeeded: s/jameS:/jgraham:/
Succeeded: s/stragey/strategy/
Succeeded: s/dominic/Domenic/
Succeeded: s/Topic: CSS WG merger into WPT//
Succeeded: s/paltform/platform/
Succeeded: s/teh current/the current/
Succeeded: s/all the tooling should work/all of the tooling is likely currently broken except the build system/
Succeeded: s/proparitary/proprietary/
Succeeded: s/tends to be shallow/tends to be far less shallow than 2.1/
Succeeded: s/pla/plan/
Found Scribe: boazsender
Inferring ScribeNick: boazsender
Found Scribe: boazsender
Inferring ScribeNick: boazsender
Found ScribeNick: boazsender
Found ScribeNick: zcorpan
Found ScribeNick: MikeSmith
ScribeNicks: boazsender, zcorpan, MikeSmith
Present: Boaz DomenicDenicola DominicCooney FrankOlivier JohnJansen MikeSmith MikeTaylor NavidZolghadr SamWeinig SimonPieters TessOConnor gsnedders jgraham rbyers nikos

WARNING: No meeting title found!
You should specify the meeting title like this:
<dbooth> Meeting: Weekly Baking Club Meeting


WARNING: No meeting chair found!
You should specify the meeting chair like this:
<dbooth> Chair: dbooth

Got date from IRC log name: 22 Sep 2016
Guessing minutes URL: http://www.w3.org/2016/09/22-testing-minutes.html
People with action items: mike mikesmith

[End of scribe.perl diagnostic output]