– DRAFT –
web-platform-tests

<jgraham> RRSAgent: stop logging

<jgraham> smcgruer_[EST]: RRSAgent is possibly even more useful than Zakim

<smcgruer_[EST]> My understanding from the wiki was that zakim would invite rssaagent for you

<smcgruer_[EST]> But you're definitely the expert here :)

<jgraham> I don't really have a preference on conference platforms; Zoom is terrible for Googlers, Meet is marginal for non-Chrome users

<jgraham> RRSAgent: stop

<jgraham> Chari: smcgruer_[EST]

<jgraham> RRSAgent: make logs public

<jgraham> RRSAgent: make minutes v2

scribenick Hexcles

<boazsender> Boaz Sender, Bocoup

<zcorpan> Simon Pieters, Bocoup

<BitBot> (14wpt) [PR] chromium-wpt-export-bot requested 13#26171 merge into 07master: Stop referencing 'useAutomationExtension' in chrome android browsers. - https://‌git.io/‌JT4zL

Logistics: https://‌docs.google.com/‌presentation/‌d/‌1cTOnIHsseiHgnv6bOGTcvjeIOORFI8IVKYdY1faoYzQ/‌edit

2020 review: https://‌docs.google.com/‌presentation/‌d/‌1WjYKtiDgfHWk0mEWx4G4Ml6-tLNwYkhPH69ywYpeyCE/‌edit

2020 review

[smcgruer going through the slides above]

rrsagent , draft minutes

<jgraham> RRSAgent: make minutes v2

smcgruer_[EST]: any comments regarding "Improve Interop"?

jgraham: I think the summary is fair. We still don't know what are the "high-priority" bugs. Tooling has improved. This seems that somewhere we haven't realized the full potential. How do we close the circle? Which tests are worth fixing?

<BitBot> (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26135 by chromium-wpt-export-bot into 07master: [CORS-RFC1918] Initial addressSpace Web Platform Tests. - https://‌git.io/‌JT3jy

<BitBot> (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26161 by chromium-wpt-export-bot into 07master: [layout] Fix OOF replaced elements with display: table. - https://‌git.io/‌JT80f

jgraham: we completely failed to do anything on mobile. We can discuss later.

zcorpan: can someone show the tool to surface high-priority bugs right now (on wpt.fyi or otherwise)? I don't have a good understanding of the status quo.

jgraham: wpt.fyi allows you to search by status (though quite complicated to get right), so you can search for browser-specific failures (tangent: we should get Edge off the homepage).
… For Gecko, I wrote another tool specifically for Firefox (prepopulated search, integration with Bugzilla components). People seem not thrilled. We also started to file bugs during import for most components (using metadata to keep track of existing bugs), which seems to work better as it integrates better into people's existing workflow. It's hard to prove what's working in general.

<jgraham> https://‌jgraham.github.io/‌wptdash/

boazsender: 1. I added a new agenda item to discuss "sunsetting" Edge (from wpt.fyi)
… 2. I have some suggestions re (implicit) definition of "priority" and tooling, adding a new agenda item, too

<jgraham> https://‌bugzilla.mozilla.org/‌show_bug.cgi?id=1667850 is a randomly selected example of a filed test bug

smcgruer_[EST]: moving on to "improving test suite quality"

jgraham: we've done something, e.g. improving the APIs, which would indirectly allow us to improve the coverage.

jgraham: long-time question: how do I know my spec is well-covered by tests?

jgraham: we're also closer to "identifying bad test material based on PR results" (due to improvements in GitHub checks)

<zcorpan> (disabled tests link = https://‌bocoup.github.io/‌wpt-disabled-tests-report/ )

smcgruer_[EST]: 1. yeah I forgot to mention improvements in GitHub checks here 2. not sure I interpreted the 2020 priorities correctly as I wasn't here 3. do we want to have a (breakout) discussion around spec coverage?

boazsender: added to the agenda

smcgruer_[EST]: moving on to "supporting CSSWG"

zcorpan: are we planning to have a joint meeting with CSSWG?

<jgraham> gsnedders: Not much interest from CSSWG in a joint meeting

jgraham: last year we originally hoped supporting manual tests was going to be a short hop but later found out their requirements to be much more complicated

gsnedders: and it's only for a small subset of CSS specs
… so I'd continue to argue that it's not high priority

<BitBot> (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26147 by chromium-wpt-export-bot into 07master: [AspectRatio] Correctly compute flex line length - https://‌git.io/‌JTZnu

<BitBot> (14wpt) [PR] pyup-bot requested 13#26172 merge into 07master: Update urllib3 to 1.25.11 - https://‌git.io/‌JT4V1

boazsender: to provide my memory: CSSWG used to maintain its own test suite, but was merged into WPT thanks to work by gsnedders etc. but there has been some pain/complaints regarding feature support etc. Last year CSSWG reported positively.
… short of a joint meeting, we can reach out to their chairs to get a sense of how they like WPT and where WPT is on their priorities.
… the fact that they don't want a meeting is possibly a good sign, but don't want to assume

zcorpan: CSSWG is a big and important group and we've been trying to nudge them on testing policy (i.e. tests for spec changes).
… they don't have a testing policy the same way as e.g. HTML.
… We want to do what we can to support them to write tests.
… If we reach out to the chairs, it'd be good to re-ask testing policies and their needs
… one of their editors said: because some CSS specs don't already have good test suites, it's hard to add new tests when changing the specs.
… That's one area we could improve by building the test suites.

Action: boazsender to send an email to CSSWG chairs

boazsender: I'll write to CSSWG chairs.

smcgruer_[EST]: zcorpan, are we going to write tests for them?

<BitBot> (14wpt) [issue] gsnedders transferred 13#10707: Automatically apply the "wg-css" label - https://‌git.io/‌JT4wi

zcorpan: yes and no. It's hard to write a test for a small spec change if there isn't existing test. e.g. 1 day to fix a spec bug and 1 month to develop the test suite.

smcgruer_[EST]: moving on to "community development"
… note that I'm not sure what "fix wpt-pr-bot" meant but I don't think it's in a great place based on my experience hacking on it

<BitBot> (14wpt) [PR] chromium-wpt-export-bot requested 13#26173 merge into 07master: Reland "Reenable scroll-animations/element-based-offset tests" - https://‌git.io/‌JT4rJ

boazsender: thank you Stephen for landing the CoC

smcgruer_[EST]: thank you to Bocoup, Igalia, etc.!

boazsender: and plh!

smcgruer_[EST]: moving on to "improve documentation"

zcorpan: "linking across websites" means adding links to navbars across all of our websites (wpt.fyi, docs site, etc.)

jgraham: yeah we should totally do that

boazsender: I added an agenda to the planning section to carry over useful stuff

smcgruer_[EST]: moving on to "reliable and actionable PRs"

jgraham: we've made progress, but there's still room for improvement (e.g. test authors still ask "why has my PR failed")

<smcgruer_[EST]> smcgruer_[EST]: +++ to jgraham

boazsender: does it need a time slot to discuss?

jgraham: let's leave it as a subitem for 2021 planning. There's little contention, but more about prioritization.

smcgruer_[EST]: moving on to "improve test ergonomics"

jgraham: re RFC on allowing tests to keep running when they keep producing results, we still have problems around e.g. how to prevent tests from running for hours
… technical issues

smcgruer_[EST]: we (Chromium) have not reviewed the RFC very thoroughly, but also have concerns about nondeterminism.

zcorpan: It's a double-edged swords: some tests could keep running forever, while some might actually timeout sooner.
… the current setup has the ergonomic downside of having to split big tests
… I don't know if the RFC is the best solution, but it's worth exploring as the current setup isn't ideal

<boazsender> should that be added to the agenda?

jgraham: yeah let's add "how to deal with large number of subtests" to agenda

smcgruer_[EST]: moving on to "improve wpt.fyi a11y and design"

boazsender: It's great that we run Lighthouse against wpt.fyi. There were also some complaints about PR test results... Back to wpt.fyi, it'd be great if we can also check assistive technologies in addition to Lighthouse.

smcgruer_[EST]: what about wpt-pr-bot [or test results]?

boazsender to check

smcgruer_[EST]: to break now or later?

boazsender: maybe we can take 15min to introduce ourselves more

<jgraham> (this was not scribed)

<jgraham> RRSAgent: make minutes v2

<boazsender> smcgruer_[EST]: http://‌ringmark.io/ was an example of facebook putting out a benchmark for browsers in ~2013.

<BitBot> (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26151 by chromium-wpt-export-bot into 07master: [:is/:where] Add WPT coverage for basic matching behavior - https://‌git.io/‌JTZKn

<BitBot> (14wpt) [PR] chromium-wpt-export-bot requested 13#26174 merge into 07master: [AspectRatio] Remove .tentative from file names - https://‌git.io/‌JT41b

Test Automation

smcgruer_[EST]: high prio bugs last

smcgruer_[EST]: 2 parts. Today for test automation we tell people to use webdriver

smcgruer_[EST]: it's a weird web api spec

smcgruer_[EST]: different layers, wpt, chrome driver, etc

smcgruer_[EST]: people on hte chromium side tend to look for other solutions

smcgruer_[EST]: aren't interoperable for the most part

smcgruer_[EST]: plan of record for safari, support test apis in ???

smcgruer_[EST]: is web driver automation solution working?

smcgruer_[EST]: if not what should we be doing instead?

<BitBot> (14wpt) [PR] LukeZielinski 04closed 13#26171 by LukeZielinski: Stop referencing 'useAutomationExtension' in chrome android browsers. - https://‌git.io/‌JT4zL

<BitBot> (14wpt) [PR] LukeZielinski reopened 13#26171 by LukeZielinski: Stop referencing 'useAutomationExtension' in chrome android browsers. - https://‌git.io/‌JT4zL

jgraham: i think i also see some of the concerns in that so far, gecko engineers have not implemented the web driver bits that aren't in ???

jgraham: gecko engineers also have the problem htat they don't want the touch the automation-specific code

jgraham: i think that it's a problem

jgraham: we also need to remember one of the reasons we initially pushed for webdriver is...

jgraham: if we can make things useful for authors, with webdriver apis, web developers can use it for their sites as well

jgraham: with test-only apis, they won't be shipped in the "real" browser

jgraham: the other concern is that it's easy to make those things accidentally super-internals-specific

jgraham: we don't have any test-only cross-browser api

jgraham: the browser testing spec, which will expose GC (garbage collect) API

jgraham: which doesn't make sense to expose in webdriver

jgraham: implemententable in multiple browsers

<Zakim> MikeSmith, you wanted to ask about what metrics we have for measuring how much difference the WebDriver automation has made?

jgraham: we can use that as a benchmark

MikeSmith: i think it's important that we try to measure adding a different webdriver

MikeSmith: the reason it's important is it has been an important missing feature for a long time

MikeSmith: ability to automate with webdriver is new

<jgraham> s/??? adding a different/measure hte impact of adding webdriver/

MikeSmith: maybe we can try to document...

MikeSmith: bigger deal long ago in general we wanted to get rid of manual tests

MikeSmith: or automate them

MikeSmith: if we have data on how many manual tests we've automated, would be useful

MikeSmith: see if we can get feedback from people working on different specs

MikeSmith: where we didn't have ability to automate

MikeSmith: not core wpt contributors

MikeSmith: see if it's working. if not, see what we can change

MikeSmith: we havne't collected that info afaik

Hexcles: mike just mentioned about the importance of the webdirver extension in wpt

Hexcles: also wider question of how useful it is in the web community

Hexcles: automation in webdriver, could also be used by wider web authors community. not just browsers

Hexcles: webdriver WG, understand extensions as part of wpt effort, if they're used widely in the real world

Hexcles: coming back to jgraham's point

Hexcles: there used to be another browser engine that implemented a different test only api

Hexcles: due to architectural issues, some browsers might not be able to ship test-only APIs ever

Hexcles: increased binary size...

Hexcles: test-only API route, may need test-only browser shells

jgraham: servo still exists

jgraham: mozilla doesn't work on it but the code is still there

jgraham: also chromium, is counter-point that it won't be implemented in more than one browser

jgraham: 1 case: stuff where APIs are widely used for web page automation, part of core webdriver

jgraham: uncontroversial, like click()

jgraham: some people might prefer we expose as JS method

jgraham: but not a super concern

jgraham: it's not fundamental problem with webdriver

jgraham: new spec with new webdriver extensions

jgraham: people aren't familiar with webdriver impl and don't want to implement extensions

jgraham: in general i'm not sure ... how much insight into how much different things get used

jgraham: can't add Telemetry for webdriver

jgraham: hard to know what's being used

<Zakim> smcgruer_[EST], you wanted to note that WebKit is implementing webxr test-only API - I think...

smcgruer_[EST]: i think webkit were looking at webxr test api

smcgruer_[EST]: most chromium devs ask us: how do we test this new cool feature

smcgruer_[EST]: mostly we say use webdriver

smcgruer_[EST]: not all cases, but most

smcgruer_[EST]: i think things break down because they don't feel it's their job to do that

smcgruer_[EST]: we know why it's painful at a high level but we haven't tried to figure out how to make it less painful

smcgruer_[EST]: measuring webdriver stuff: 66 type:untestable issues

smcgruer_[EST]: not all webdriver, but a source of data

boazsender: i heard an action item.. reach out to people for feedback

boazsender: how do we record those

boazsender: reach out to BTT

boazsender: reach out to selenium

boazsender: converge with devtools

smcgruer_[EST]: on tracking action items...

Action: reach out to BTT

Action: reach out to selenium

Action: partner with browser engineering team(s) to find out where the process sucks

Action: 2021 MDN DNA research design to inform test automation

<Hexcles> RRSAgent: make minutes v2

jgraham: the discussion with webdriver pain here is more browser internal

<Zakim> gsnedders, you wanted to ask how many WebDriver/test only APIs we have defined and for what features

jgraham: does the webdriver failure line mean we have to adopt a different approach?

gsnedders: at a high level, what are the things we currently have test api for in specs, regardless of whether they're implemented?

gsnedders: mostly click events, kb events

gsnedders: webauthn

gsnedders: what else is there?

smcgruer_[EST]: i have a list... permissions,

jgraham: other test-only apis, web xr, web bluetooth... tends to be device access stuff

gsnedders: so take those lists, and write up which of those have cross-browser support

gsnedders: and which have cross-browser support for webdriver end point

Action: compare test only API and webdriver API cross-browser support

<Zakim> MikeSmith, you wanted to mention specifically asking WebKit contributors for feedback

MikeSmith: general suggestion

MikeSmith: we've had close involvement from the chromium team and firefox

MikeSmith: less so from webkit

MikeSmith: well known that there's some general dissatisfaction from webkit about writing wpt tests

MikeSmith: obviously it would be nice to try to reduce some of that

MikeSmith: opportunity to find out what we need to fix

MikeSmith: i think we know a lot of the pain points...

MikeSmith: since gsnedders is here

MikeSmith: from people at the webkit project about this particular thing

Action: gsnedders to get feedback from webkit developers about painpoints with wpt

jgraham: to the extent that this is not just an org thing... about webdriver being limited

jgraham: work with bidi webdriver

jgraham: lower-level access to control the browsers

jgraham: fix some limitations

jgraham: the way in which wpt runner works too

jgraham: ability to talk to more than one frame at once

jgraham: not just command-response

<Domenic> What's the modern way to bail out of a test early if a feature isn't implemented? Things got split up a good amount...

jgraham: also webdriver apis...

jgraham: once we can have more advanced stuff in webdriver

jgraham: direction webdriver is moving in

jgraham: stuff we can benefit from

jgraham: if we're making decisions here, having this future trajectory in mind

jgraham: and to pre-emptively answer smcgruer_[EST] question: no, don't think so

<boazsender> ?

jgraham: CDP endpoint,

jgraham: if it was possible to expose ???

<Zakim> boazsender, you wanted to react to boazsender

boazsender: the APIs that are not using webdriver are only partially tested at all... are bluetooth, xr...

boazsender: those are different to test, maybe not harder

<jgraham> jgraham: In Chromium CDP is implemented directly in the product code. If WebDriver-bidi was exposed directly in the product code rather than needing an abstraction layer, this might solve the problem somewhat

boazsender: how to mock them across the system

boazsender: is unique

<jgraham> jgraham: "product code" here being the implementation of the web-exposed spec piece in C++ code

boazsender: BTT could maybe be more welcoming when folks come and want to automate their tests

<Zakim> smcgruer_[EST], you wanted to ask if webdriver bidi will actually make adding new webdriver features easier (whats the reduced overhead for feature engineers?)

<jgraham> BTT is next week; idk if it's on the TPAC agenda

smcgruer_[EST]: hoping bidi can help

smcgruer_[EST]: in summary, work to do to make webdriver path... know where the painpoints are and work to fixing them. still webdriver as the main solution

Screen reader automation (for ARIA-AT)

zcorpan: there's a blog post introducing ARIA-AT

<gsnedders> https://‌bocoup.com/‌blog/‌interoperability-testing-for-assistive-technologies-and-the-web-platform

zcorpan: describing ARIA-AT
… There's been existing effort testing a11y API (e.g. testing the a11y tree), but not at the screen reader level (the end users).
… One problem is that ARIA-AT tests are manual. No one knows how to automate screen readers. Prototype TBD next month with NVDA
… I want to standardize a protocol for automating screen readers similar to webdriver. This is a heads up on this upcoming work.
… We don't want "pixel-perfect" interop for screen readers but want to interop on user interface, functionalities, etc.
… We haven't started yet. Anyone interested is welcome to collaborate.

<BitBot> (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26173 by chromium-wpt-export-bot into 07master: Reland "Reenable scroll-animations/element-based-offset tests" - https://‌git.io/‌JT4rJ

jgraham: It's not clear to me what kind of API surface you're looking for.

<jcraig> s/it's clear/it's not clear/

zcorpan: It needs to be specific to each screen reader e.g. w.r.t. what it says, so perhaps specific assertions depending on screen readers. Though in some cases, screen readers may agree. WebDriver may have similar issues (special browser-specific handling).

jgraham: yeah webdriver does have some special magic for each browser (e.g. form elements), which is tricky.

<Zakim> jcraig, you wanted to emphasize Security should be central through the whole process (re: automating AT)

jcraig: I want to bring up Security as a goal for ARIA-AT. Assistive Technology (on any platform) is authorized to do whatever any human user can do, so occasionally it's exploited as a Security attack vector. Since you are working on NVDA first, closely coordinate on Sec with Windows Accessibility at Microsoft, and the NVDA core
… Apple has taken a lot of caution w.r.t. automation for example, Xcode will only automate a dev copy of your own app, not other system apps.
… Locking down the security of VocieOver is one blocker for shipping automation.

muodov: Is the plan to work together with the vendors of a11y technologies?

zcorpan: yes we'll reach out to vendors, hopefully working together.

<Zakim> smcgruer_[EST], you wanted to ask if Simon has thoughts on what merging with wpt looks like

smcgruer_[EST]: what kind of integration with WPT do you have in mind?

zcorpan: for right now, it won't be useful to integrate. Down the line, it'd be useful to have results no wpt.fyi and source code in the same repo if it makes sense.
… fundamentally we're automating screen readers that automate browsers, so it seems compatible and related.

<BitBot> (14wpt) [PR] domenic requested 13#26175 merge into 07master: Test that dynamic import() is disallowed in paint worklets - https://‌git.io/‌JT47u

<jcraig> s/I want to bring up Security as a goal for ARIA-AT. Assistive Technology (on any platform) is authorized to do whatever any human user can do, so occasionally it's exploited as a Security attack vector. Since you are working on NVDA first, closely coordinate on Sec with Windows Accessibility at Microsoft, and the NVDA core team./

<jgraham> RRSAgent: Make minutes v2

<jcraig> team./

boazsender: re WPT integration, we don't know yet, but it'd be great to share tools. Some aspects of a11y might be outside of the web platform. The spirit of the WPT is in browsers and the things around. There were discussions of bringing JS testing. ARIA-AT is similar. It's also similar to automating devices, etc. So we'd like some conversions on this theme. We're also building yet another manual testing system, in addition to CSSWG's (by plinss).

surfacing high priority bugs / how to identify test coverage for specs

jgraham: actually two topics...

jgraham: expertise in this group, makes sense to talk here

jgraham: can still also discuss in a breakout

<Zakim> MikeSmith, you wanted to comment

MikeSmith: as far as the second part goes, coverage

MikeSmith: perma-want that we've talked abou tforever

MikeSmith: haven't made much progress on

MikeSmith: a good strategy would be to look at what we've tried so far

MikeSmith: what benefit has it provided

<Zakim> jgraham, you wanted to give a summary of the things I remember

MikeSmith: continue doing it? other ways ?

jgraham: historically, 2 similar but different conceptualisations

jgraham: spec-centric model

jgraham: question is: for each testable assertion

jgraham: does it have a related test case

jgraham: typically the approach that people advocate ... bikeshed has some support ...

jgraham: either link from test to the relevant spec section

jgraham: <link rel=help> in the test

jgraham: traditional problem with that, reason we often push back, is that when you insist people add that metadata...

jgraham: people refuse to write the test

jgraham: if you don't have automation to update it, then they get out of date

jgraham: spec authors aren't keen to update test metadata to land their spec PRs

jgraham: CSS trying now, in the spec you link to the test

jgraham: in the process you're encouraged to find all the tests and add them to the spec

jgraham: but still have hte inverse problem

jgraham: CSS WG are trying this out...

jgraham: statements in the spec that links to test is ???

jgraham: other approach is code coverage, in the implementation how much is used by running the test

jgraham: limitations... doesn't tell you about code you didn't write

jgraham: doesn't tell you you didn't implement a section of the spec

jgraham: also how do you compare between browsers

jgraham: or how do you compare with the spec

<gsnedders> jgraham: so I think that's how this has been framed before, either for progressing specs and wanting to check features are sufficiently tested to demonstrate interop, or from browsers using coverage to see where there are gaps

<Zakim> MikeSmith, you wanted to remember that CSSWG has some stuff intended to help identify gaps in test coverage; wonder how well that has been working for them? and to realize that step 1 is actually identifying gaps in coverage, and can do we do more tooling to help with that at least?

<jgraham> jgraham: Never found a solution we can require as part of the process

MikeSmith: we end up having this discussion each time, so it seems a bit of an intractable problem

MikeSmith: so of the groups that are doing this, the CSS has some processes they put in place

gsnedders: from the css point of view

gsnedders: the side that historically has been used most

gsnedders: css wg own's tooling

gsnedders: tests within each section

gsnedders: pass result for each browser

gsnedders: more recently with the work happening in bikeshed

gsnedders: the link is within the spec instead of within the test

gsnedders: an argument that it helps.. when reviewing tests that exist

gsnedders: having very large testsuites

gsnedders: large number of duplicate tests

gsnedders: true for things that are moving towards rec, like flexbox

gsnedders: testsuites predate wpt

gsnedders: tests being direction upstreamed directly into wpt

gsnedders: leading to more duplication

gsnedders: from the css POV, advantage to the process

gsnedders: not large amount of manual work to review the testsuite to understand what the coverage is

<Zakim> leobalter, you wanted to coverage

gsnedders: hope it answers mike's question

MikeSmith: what we've heard in the past...

MikeSmith: 2 sides, one is the side of trying to get people for lack of a better word, pre-emptively ??? test coverage

MikeSmith: it's a lot of work

MikeSmith: for someone who has another task to do, a lot to ask for

MikeSmith: not a problem that tooling can help with

MikeSmith: we don't have a tool that writes tests

MikeSmith: not that way at least

MikeSmith: identifying gaps in test coverage

MikeSmith: i know jgraham has done work with that

MikeSmith: tooling that w ehave, surface info for other contributors

MikeSmith: maybe we can get new contributers

MikeSmith: part of the history of this project, we had the effort "test the web forward"

MikeSmith: that got us some good contributors but generated more work for others

MikeSmith: needed more review

MikeSmith: shifted work to more places

MikeSmith: first step is, document in some public fasion, where the gaps in test coverage are that we know about

MikeSmith: then encourage quality contribution

<gsnedders> perhaps also worth noting the origins of WPT in the HTML Testing TF where the goal was getting HTML 5 to REC, and in a sense the coverage gaps question is the same here as it is now

MikeSmith: in ways that doesn't repeat the mistakes with test the web forward

MikeSmith: also created a bunch of PRs that went stale

MikeSmith: because the tests weren't great or important

MikeSmith: we know we have gaps in coverage in lots of places

MikeSmith: problem is priority

<Zakim> leobalter, you wanted to react to leobalter

leobalter: i second jgraham

leobalter: similar with test262

leobalter: coverage though automation

leobalter: good points , validating coverage

leobalter: bring beginners, save some work

leobalter: experience with tests in projects

leobalter: remember bocoup some automation for coverage

leobalter: that rick was exploring

boazsender: we ran out of funding for that, but also ran into some of the problems....

boazsender: we ran out of funding

zcorpan: I had a comment on what bikeshed were doing w/ annotating specs

zcorpan: Would only support linking at a paragraph level, which a coarser granularity than you can test. No good way to map those tests onto the spec requirements, which can be single line assertions.

zcorpan: second point, as far as I know, WHATWG editors tried this and it just wasn't maintainable. Can't remember why, but sheer number of tests is one I believe (spec source increases, time to maintain).

boazsender: in test262, each test has annotation for spec

boazsender: i know leobalter has a lot of experience with this

boazsender: one way we've done this, manually annotating specs, making a fork of the spec and annotate to make a test plan

boazsender: conversation about the workflow

boazsender: who is responsible for this

boazsender: in addition to automated tools

boazsender: thinking about what coverage means and tracking it

boazsender: as the spec changes, we can update the test plan and the relationship

boazsender: in order for a spec to reach maturity at the w3c

boazsender: it has to have tests

boazsender: is there a requirement on how ... how fully tested it is

boazsender: job of the editor?

boazsender: to assess test coverage

boazsender: i dont' think we're talking about the beginner story...

boazsender: want to come back to that

boazsender: if we don't annotate specs or tests and enforce that... maybe we can have test plans be part of the work

boazsender: part of the w3c process

smcgruer_[EST]: we've talked about this before...

smcgruer_[EST]: is this something that anyone can prioritize?

<Zakim> MikeSmith, you wanted to speak about HTML spec experience and to comment specifically about boazsender question regarding W3C policy and to comment also on beginner story

MikeSmith: w3c policy, no policy for WGs around ensuring they have test coverage

MikeSmith: based on faith

MikeSmith: also hard to measure to evaluate

MikeSmith: for beginners, we can have a better strategy

<jgraham> close the queue

MikeSmith: whatwg work, we're not doing anything getting test coverage integration into specs

MikeSmith: editors don't have the time ... or not priority

MikeSmith: tooling in bikeshed and the html spec, but we're not taking advantage of the tooling

MikeSmith: can make it a "good first bug" thing

MikeSmith: for each spec, need some info added

MikeSmith: to at least indicate the current test coverage

<leobalter> smcgruer_[EST]: can we add a follow up in the agenda to discuss coverage on the next day?

MikeSmith: could be appealing to some contributors

jgraham: in theory, mike said it's trust basis, but one success is we've had more group insist on tests to land PRs

jgraham: should help with test coverage

jgraham: concrete achievement

<leobalter> smcgruer_[EST]: I understand that, thanks!

jgraham: enforcing coverage is misaligned incentives...

jgraham: not in anyone's interest

jgraham: we don't have coverage in browser code...

jgraham: telling spec editors to do something that browser developers don't do, is a tough sell

smcgruer_[EST]: for wednesday, i'll clean up the agenda...

<boazsender> just want to quickly say that I don't think the beginner answer can work with out funding to mentor and support beginners

smcgruer_[EST]: thanks everybody

<jgraham> RRSAgent: make minutes v2