13:16:37 RRSAgent has joined #testing 13:16:37 logging to https://www.w3.org/2020/10/19-testing-irc 13:16:47 RRSAgent: stop logging 13:16:47 I'm logging. I don't understand 'stop logging', jgraham. Try /msg RRSAgent help 13:17:04 smcgruer_[EST]: RRSAgent is possibly even more useful than Zakim 13:17:25 My understanding from the wiki was that zakim would invite rssaagent for you 13:17:37 But you're definitely the expert here :) 13:17:48 I don't really have a preference on conference platforms; Zoom is terrible for Googlers, Meet is marginal for non-Chrome users 13:18:31 RRSAgent: stop 14:05:58 jesdaigle has joined #testing 14:06:07 Meeting: web-platform-tests 14:06:15 Chari: smcgruer_[EST] 14:06:41 Agenda: https://docs.google.com/document/d/1XhUwVbitdV03eX2GefmokCliWjQdAxBgblwITwNqsgc/edit#heading=h.6r5o8clyt3zl 14:07:03 RRSAgent: make logs public 14:07:12 RRSAgent: make minutes v2 14:07:12 I have made the request to generate https://www.w3.org/2020/10/19-testing-minutes.html jgraham 14:07:55 present+ 14:08:07 q? 14:08:21 present+ 14:08:22 present+ 14:08:32 present+ 14:09:09 present+ 14:09:17 present+ 14:09:26 scribenick Hexcles 14:10:00 present+ 14:10:05 Boaz Sender, Bocoup 14:10:41 Simon Pieters, Bocoup 14:10:41 ScribeNick: Hexcles 14:11:04 Chair: smcgruer_[EST] 14:11:47 (14wpt) [PR] chromium-wpt-export-bot requested 13#26171 merge into 07master: Stop referencing 'useAutomationExtension' in chrome android browsers. - https://git.io/JT4zL 14:12:01 bkardell_ has joined #testing 14:12:09 Agenda: https://docs.google.com/document/d/1XhUwVbitdV03eX2GefmokCliWjQdAxBgblwITwNqsgc/edit 14:12:29 Logistics: https://docs.google.com/presentation/d/1cTOnIHsseiHgnv6bOGTcvjeIOORFI8IVKYdY1faoYzQ/edit 14:12:39 2020 review: https://docs.google.com/presentation/d/1WjYKtiDgfHWk0mEWx4G4Ml6-tLNwYkhPH69ywYpeyCE/edit 14:15:17 [smcgruer going through the slides above] 14:15:45 leobalter has joined #testing 14:16:04 rrsagent , draft minutes 14:17:09 RRSAgent: make minutes v2 14:17:09 I have made the request to generate https://www.w3.org/2020/10/19-testing-minutes.html jgraham 14:20:30 q? 14:20:38 smcgruer_[EST]: any comments regarding "Improve Interop"? 14:21:12 q+ 14:21:32 ack jgraham 14:22:00 msanchez has joined #testing 14:22:07 muodov has joined #testing 14:23:11 q? 14:23:34 jgraham: I think the summary is fair. We still don't know what are the "high-priority" bugs. Tooling has improved. This seems that somewhere we haven't realized the full potential. How do we close the circle? Which tests are worth fixing? 14:23:46 (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26135 by chromium-wpt-export-bot into 07master: [CORS-RFC1918] Initial addressSpace Web Platform Tests. - https://git.io/JT3jy 14:23:51 (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26161 by chromium-wpt-export-bot into 07master: [layout] Fix OOF replaced elements with display: table. - https://git.io/JT80f 14:23:58 q+ 14:24:01 jgraham: we completely failed to do anything on mobile. We can discuss later. 14:24:03 ack zcorpan 14:25:05 zcorpan: can someone show the tool to surface high-priority bugs right now (on wpt.fyi or otherwise)? I don't have a good understanding of the status quo. 14:26:11 q+ 14:26:17 jgraham: wpt.fyi allows you to search by status (though quite complicated to get right), so you can search for browser-specific failures (tangent: we should get Edge off the homepage). 14:27:41 ack boazsender 14:28:30 ... For Gecko, I wrote another tool specifically for Firefox (prepopulated search, integration with Bugzilla components). People seem not thrilled. We also started to file bugs during import for most components (using metadata to keep track of existing bugs), which seems to work better as it integrates better into people's existing workflow. It's hard to prove what's working in general. 14:29:26 https://jgraham.github.io/wptdash/ 14:29:45 boazsender: 1. I added a new agenda item to discuss "sunsetting" Edge (from wpt.fyi) 14:30:42 q? 14:30:53 ack smcgruer_[EST] 14:31:05 ... 2. I have some suggestions re (implicit) definition of "priority" and tooling, adding a new agenda item, too 14:31:23 q? 14:31:28 jib has joined #testing 14:31:44 https://bugzilla.mozilla.org/show_bug.cgi?id=1667850 is a randomly selected example of a filed test 14:32:08 q? 14:32:19 s/filed test/filed test bug/ 14:32:27 smcgruer_[EST]: moving on to "improving test suite quality" 14:32:39 q+ 14:32:47 ack jgraham 14:33:58 jgraham: we've done something, e.g. improving the APIs, which would indirectly allow us to improve the coverage. 14:34:40 jgraham: long-time question: how do I know my spec is well-covered by tests? 14:35:52 ack smcgruer_[EST] 14:36:00 jgraham: we're also closer to "identifying bad test material based on PR results" (due to improvements in GitHub checks) 14:36:21 (disabled tests link = https://bocoup.github.io/wpt-disabled-tests-report/ ) 14:37:39 smcgruer_[EST]: 1. yeah I forgot to mention improvements in GitHub checks here 2. not sure I interpreted the 2020 priorities correctly as I wasn't here 3. do we want to have a (breakout) discussion around spec coverage? 14:37:51 q+ 14:37:52 q? 14:37:54 ack boazsender 14:37:58 q- 14:38:11 boazsender: added to the agenda 14:39:16 smcgruer_[EST]: moving on to "supporting CSSWG" 14:39:48 q+ 14:39:48 q+ 14:39:51 q+ 14:39:54 ack zcorpan 14:40:14 zcorpan: are we planning to have a joint meeting with CSSWG? 14:41:00 gsnedders: Not much interest from CSSWG in a joint meeting 14:41:13 q? 14:41:24 ack jgraham 14:42:47 jgraham: last year we originally hoped supporting manual tests was going to be a short hop but later found out their requirements to be much more complicated 14:42:51 q+ 14:43:07 ack boazsender 14:43:13 gsnedders: and it's only for a small subset 14:43:22 ... so I'd continue to argue that it's not high priority 14:43:42 s/for a small subset/for a small subset of CSS specs/ 14:43:49 (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26147 by chromium-wpt-export-bot into 07master: [AspectRatio] Correctly compute flex line length - https://git.io/JTZnu 14:44:04 (14wpt) [PR] pyup-bot requested 13#26172 merge into 07master: Update urllib3 to 1.25.11 - https://git.io/JT4V1 14:45:02 q? 14:45:15 boazsender: to provide my memory: CSSWG used to maintain its own test suite, but was merged into WPT thanks to work by gsnedders etc. but there has been some pain/complaints regarding feature support etc. Last year CSSWG reported positively. 14:45:58 ... short of a joint meeting, we can reach out to their chairs to get a sense of how they like WPT and where WPT is on their priorities. 14:46:22 ack zcorpan 14:46:38 ... the fact that they don't want a meeting is possibly a good sign, but don't want to assume 14:47:25 zcorpan: CSSWG is a big and important group and we've been trying to nudge them on testing policy (i.e. tests for spec changes). 14:47:46 ... they don't have a testing policy the same way as e.g. HTML. 14:47:59 ... We want to do what we can to support them to write tests. 14:48:23 ... If we reach out to the chairs, it'd be good to re-ask testing policies and their needs 14:49:08 ... one of their editors said: because some CSS specs don't already have good test suites, it's hard to add new tests when changing the specs. 14:49:14 q+ 14:49:36 ... That's one area we could improve by building the test suites. 14:49:41 ack smcgruer_[EST] 14:49:52 boazsender: I'll write to CSSWG chairs. 14:50:19 smcgruer_[EST]: zcorpan, are we going to write tests for them? 14:50:44 (14wpt) [issue] gsnedders transferred 13#10707: Automatically apply the "wg-css" label - https://git.io/JT4wi 14:51:14 zcorpan: yes and no. It's hard to write a test for a small spec change if there isn't existing test. e.g. 1 day to fix a spec bug and 1 month to develop the test suite. 14:52:02 smcgruer_[EST]: moving on to "community development" 14:52:42 ... note that I'm not sure what "fix wpt-pr-bot" meant but I don't think it's in a great place based on my experience hacking on it 14:53:18 (14wpt) [PR] chromium-wpt-export-bot requested 13#26173 merge into 07master: Reland "Reenable scroll-animations/element-based-offset tests" - https://git.io/JT4rJ 14:53:36 boazsender: thank you Stephen for landing the CoC 14:53:53 smcgruer_[EST]: thank you to Bocoup, Igalia, etc.! 14:53:58 boazsender: and plh! 14:54:35 smcgruer_[EST]: moving on to "improve documentation" 14:54:52 q+ 14:55:00 q+ 14:55:04 ack zcorpan 14:55:22 q- 14:55:47 q+ 14:56:01 zcorpan: "linking across websites" means adding links to navbars across all of our websites (wpt.fyi, docs site, etc.) 14:56:37 ack jgraham 14:57:12 jgraham: yeah we should totally do that 14:58:41 boazsender: I added an agenda to the planning section to carry over useful stuff 14:59:12 smcgruer_[EST]: moving on to "reliable and actionable PRs" 14:59:57 q+ 14:59:59 q+ 15:00:58 jgraham: we've made progress, but there's still room for improvement (e.g. test authors still ask "why has my PR failed") 15:00:59 smcgruer_[EST]: +++ to jgraham 15:01:42 boazsender: does it need a time slot to discuss? 15:02:18 jgraham: let's leave it as a subitem for 2021 planning. There's little contention, but more about prioritization. 15:02:57 smcgruer_[EST]: moving on to "improve test ergonomics" 15:03:07 q+ 15:04:20 jgraham: re RFC on allowing tests to keep running when they keep producing results, we still have problems around e.g. how to prevent tests from running for hours 15:04:29 ... technical issues 15:04:52 q+ 15:05:48 smcgruer_[EST]: we (Chromium) have not reviewed the RFC very thoroughly, but also have concerns about nondeterminism. 15:06:15 zcorpan: It's a double-edged swords: some tests could keep running forever, while some might actually timeout sooner. 15:06:44 ... the current setup has the ergonomic downside of having to split big tests 15:07:35 ... I don't know if the RFC is the best solution, but it's worth exploring as the current setup isn't ideal 15:07:50 should that be added to the agenda? 15:08:39 jgraham: yeah let's add "how to deal with large number of subtests" to agenda 15:09:02 smcgruer_[EST]: moving on to "improve wpt.fyi a11y and design" 15:09:33 q+ 15:09:38 ack boazsender 15:11:20 boazsender: It's great that we run Lighthouse against wpt.fyi. There were also some complaints about PR test results... Back to wpt.fyi, it'd be great if we can also check assistive technologies in addition to Lighthouse. 15:11:44 smcgruer_[EST]: what about wpt-pr-bot [or test results]? 15:11:54 boazsender to check 15:14:04 smcgruer_[EST]: to break now or later? 15:14:17 boazsender: maybe we can take 15min to introduce ourselves more 15:15:40 (this was not scribed) 15:15:48 RRSAgent: make minutes v2 15:15:48 I have made the request to generate https://www.w3.org/2020/10/19-testing-minutes.html jgraham 15:17:29 xiaoqian has joined #testing 15:23:37 smcgruer_[EST]: http://ringmark.io/ was an example of facebook putting out a benchmark for browsers in ~2013. 15:28:27 krosylight has joined #testing 15:43:02 (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26151 by chromium-wpt-export-bot into 07master: [:is/:where] Add WPT coverage for basic matching behavior - https://git.io/JTZKn 15:51:36 (14wpt) [PR] chromium-wpt-export-bot requested 13#26174 merge into 07master: [AspectRatio] Remove .tentative from file names - https://git.io/JT41b 15:59:58 yhirano_ has joined #testing 16:03:52 scribenick: zcorpan 16:03:55 topic: Test Automation 16:04:20 smcgruer_[EST]: high prio bugs last 16:04:38 boazsender has joined #testing 16:04:38 i/smcgruer going through the slides above/Topic: 2020 review 16:04:50 smcgruer_[EST]: 2 parts. Today for test automation we tell people to use webdriver 16:04:59 smcgruer_[EST]: it's a weird web api spec 16:05:07 smcgruer_[EST]: different layers, wpt, chrome driver, etc 16:05:22 smcgruer_[EST]: people on hte chromium side tend to look for other solutions 16:05:29 smcgruer_[EST]: aren't interoperable for the most part 16:05:48 smcgruer_[EST]: plan of record for safari, support test apis in ??? 16:05:56 q+ 16:06:00 smcgruer_[EST]: is web driver automation solution working? 16:06:08 smcgruer_[EST]: if not what should we be doing instead? 16:06:18 (14wpt) [PR] LukeZielinski 04closed 13#26171 by LukeZielinski: Stop referencing 'useAutomationExtension' in chrome android browsers. - https://git.io/JT4zL 16:06:19 (14wpt) [PR] LukeZielinski reopened 13#26171 by LukeZielinski: Stop referencing 'useAutomationExtension' in chrome android browsers. - https://git.io/JT4zL 16:06:24 q? 16:06:44 jgraham: i think i also see some of the concerns in that so far, gecko engineers have not implemented the web driver bits that aren't in ??? 16:06:59 q+ to ask about what metrics we have for measuring how much difference the WebDriver automation has made? 16:07:20 jgraham: gecko engineers also have the problem htat they don't want the touch the automation-specific code 16:07:26 jgraham: i think that it's a problem 16:07:40 jgraham: we also need to remember one of the reasons we initially pushed for webdriver is... 16:08:05 jgraham: if we can make things useful for authors, with webdriver apis, web developers can use it for their sites as well 16:08:19 jgraham: with test-only apis, they won't be shipped in the "real" browser 16:08:32 jgraham: the other concern is that it's easy to make those things accidentally super-internals-specific 16:08:47 jgraham: we don't have any test-only cross-browser api 16:09:04 jgraham: the browser testing spec, which will expose GC (garbage collect) API 16:09:17 jgraham: which doesn't make sense to expose in webdriver 16:09:25 jgraham: implemententable in multiple browsers 16:09:36 MikeSmith, you wanted to ask about what metrics we have for measuring how much difference the WebDriver automation has made? 16:09:37 jgraham: we can use that as a benchmark 16:10:04 q+ 16:10:12 q+ 16:10:13 MikeSmith: i think it's important that we try to ??? adding a different webdriver 16:10:29 MikeSmith: the reason it's important is it has been an important missing feature for a long time 16:10:39 MikeSmith: ability to automate with webdriver is new 16:10:46 s/???/measure/ 16:10:53 s/??? adding a different/measure hte impact of adding webdriver/ 16:11:01 MikeSmith: maybe we can try to document... 16:11:16 MikeSmith: bigger deal long ago in general we wanted to get rid of manual tests 16:11:20 MikeSmith: or automate them 16:11:32 MikeSmith: if we have data on how many manual tests we've automated, would be useful 16:11:45 MikeSmith: see if we can get feedback from people working on different specs 16:11:54 MikeSmith: where we didn't have ability to automate 16:11:59 MikeSmith: not core wpt contributors 16:12:07 MikeSmith: see if it's working. if not, see what we can change 16:12:14 q+ 16:12:14 MikeSmith: we havne't collected that info afaik 16:12:32 Hexcles: mike just mentioned about the importance of the webdirver extension in wpt 16:12:44 Hexcles: also wider question of how useful it is in the web community 16:13:05 Hexcles: automation in webdriver, could also be used by wider web authors community. not just browsers 16:13:24 Hexcles: webdriver WG, understand extensions as part of wpt effort, if they're used widely in the real world 16:13:35 Hexcles: coming back to jgraham's point 16:13:50 Hexcles: there used to be another browser engine that implemented a different test only api 16:14:07 Hexcles: due to architectural issues, some browsers might not be able to ship test-only APIs ever 16:14:13 Hexcles: increased binary size... 16:14:27 Hexcles: test-only API route, may need test-only browser sheels 16:14:32 s/sheels/shells/ 16:14:45 q? 16:14:47 jgraham: servo still exists 16:14:58 jgraham: mozilla doesn't work on it but the code is still there 16:15:26 jgraham: also chromium, is counter-point that it won't be implemented in more than one browser 16:15:45 jgraham: 1 case: stuff where APIs are widely used for web page automation, part of core webdriver 16:15:51 jgraham: uncontroversial, like click() 16:16:04 jgraham: some people might prefer we expose as JS method 16:16:09 jgraham: but not a super concern 16:16:23 jgraham: it's not fundamental problem with webdriver 16:16:34 jgraham: new spec with new webdriver extensions 16:16:55 jgraham: people aren't familiar with webdriver impl and don't want to implement extensions 16:17:19 jgraham: in general i'm not sure ... how much insight into how much different things get used 16:17:27 jgraham: can't add Telemetry for webdriver 16:17:34 jgraham: hard to know what's being used 16:17:54 q? 16:18:09 q+ 16:18:12 smcgruer_[EST], you wanted to note that WebKit is implementing webxr test-only API - I think... 16:18:32 smcgruer_[EST]: i think webkit were looking at webxr test api 16:18:46 smcgruer_[EST]: most chromium devs ask us: how do we test this new cool feature 16:18:48 q+ to ask how many WebDriver/test only APIs we have defined and for what features 16:18:52 smcgruer_[EST]: mostly we say use webdriver 16:18:58 smcgruer_[EST]: not all cases, but most 16:19:11 smcgruer_[EST]: i think things break down because they don't feel it's their job to do that 16:19:11 q+ to mention webdriver bidi 16:19:34 smcgruer_[EST]: we know why it's painful at a high level but we haven't tried to figure out how to make it less painful 16:19:49 smcgruer_[EST]: measuring webdriver stuff: 66 type:untestable issues 16:19:57 smcgruer_[EST]: not all webdriver, but a source of data 16:20:20 boazsender: i heard an action item.. reach out to people for feedback 16:20:26 boazsender: how do we record those 16:20:40 boazsender: reach out to BTT 16:20:47 boazsender: reach out to selenium 16:20:59 boazsender: converge with devtools 16:21:06 q+ to mention specifically asking WebKit contributors for feedback 16:21:21 smcgruer_[EST]: on tracking action items... 16:22:19 ACTION: reach out to BTT 16:22:28 ACTION: reach out to selenium 16:23:31 ACTION: partner with browser engineering team(s) to find out where the process sucks 16:23:32 i/boazsender: I'll write to CSSWG chairs./ACTION: boazsender to send an email to CSSWG chairs 16:24:21 ACTION: 2021 MDN DNA research design to inform test automation 16:24:24 RRSAgent: make minutes v2 16:24:24 I have made the request to generate https://www.w3.org/2020/10/19-testing-minutes.html Hexcles 16:24:57 jgraham: the discussion with webdriver pain here is more browser internal 16:25:18 gsnedders, you wanted to ask how many WebDriver/test only APIs we have defined and for what features 16:25:19 jgraham: does the webdriver failure line mean we have to adopt a different approach? 16:25:27 q+ 16:26:02 gsnedders: at a high level, what are the things we currently have test api for in specs, regardless of whether they're implemented? 16:26:10 gsnedders: mostly click events, kb events 16:26:19 gsnedders: ??? 16:26:27 gsnedders: what else is there? 16:26:46 smcgruer_[EST]: i have a list... permissions, 16:27:11 jgraham: other test-only apis, web xr, web bluetooth... tends to be device access stuff 16:27:18 q+ 16:27:24 gsnedders: so take those lists, and write up which of those have cross-browser support 16:27:39 gsnedders: and which have cross-browser support for webdriver end point 16:27:55 ACTION: compare test only API and webdriver API cross-browser support 16:27:59 MikeSmith, you wanted to mention specifically asking WebKit contributors for feedback 16:28:15 MikeSmith: general suggestion 16:28:15 s/gsnedders: ???/gsnedders: webauthn/ 16:28:29 MikeSmith: we've had close involvement from the chromium team and firefox 16:28:34 MikeSmith: less so from webkit 16:28:49 MikeSmith: well known that there's some general dissatisfaction from webkit about writing wpt tests 16:29:00 MikeSmith: obviously it would be nice to try to reduce some of that 16:29:09 MikeSmith: opportunity to find out what we need to fix 16:29:16 MikeSmith: i think we know a lot of the pain points... 16:29:22 MikeSmith: since gsnedders is here 16:29:37 MikeSmith: from people at the webkit project about this particular thing 16:29:53 ACTION: gsnedders to get feedback from webkit developers about painpoints with wpt 16:30:21 jgraham: to the extent that this is not just an org thing... about webdriver being limited 16:30:31 jgraham: work with bidi webdriver 16:30:38 jgraham: lower-level access to control the browsers 16:30:46 jgraham: fix some limitations 16:30:55 jgraham: the way in which wpt runner works too 16:31:05 jgraham: ability to talk to more than one frame at once 16:31:11 jgraham: not just command-response 16:31:20 What's the modern way to bail out of a test early if a feature isn't implemented? Things got split up a good amount... 16:31:35 jgraham: also webdriver apis... 16:31:51 jgraham: once we can have more advanced stuff in webdriver 16:31:56 jgraham: direction webdriver is moving in 16:32:01 jgraham: stuff we can benefit from 16:32:12 jgraham: if we're making decisions here, having this future trajectory in mind 16:32:27 jgraham: and to pre-emptively answer smcgruer_[EST] question: no, don't think so 16:32:33 ? 16:32:42 jgraham: CDP endpoint, 16:32:55 jgraham: if it was possible to expose ??? 16:33:18 boazsender, you wanted to react to boazsender 16:33:28 jcraig has joined #testing 16:33:44 boazsender: the APIs that are not using webdriver are only partially tested at all... are bluetooth, xr... 16:33:54 boazsender: those are different to test, maybe not harder 16:34:00 jgraham: In Chromium CDP is implemented directly in the product code. If WebDriver-bidi was exposed directly in the product code rather than needing an abstraction layer, this might solve the problem somewhat 16:34:03 boazsender: how to mock them across the system 16:34:12 boazsender: is unique 16:34:53 jgraham: "product code" here being the implementation of the web-exposed spec piece in C++ code 16:35:31 boazsender: BTT could maybe be more welcoming when folks come and want to automate their tests 16:36:00 smcgruer_[EST], you wanted to ask if webdriver bidi will actually make adding new webdriver features easier (whats the reduced overhead for feature engineers?) 16:36:26 BTT is next week; idk if it's on the TPAC agenda 16:36:47 smcgruer_[EST]: hoping bidi can help 16:37:21 smcgruer_[EST]: in summary, work to do to make webdriver path... know where the painpoints are and work to fixing them. still webdriver as the main solution 16:38:12 ScribeNick: Hexcles 16:38:25 Topic: Screen reader automation (for ARIA-AT) 16:38:37 zcorpan: there's a blog post introducing ARIA-AT 16:39:29 https://bocoup.com/blog/interoperability-testing-for-assistive-technologies-and-the-web-platform 16:39:49 ... describing ARIA-AT 16:40:43 ... There's been existing effort testing a11y API (e.g. testing the a11y tree), but not at the screen reader level (the end users). 16:41:34 ... One problem is that ARIA-AT tests are manual. No one knows how to automate screen readers. Prototype TBD next month with NVDA 16:41:42 q+ 16:42:44 ... I want to standardize a protocol for automating screen readers similar to webdriver. This is a heads up on this upcoming work. 16:43:35 ... We don't want "pixel-perfect" interop for screen readers but want to interop on user interface, functionalities, etc. 16:44:00 ... We haven't started yet. Anyone interested is welcome to collaborate. 16:44:02 q+ to emphasize Security should be central through the whole process (re: automating AT) 16:44:11 (14wpt) [PR] chromium-wpt-export-bot 03merged 13#26173 by chromium-wpt-export-bot into 07master: Reland "Reenable scroll-animations/element-based-offset tests" - https://git.io/JT4rJ 16:44:44 jgraham: It's clear to me what kind of API surface you're looking for. 16:45:12 s/it's clear/it's not clear/ 16:47:48 zcorpan: It needs to be specific to each screen reader e.g. w.r.t. what it says, so perhaps specific assertions depending on screen readers. Though in some cases, screen readers may agree. WebDriver may have similar issues (special browser-specific handling). 16:47:57 q+ 16:48:15 jgraham: yeah webdriver does have some special magic for each browser (e.g. form elements), which is tricky. 16:48:15 jcraig, you wanted to emphasize Security should be central through the whole process (re: automating AT) 16:49:06 jcraig: I work on VoiceOver. I want to bring up the security concern. a11y is an attack vector as it can do whatever users can do. 16:49:27 ... Apple has taken a lot of caution w.r.t. automation. 16:50:08 ... Locking down the security of VocieOver is one blocker for shipping automation. 16:50:44 muodov: Is the plan to work together with the vendors of a11y technologies? 16:51:06 zcorpan: yes we'll reach out to vendors, hopefully working together. 16:51:15 smcgruer_[EST], you wanted to ask if Simon has thoughts on what merging with wpt looks like 16:51:45 s/Apple has taken a lot of caution w.r.t. automation./Apple has taken a lot of caution w.r.t. automation for example, Xcode will only automate a dev copy of your own app, not other system apps./ 16:52:05 smcgruer_[EST]: what kind of integration with WPT do you have in mind? 16:52:58 zcorpan: for right now, it won't be useful to integrate. Down the line, it'd be useful to have results no wpt.fyi and source code in the same repo if it makes sense. 16:53:26 ... fundamentally we're automating screen readers that automate browsers, so it seems compatible and related. 16:53:47 q+ 16:53:52 (14wpt) [PR] domenic requested 13#26175 merge into 07master: Test that dynamic import() is disallowed in paint worklets - https://git.io/JT47u 16:54:35 s/I want to bring up Security as a goal for ARIA-AT. Assistive Technology (on any platform) is authorized to do whatever any human user can do, so occasionally it's exploited as a Security attack vector. Since you are working on NVDA first, closely coordinate on Sec with Windows Accessibility at Microsoft, and the NVDA core team./ 16:55:16 RRSAgent: Make minutes v2 16:55:16 I have made the request to generate https://www.w3.org/2020/10/19-testing-minutes.html jgraham 16:55:47 s/I work on VoiceOver. I want to bring up the security concern. a11y is an attack vector as it can do whatever users can do./I want to bring up Security as a goal for ARIA-AT. Assistive Technology (on any platform) is authorized to do whatever any human user can do, so occasionally it's exploited as a Security attack vector. Since you are working on NVDA first, closely coordinate on Sec with Windows Accessibility at Microsoft, and the NVDA core 16:55:47 team./ 16:56:18 rrsagent, make minutes 16:56:18 I have made the request to generate https://www.w3.org/2020/10/19-testing-minutes.html jcraig 16:57:31 boazsender: re WPT integration, we don't know yet, but it'd be great to share tools. Some aspects of a11y might be outside of the web platform. The spirit of the WPT is in browsers and the things around. There were discussions of bringing JS testing. ARIA-AT is similar. It's also similar to automating devices, etc. So we'd like some conversions on this theme. We're also building yet another manual testing system, in addition to CSSWG's (by plinss). 16:58:57 scribenick: zcorpan 16:59:46 topic: surfacing high priority bugs / how to identify test coverage for specs 16:59:57 jgraham: actually two topics... 17:00:12 jgraham: expertise in this group, makes sense to talk here 17:00:12 q+ to comment 17:00:22 jgraham: can still also discuss in a breakout 17:00:23 MikeSmith, you wanted to comment 17:00:24 ack MikeSmith 17:00:34 MikeSmith: as far as the second part goes, coverage 17:00:45 MikeSmith: perma-want that we've talked abou tforever 17:00:49 MikeSmith: haven't made much progress on 17:01:03 q+ to give a summary of the things I remember 17:01:14 MikeSmith: a good strategy would be to look at what we've tried so far 17:01:21 MikeSmith: what benefit has it provided 17:01:25 jgraham, you wanted to give a summary of the things I remember 17:01:32 MikeSmith: continue doing it? other ways ? 17:01:51 jgraham: historically, 2 similar but different conceptualisations 17:01:57 jgraham: spec-centric model 17:02:04 jgraham: question is: for each testable assertion 17:02:09 jgraham: does it have a related test case 17:02:32 jgraham: typically the approach that people advocate ... bikeshed has some support ... 17:02:42 jgraham: either link from test to the relevant spec section 17:02:49 jgraham: in the test 17:03:10 q+ to remember that CSSWG has some stuff intended to help identify gaps in test coverage; wonder how well that has been working for them? 17:03:11 jgraham: traditional problem with that, reason we often push back, is that when you insist people add that metadata... 17:03:18 jgraham: people refuse to write the test 17:03:33 jgraham: if you don't have automation to update it, then they get out of date 17:03:49 jgraham: spec authors aren't keen to update test metadata to land their spec PRs 17:04:04 jgraham: CSS trying now, in the spec you link to the test 17:04:19 jgraham: in the process you're encouraged to find all the tests and add them to the spec 17:04:27 jgraham: but still have hte inverse problem 17:04:42 jgraham: CSS WG are trying this out... 17:04:56 jgraham: statements in the spec that links to test is ??? 17:05:18 jgraham: other approach is code coverage, in the implementation how much is used by running the test 17:05:29 miketaylr has joined #testing 17:05:30 jgraham: limitations... doesn't tell you about code you didn't write 17:05:39 jgraham: doesn't tell you you didn't implement a section of the spec 17:05:45 jgraham: also how do you compare between browsers 17:05:48 q+ to realize that step 1 is actually identifying gaps in coverage, and can do we do more tooling to help with that at least? 17:05:51 jgraham: or how do you compare with the spec 17:06:33