21:50:43 RRSAgent has joined #qa 21:50:48 logging to https://www.w3.org/2025/11/10-qa-irc 21:50:48 RRSAgent, do not leave 21:50:49 RRSAgent, this meeting spans midnight 21:50:49 RRSAgent, make logs public 21:50:51 Meeting: Restarting W3C QA - Quality Assurance 21:50:51 Chair: Sarven Capadisli 21:50:51 Agenda: https://github.com/w3c/tpac2025-breakouts/issues/36 21:50:51 Zakim has joined #qa 21:50:52 Zakim, clear agenda 21:50:52 agenda cleared 21:50:52 Zakim, agenda+ Pick a scribe 21:50:53 agendum 1 added 21:50:53 Zakim, agenda+ Reminders: code of conduct, health policies, recorded session policy 21:50:54 agendum 2 added 21:50:54 Zakim, agenda+ Goal of this session 21:50:55 agendum 3 added 21:50:55 Zakim, agenda+ Discussion 21:50:55 agendum 4 added 21:50:55 Zakim, agenda+ Next steps / where discussion continues 21:50:57 agendum 5 added 21:50:57 Zakim, agenda+ Adjourn / Use IRC command: Zakim, end meeting 21:50:57 agendum 6 added 21:50:57 breakout-bot has left #qa 23:30:43 dom has joined #qa 23:31:08 gerald has joined #qa 23:31:14 kenneth has joined #qa 23:32:46 Present+ Hidde, Manu, AlanS, Sarven, Dom, Jean-Gui, Gerald, Denis, Carine, Aki, Florian, Jeffrey, Pierre-Antoine 23:32:48 scribe+ 23:33:20 denis has joined #qa 23:34:27 caribou has joined #qa 23:34:32 Slideset: https://csarven.ca/presentation/restarting-w3c-qa 23:34:43 mikbar has joined #qa 23:34:55 s/tation/tations/ 23:35:07 Sarven: trying to get a sense from the room about QA at W3C 23:35:13 [slide 3] 23:35:24 jyasskin has joined #qa 23:35:25 Sarven: history of QA Activity in W3C started in 2000 23:35:29 present+ 23:35:50 ... nowadays, WGs are expected to produce open test suites; charters have explicit success criteria on interop 23:35:54 florian has joined #qa 23:36:16 ... typically 2 independently developed software that implement the features described by the specs 23:36:27 ... these test suites lead to interop report by groups 23:36:48 ... and ideally, these test suites remain useful and functional long after the spec has been developed 23:37:07 ... this isn't only about QA, but about how specs are developed 23:37:25 [slide 4] 23:37:53 Sarven: the former QA WG/IG produced a number of documents: QA Framework and its handbook, spec guidelines, variability in specs 23:38:16 ... a lot of if is how to express different layer of structure in specs and how they relate to test suites aligned with the specs 23:38:28 ... including considerations about how to express conformance, requirements 23:39:47 [reviewing quickly spec guidelines, variability in spec, testing faq] 23:40:02 [slide 5] 23:40:29 astearns has joined #qa 23:40:35 sarven: adjacent work include WPT (used by a large number of specs, but not all W3C specs are in scope of it either) 23:40:57 ... there was also an HTTP vocabulary, a schema to describe test results (EARL) 23:41:28 ... Spec Terms is a vocabulary to express the significant units in specifications 23:42:00 ... the Solid QA work builds on the the W3C QA work and tries to some of the loose ends - it came out of the Solid CG, but it's not particularly tied to Solid itself 23:42:11 [slide 6] 23:42:55 Sarven: Technical Reports have a long list of significant units in specifications - a number of which could be made machine readable 23:44:49 [slide 7] 23:45:12 Sarven: many (but not all) specs use controlled vocabulary to express normative content and distinguish informative content 23:45:31 ... that includes but isn't restricted to RFC2119/8174 keywords 23:45:49 [slide 8] 23:45:58 Sarven: a few questions to the room: 23:46:18 ... which product classes defined in a spec are required to work together? how can we extract that from a test suite development perspective? 23:46:33 ... how do we demonstrate actual interoperation? incl consideration for optional features 23:47:10 q? 23:47:24 ... who ensures interop as specs evolve? 23:47:50 [slide 9] 23:48:18 Sarven: helping anyone to contribute to spec development is a way to improve QA on specs 23:48:43 ... there are no limitations to whcih tool can be used to write specs - with respec and bikeshed popular options 23:48:52 ... as long as the result conform to pubrules 23:49:17 ... current tools focus more on publishing than authoring - their accessibility for the latter isn't great 23:49:25 ... machine readability of spec is quite limited at the moment 23:53:55 [showing screencast of an improved spec authoring environment based on dokieli] 23:54:03 q+ jyasskin 23:54:11 q+ 23:54:30 q+ to ask how a testing tool uses a Requirement link that's as big as "MUST conform to HTTP Semantics"? 23:58:36 manu_ has joined #qa 23:58:39 q? 23:58:44 present+ 23:58:51 jyasskin: restarting QA sounds good and do work in that direction 23:58:52 Jemma has joined #qa 23:59:02 present+ 23:59:27 ... in terms of the semantics representation of the "MUST" statements; one of them is "MUST conform to HTTP semantics" - that's too big for a single test case; how does this help with developing a test suite? 23:59:37 ... that might be guidance a new QA activity could help develop 23:59:56 Sarven: writing requirements is more an art than a science 00:00:23 q+ to ask about value proposition of QA to implementers -- drives requirements. 00:00:23 ... here the assumption that when you build on top of HTTP, it would assume the underlying implementation would be tested against the existing HTTP suite 00:00:53 pchampin has joined #qa 00:00:57 Jemma has joined #qa 00:01:28 ... one requirement might have one or more test cases - the requirements needs to be specific enough, but may not hash out all the details you might need, and part of it might be optional 00:01:34 q+ 00:01:46 ack jyasskin 00:01:46 jyasskin, you wanted to ask how a testing tool uses a Requirement link that's as big as "MUST conform to HTTP Semantics"? 00:02:16 jyasskin: that points to another potential output: when a spec is used as a dependency, how to build your test suite to make it useful for specs that dpeend on yours 00:02:20 ack florian 00:02:41 florian: the premise here is we should restart QA - how we do QA has evolved 00:02:52 ... we're doing a lot of testing, thinking on interop 00:03:49 q+ to mention non-browser specs 00:04:07 q+ 00:04:27 ... if we were to restart work in this space, it would be useful to be more specific on what exactly needs more focused attention 00:05:25 sarven: fair enough - I have tried to identify some of the things that needs more attention 00:05:38 florian: clarifying the assumptions of the state of the world and what wish it was 00:05:45 q? 00:05:50 ack manu_ 00:05:50 manu_, you wanted to ask about value proposition of QA to implementers -- drives requirements. 00:05:55 Jemma4 has joined #qa 00:06:21 c/me an I get the url for the doc on the screen or the previous one? 00:06:27 manu: what is the value proposition for QA at W3C? it's almost thought about testing the spec, whereas what's more useful having test suite that are testing implementations for interop 00:06:57 ... the conformance statements in the specs are important, but if the test suite is only for spec, its value disappear quickly 00:07:14 Jemma4: https://csarven.ca/presentations/restarting-w3c-qa 00:07:17 ... the continuous testing of what WPT is doing is good 00:07:23 ... but not all specs are in this category 00:08:03 q? 00:08:05 scribe+ 00:08:21 ack dom 00:08:30 dom: The QA activity stopped in 2017 -- low participation. 00:08:34 dom: QA activity stopped in 2007, because there was a low level of participation, and tended to be staff-heavy with excellent but limited member participation 00:08:46 ... Lot of practice has evolved in the meantime. 00:09:13 ... 2 tracks of thoughts, both of which are interesting. 1) spec engineering: what have we learned about how to write and maintain specs that helps increase the final interoperability 00:09:57 ... some of the work of Francios and I have done in specref, we've extracted information from specs and tied them to implementations in various ways, ensured they're aligned. Semi-formal analysis of algorithms in browser specs. 00:10:48 ... There's a whole set of things we've been doing on the side, which has grown over time, with useful outcomes. But it's a side-project. Very focused on browser specs, partly because there's more spec infrastructure on browser specs, and there are many more of them. Fairly confident that there are useful things we could broaden to other ecosystems. 00:11:57 ... 2) not stopping at the spec level. Specs are a means to an end, but we really care about interoperability. We've learned enormously. WPT has grown from a side project to a huge project that drives a lot of work. On top of it, Interop has proved really interesting to not stop at just testing the spec. After the spec there's a lot more, and need to keep iterating. 00:12:36 ... WebDX CG work on web features and baseline. These are browser-focused, but the lessons deserve being extracted, to see if they apply to other ecosystems. These are 2 potential interesting tracks that could deserve more structured attention. 00:12:49 ... WG? 2 WGs? CG? 00:12:54 q+ to ask about concrete tooling that could be helpful 00:12:55 q? 00:13:39 jyasskin: browser specs and the other ecosystems have evolved at a different pace: browser specs have put a lot of effort in their QA efforts with WPT and other tools 00:13:50 ... which sometimes create confusion in terms of mismatched expectations 00:13:58 ... more cross-polination would be useful 00:14:14 Jem_ has joined #qa 00:14:20 ... a lot of what Sarven shows was around non-browser ecosystem - there would be value in talking to each other 00:14:21 q- 00:14:26 scribe- 00:14:43 sarven: re RDF, I didn't mean to suggest a particular format or convention is needed 00:14:50 ... whatever tool gets us where we need 00:14:53 +1 00:15:33 ack florian 00:16:01 florian: manu mentioned the idea of test suites only used to get to Rec 00:16:14 ... that's not what we do for WPT, but in terms of the Process, that's correct 00:16:47 And you can see that a bit in how few web specs get to REC. 00:16:49 ... in WPT, the spec is a mean to understand why test should fail or not 00:17:20 ... there may need to be a bit of a cultural shift in that regard - e.G. with more emphasis on test suites in charter 00:19:13 ... re dokieli, I'm not sure how realistic it is to expect spec authors to take such a structured approach - e.g. writing requirements in a way that they can be extracted out of context is non trivial, and might be hard to get applied at scale 00:19:25 ... it may still be useful as a way to motivate getting closer to that approach 00:20:08 sarven: the authoring part is a gap I'm highlighting - I don't feel we have a robsust ecosystem for authoring; you're expected to use certain publishing tools and go through a bunch of hoops 00:20:25 ... we can increase the possibly for contributions through more accessible authoring tools 00:22:00 siri9 has joined #qa 00:23:06 florian: even I had a magic tool that did everything perfectly - this would require a lot of additional work from the author 00:23:46 Some specs (like WOFF, I believe) have markup that isolates testable assertions 00:24:28 https://speced.github.io/bikeshed/#testing has a way to point from a spec to its tests. 00:24:32 ... in WPT, the CSSWG used to be on more-metadata camp with link back to specific requirements; we've been given up to promote ease of test writing 00:25:54 the WebRTC spec has a "toggle test annotation" in the ReSpec pill that highlight conformance requirements and whether they have known associated WPT test cases https://w3c.github.io/webrtc-pc/ 00:26:17 sarven: this is a pay-as-you-go approach where additional efforts provide progressive additional benefits 00:26:31 ack manu_ 00:26:31 manu_, you wanted to ask about concrete tooling that could be helpful 00:26:49 manu: is there a base level tooling that a QA group could work on? 00:27:14 q+ to suggest a next step 00:27:18 ... e.g. adding a conformance requirement identification feature in respec to help cross-linking from test cases 00:27:37 ... not necessarily for WPT, but for other specs 00:27:38 ... we had to rewrite the test infrastructure for VC/DID 00:27:57 ... we think it's useful, but we don't know if it's reproducible and worth reproducing 00:29:06 jyasskin: +1 on spec authoring formats to help with normative statements 00:29:45 ... having a forum to discuss these approaches would be good - a CG would be a good way to start 00:29:58 -> https://www.w3.org/TR/test-methodology/ A Method for Writing Testable Conformance Requirements 00:29:59 karlcow has joined #qa 00:30:26 Sarven: having a way to id conformance requirements sounds like a good addition 00:30:35 FTR, Respec already has this https://respec.org/docs/#data-tests (but this is the other way around, normative statement → test) 00:30:53 ... there are a number of potential contributors that can't use the authoring tools, we should lower that bar 00:33:22 sarven: sense of the room of interest in such a CG? 00:33:30 florian: I strongly suspect there is value 00:33:44 jyasskin: I'd be happy to help draft the proposal and bring WPT people 00:34:21 I did suggest that csarven needs to drive it. :) 00:34:24 manu: it would be good to encourage WG test facilitators to participate in such a group 00:34:46 RRSAgent, draft minutes 00:34:47 I have made the request to generate https://www.w3.org/2025/11/10-qa-minutes.html dom 00:43:21 denis has left #qa 00:49:19 karlcow has joined #qa 00:51:11 manu_ has joined #qa 01:19:35 tidoust has joined #qa 01:22:02 RRSagent, bye 01:22:02 I see no action items