13:51:50 RRSAgent has joined #wot-pf 13:51:50 logging to https://www.w3.org/2022/11/09-wot-pf-irc 13:51:58 meeting: WoT Plugfest/Testing 13:56:21 present+ Kaz_Ashimura 13:58:39 mlagally_ has joined #wot-pf 13:59:00 ryuichi has joined #wot-pf 14:00:08 agenda: https://www.w3.org/WoT/IG/wiki/PlugFest_WebConf#Agenda_09.11.2022 14:00:20 present+ Ege_Korkan, Michael_Lagally 14:02:05 present+ Tomoaki_Mizushima 14:02:48 Mizushima has joined #wot-pf 14:04:57 regrets+ McCool, Sebastian 14:05:50 scribenick: kaz 14:05:55 q+ 14:06:25 topic: Agenda 14:06:50 kaz: We need to clarify the targets for this call and also the Testfest in December 14:07:21 ack k 14:07:35 ... I mean which specs to be discussed 14:08:18 ... I think we should prioritize Architecture at least for today 14:08:35 ml: OK. Let's start with Architecture 14:08:37 topic: Minutes 14:08:54 -> https://www.w3.org/2022/11/02-wot-pf-minutes.html Nov-2 14:09:36 chair: Lagally 14:09:49 approved 14:10:11 topic: WoT Architecture Implementation Report 14:10:36 -> https://w3c.github.io/wot-architecture/ WoT Architecture 1.1 spec 14:10:56 -> https://w3c.github.io/wot-architecture/testing/report11.html Implementation Report for WoT Architecture 1.1 14:11:15 ml: some duplication here 14:11:28 ... e.g., Thing Description Playground 14:11:52 ek: will look into the problem 14:12:26 ... seems I forgot to fix the title for one of the two entries 14:12:29 ml: ok 14:13:06 -> https://w3c.github.io/wot-architecture/testing/report11.html#manual_validation_results Test results table 14:13:28 ml: (skims the table) 14:13:33 q+ 14:13:41 https://github.com/w3c/wot-testing/pull/476 this fixes the previous title duplication issue we have seen 14:13:58 @@@ to be moved 14:16:17 kaz: before diving into the detailed check, we should revisit the issue I pointed out 14:16:58 ... the question is that the order of the assertions within this result table are sorted by the ID names 14:17:18 ... but they should be sorted by the order of their appearance 14:17:37 ... i.e., the order of the text within the specification should be preserved for manual assertions 14:17:40 ml: agree 14:17:51 ... let's create an Issue for wot-testing 14:18:03 s/.../kaz:/ 14:18:17 ... and ask McCool to modify the table for manual assertions 14:18:19 ml: ok 14:18:30 ... (creates an Issue for wot-testing) 14:19:32 -> https://github.com/w3c/wot-testing/issues/477 wot-testing Issue 477 - Order of assertions in Implementation Reports should follow order in specifications 14:19:45 ml: let's check the assertions then 14:20:51 ... assertions on security to be made non-normative if we don't get sufficient implementations 14:21:30 q+ 14:23:01 kaz: note that we need to update the SoTD section with the list of the features at-risk 14:23:03 ack k 14:23:28 ml: ok 14:23:39 ... (visits the static HTML for CR) 14:24:17 https://cdn.statically.io/gh/w3c/wot-architecture/main/publication/ver11/3-cr/index.html 14:24:28 s/https/-> https/ 14:24:37 s/html/html static HTML for CR/ 14:25:46 kaz: it would make sense to compare the list within SoTD and the Implementation Report to make sure 14:25:48 ml: ok 14:26:08 -> https://w3c.github.io/wot-architecture/publication/ver11/3-cr/Overview.html#sotd Status section 14:26:19 -> https://w3c.github.io/wot-architecture/testing/report11.html#manual_validation_results Implementation Report 14:27:59 (16 assertions vs 16 assertions) 14:28:06 ml: confirmed the match 14:28:40 topic: December Testfest 14:28:52 ml: would like to talk about the target for the December Testfest 14:28:58 q+ 14:29:22 ack e 14:29:46 ek: note that we need to update the Testimonial section for the Architecture Implementation Report 14:29:55 s/ml: would like to talk about the target for the December Testfest// 14:30:04 s/topic: December Testfest// 14:30:06 rrsagent, make log public 14:30:12 rrsagent, draft minutes 14:30:12 I have made the request to generate https://www.w3.org/2022/11/09-wot-pf-minutes.html kaz 14:31:28 ml: would like to ask McCool to handle that 14:32:20 q+ 14:33:27 kaz: we've already started some discussion on how to deal with testimonials for 1.1 version specs 14:33:36 ml: yes, let's continue the discussion during the main call 14:34:03 topic: December Testfest 14:34:15 -> https://github.com/w3c/wot-testing/tree/main/events/2022.12.Online December Testfest 14:35:13 q+ 14:35:53 q+ 14:37:07 ack k 14:37:29 kaz: technically, we should clarify the requirements and the descriptions within the spec first 14:37:35 ... and then think about how to test them 14:38:03 ... when we think about how to test them, we need to consider which assertions to be tested how, automatically or manually 14:38:24 ... the point is that not all the assertions need to be tested automatically using some smart tool 14:38:27 ack e 14:38:33 ml: agree 14:39:07 ek: what about the UI for accessibility? 14:39:17 ... e.g., speech synthesis 14:39:38 ... note that from my viewpoint, human users are not Consumers 14:39:42 q+ 14:40:15 ... that might be a philosophical question, though 14:40:36 ml: the question is how to process the data 14:44:35 kaz: yeah 14:45:05 ... editor tools and browser tools which include human controls are also WoT implementations 14:46:35 ... the more important here is improving the text within the WoT Profile spec before extracting assertions 14:47:08 ... so we should add that kind of preparation tasks as well for WoT Profile 14:47:27 i/technically/subtopic: WoT Profile/ 14:48:42 ack k 14:48:46 q+ 14:48:54 ack k 14:50:16 ml: (then move ahead to the discussion on the test cases) 14:50:57 ... standalone test of the Consumer assertions first? 14:51:12 q+ 14:53:22 kaz: we should clarify the whole structure of the testing specifically for the WoT Profile spec 14:53:46 ... some of the assertions can be tested based on the existing tests for the other specs like TD 14:53:56 ... what still need to be tested for WoT Profile? 14:54:50 ek: we need actual device implementations 14:54:59 ... maybe Toumura-san, etc., can provide some 14:55:32 ml: not necessarily fully compliant to WoT Profile? 14:55:33 q+ 14:57:32 ack k 14:57:43 kaz: sorry but getting kind of confused 14:59:16 ... what we should discuss as the starting point for Testfests is rather how to get the result CSV 14:59:34 ml: right 14:59:46 ... (adds note on people...) 14:59:56 mizu: we should clarify who would participate 15:00:15 ... and who would tackle which assertions and provide results for them 15:00:20 ack mizu 15:00:45 ... it's difficult for developers to fill in the results during the Testfest itself 15:01:14 ... we should assign assertions to the participants in the Testfest 15:02:05 ... I'm concerned about the output from the Testfest 15:02:16 ml: yeah, I share the same concern... 15:02:46 q+ 15:05:52 ack k 15:08:39 kaz: we should clarify which assertions from the WoT Profile to be tested how 15:09:02 ... WoT Profile is basically consolidated description of the whole WoT mechanism 15:09:19 ... so we should be able to refer to the test results of the other WoT specs 15:09:52 ... though newly added clarifications by the WoT Profile itself need to be tested by the WoT Profile spec itself 15:09:55 ml: ok 15:10:45 -> https://github.com/w3c/wot-testing/tree/main/events/2022.12.Online#goals-and-priorities Lagally updates the "Goals and Priorities" section of the Testfest page 15:10:53 [adjourned] 15:10:59 rrsagent, make log public 15:11:06 rrsagent, draft minutes 15:11:06 I have made the request to generate https://www.w3.org/2022/11/09-wot-pf-minutes.html kaz