20:04:07 <RRSAgent> RRSAgent has joined #aria-at
20:04:11 <RRSAgent> logging to https://www.w3.org/2024/02/22-aria-at-irc
20:04:14 <jugglinmike> rrsagent, make log public
20:04:22 <jugglinmike> Zakim, start the meeting
20:04:22 <Zakim> RRSAgent, make logs Public
20:04:24 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
20:04:37 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
20:04:56 <Joe_Humbert> Joe_Humbert has joined #aria-at
20:05:06 <Matt_King> Matt_King has joined #aria-at
20:05:14 <Joe_Humbert> present+
20:05:17 <jugglinmike> present+ jugglinmike
20:05:20 <jugglinmike> scribe+ jugglinmike
20:05:31 <jugglinmike> present+ IsaDC
20:05:36 <jugglinmike> present+ James_Scholes
20:05:41 <jugglinmike> present+ Matt_King
20:05:48 <jugglinmike> present+ howard-e
20:06:03 <jugglinmike> present+ Sam_Shaw
20:06:03 <Sam_PAC> Sam_PAC has joined #aria-at
20:06:12 <jugglinmike> present- Sam_Shaw
20:06:28 <jugglinmike> present+ Sam_PAC
20:06:54 <jugglinmike> Topic: Review agenda and next meeting dates
20:07:09 <jugglinmike> Matt_King: Next meeting will be February 28
20:07:19 <jugglinmike> Topic: Automation subgroup meeting planning
20:07:30 <jugglinmike> Matt_King: I saw an e-mail about a poll
20:08:01 <jugglinmike> present+ Joe_Humbert
20:08:11 <howard-e> howard-e has joined #aria-at
20:08:12 <jugglinmike> Joe_Humbert: Yup, it was sent to this group's e-mail list
20:08:24 <jugglinmike> jugglinmike: The poll is here: https://forms.gle/CqLE3ujT3UfEDtqC9
20:08:36 <howard-e> present+
20:09:11 <jugglinmike> jugglinmike: This poll is open until Tuesday, Feb 27th, at 9:00 AM US Pacific Time.
20:09:23 <jugglinmike> Matt_King: Anyone who's interested in automation, please submit a response there!
20:09:38 <jugglinmike> Sam_PAC: I submitted a single response to represent PAC--both James_Scholes and myself
20:09:54 <jugglinmike> Topic: Set schedule for toggle and dialog
20:10:12 <jugglinmike> Subtopic: Toggle button plan: set date to resolve
20:11:15 <jugglinmike> github: https://github.com/w3c/aria-at/issues/1031
20:11:51 <jugglinmike> Matt_King: Given the consenses we reached last time, this is a one-word change to one test plan
20:12:16 <jugglinmike> Matt_King: Because the assertion changes, verdicts for any tests which include role will become invalid
20:12:34 <jugglinmike> IsaDC: All assertions that say "toggle button", I believe
20:12:49 <jugglinmike> Matt_King: Let's set a schedule for figuring this out
20:13:33 <jugglinmike> Matt_King: I will assign this re-assign this from me to IsaDC
20:13:44 <jugglinmike> s/assign this //
20:14:04 <jugglinmike> James_Scholes: We're doing "toggle button" as two words, correct?
20:14:08 <jugglinmike> IsaDC: Yes
20:14:09 <jugglinmike> Matt_King: Yes
20:14:41 <jugglinmike> Matt_King: So that new version of the "toggle button" test plan won't be available until you make the change, merge it, and advance the test from "R&D" to "review"
20:15:54 <jugglinmike> Matt_King: The only time we have the option to preserve data is if there is an incomplete run in the test queue
20:16:45 <jugglinmike> Matt_King: It seems like we'd want a way to copy all data for tests which haven't changed whenever we add a new test plan to the test queue. Then again, that seems like something that ought to be done automatically
20:17:15 <jugglinmike> James_Scholes: I guess it depends on the complexity of the changes between the two test plan versions
20:17:28 <jugglinmike> Matt_King: I don't know where that fits in with our roadmap
20:17:41 <jugglinmike> Matt_King: For now, though, for anybody assigned to NVDA with Chrome, we do have the bot available
20:18:09 <jugglinmike> Matt_King: And as far as I can tell, it's 100% reliable. I have not found a case where the NVDA bot gives me different output, and I've done this a number of times for three different test plans
20:19:07 <jugglinmike> jugglinmike: I can't say when Firefox support will be available, but I can see about getting an estimate for next week
20:19:54 <jugglinmike> IsaDC: Would anyone like to re-run all of "toggle button" from scratch?
20:20:22 <jugglinmike> Joe_Humbert: I can re-run the ones that I originally submitted--VoiceOver and Safari
20:21:13 <jugglinmike> Matt_King: A two-week timeframe would be great. That's because my next meeting with Vispero is three weeks from yesterday
20:21:28 <jugglinmike> present+ Hadi
20:21:36 <jugglinmike> Hadi: I volunteer to run in Chrome and JAWS again
20:22:30 <jugglinmike> [Matt_King reiterates the rationale for re-running the "toggle button" tests]
20:27:57 <jugglinmike> howard-e: Copying data in that way is currently possible, actually, at least for the Test Plan Reports which have not been marked "final"
20:30:07 <jugglinmike> Michael_Fairchild: in order to save time, could we ask the database administrators to manually copy the test results?
20:31:16 <jugglinmike> howard-e: That's feasible, but I'm also saying that there's a high chance that the data for the unchanged tests will be copied automatically
20:34:52 <jugglinmike> Matt_King: It sounds like the plan--first we get the new version. IsaDC does that.
20:34:57 <jugglinmike> Matt_King: That gets moved to Draft Review
20:35:44 <jugglinmike> Matt_King: Actually, howard-e, before she does that, she would deploy the latest changes in staging (following my completing my functional testing)?
20:36:23 <jugglinmike> Matt_King: If we deployed that, and then IsaDC does her part, then we would see some results carry forward automatically
20:36:49 <jugglinmike> Matt_King: After that, should we have howard-e try to manually copy some of Joe_Humbert's test results forward?
20:37:18 <jugglinmike> James_Scholes: I think we should. It would take multiple hours to re-run this test
20:37:48 <jugglinmike> James_Scholes: Should we add the new version to the queue now? Or should we do that later?
20:39:14 <jugglinmike> [Matt_King investigates the current behavior of the application]
20:40:04 <jugglinmike> Matt_King: IsaDC can make the change, and she can merge it, but do not advance it to draft review until howard-e has updated the app!
20:40:14 <jugglinmike> Matt_King: And howard-e will not update the app until Monday
20:40:17 <jugglinmike> IsaDC: Got it
20:40:44 <jugglinmike> James_Scholes: Should I un-assign you from JAWS and NVDA, Matt_King?
20:40:49 <jugglinmike> Matt_King: Sure
20:41:06 <jugglinmike> James_Scholes: Got it (because we already have results from IsaDC and Hadi)
20:41:48 <jugglinmike> s/because we already/I asked to un-assign you because we already/
20:42:30 <jugglinmike> Matt_King: There are some problems with this feature, and I've raised an issue about them
20:43:20 <jugglinmike> Matt_King: It's related to the "trend reporting" project that Bocoup is going to be working on next quarter
20:43:55 <jugglinmike> James_Scholes: the status says "draft", and that means there are no conflicts (because if there were conflicts, the status would describe them)
20:44:11 <jugglinmike> Matt_King: The order of events:
20:44:22 <jugglinmike> Matt_King: 1 - IsaDC is going to merge a new version of the test plan
20:44:36 <jugglinmike> Matt_King: 2 - I am going to test what's in staging and give the green light to howard-e
20:44:52 <jugglinmike> Matt_King: 3 - howard-e will release the changes to production
20:45:06 <jugglinmike> Matt_King: 4 - IsaDC can advance the new version from R&D to Draft Review
20:45:35 <jugglinmike> James_Scholes: after the new version is advanced, it doesn't automatically appear in the test queue, does it?
20:45:41 <jugglinmike> Matt_King: Yes, it does
20:47:17 <jugglinmike> Matt_King: The status of the new (February 23rd) version, will show "5 of 8 complete" (or something similar)
20:48:32 <jugglinmike> James_Scholes: that all sounds good. It does leave a feature result outstanding, though: to copy results from a test plan with a different status
20:48:37 <jugglinmike> Matt_King: Yes, that's right
20:48:55 <jugglinmike> Matt_King: We have an issue open for that, and it's related to the "trend reporting" work
20:49:11 <jugglinmike> Subtopic: Schedule for refactor and review of dialog test plan
20:49:29 <jugglinmike> Matt_King: Would it be reasonable for the meeting two weeks from today that we have people sign up for the testing of "dialog"?
20:49:39 <jugglinmike> IsaDC: I would like to take a look and then discuss with you next Tuesday
20:49:57 <jugglinmike> Matt_King: That sounds good. We may want several days of reviewing it even before we ask people to run the test
20:50:12 <jugglinmike> James_Scholes: I would love to see a test plan for the dialog element
20:50:23 <jugglinmike> Matt_King: I agree, though it's a separate topic
20:52:22 <jugglinmike> Topic: Review proposal for test format V2 enhancement
20:52:34 <jugglinmike> Matt_King: This is going to get technical, fast
20:53:24 <jugglinmike> Matt_King: Right now, we have a way of making it so that you can have several assertions assigned to a test (e.g. an assertion for "role" and one for "state"), and then an additional assertion (e.g. for when the AT changed mode)
20:53:45 <jugglinmike> Matt_King: We can have another assertion on the test, but we only expect that assertion to happen for "tab"
20:54:42 <jugglinmike> Matt_King: The way we currently specify that, is we put the assertion on the test (the "mode switch assertion") and then place assertion exception on the specific commands where that assertion does not actually apply
20:55:05 <jugglinmike> Matt_King: The way we specify the assertion is we give it "zero" priority (prefixing it with the characters "0:")
20:55:14 <jugglinmike> Matt_King: This proposal is to flip that logic on its head
20:55:52 <jugglinmike> Matt_King: In the test file, we will put the mode switch assertion in the test file, where it will mean, "don't include this for any commands by default"
20:56:09 <jugglinmike> Matt_King: If you want it on a command, you would then add that assertion identifier for that command only
20:56:25 <jugglinmike> Matt_King: That means that there are a lot fewer exceptions and that the exceptions are more clear
20:56:45 <jugglinmike> Matt_King: As a result, you wouldn't see any exceptions at all in VoiceOver (since it doesn't do mode switching)
20:57:21 <jugglinmike> Hadi: It sounds like we are mixing objective testing and specific screen reader features
20:58:00 <jugglinmike> James_Scholes: this is very much a test authorship syntax change. As a tester, you will probably never see a difference--it all takes place in the underlying files
20:59:01 <jugglinmike> James_Scholes: Are you claiming that these two approaches should be mutually exclusive, or should it support both?
20:59:08 <jugglinmike> Matt_King: I think it should support both
20:59:26 <jugglinmike> James_Scholes: Good. I think that's important because there are cases where the current syntax is more appropriate
21:00:39 <jugglinmike> Matt_King: Something else that happens with Radio Groups is that when you navigate backwards with "f" or "r", you always end on the last radio button, but if you navigate backwards with other commands, you land on the *first* item
21:01:10 <jugglinmike> Matt_King: I've been imagining that this could be another use case for the assertion exceptions
21:02:31 <jugglinmike> James_Scholes: I think that would be appropriate. I think that it may end up a bit confusing in some test plans. It could turn test plans into a sort of spaghetti
21:02:46 <jugglinmike> Matt_King: I was just trying to make the test plan simpler for testers
21:02:55 <jugglinmike> James_Scholes: Absolutely
21:04:00 <jugglinmike> Matt_King: howard-e I would like to get a date on how fast you could handle this issue--1039
21:04:04 <jugglinmike> https://github.com/w3c/aria-at/issues/1039
21:04:21 <jugglinmike> howard-e: I have't been able to triage this, yet, so I'm not prepared to estimate right now
21:04:48 <jugglinmike> Matt_King: I would like to see this within the next sprint at the latest
21:04:51 <jugglinmike> Matt_King: Sooner is better
21:04:54 <jugglinmike> howard-e: Of course
21:05:07 <jugglinmike> howard-e: I'll share that with Carmen and follow up in the issue
21:08:57 <jugglinmike> Zakim, end the meeting
21:08:57 <Zakim> As of this point the attendees have been Joe_Humbert, jugglinmike, IsaDC, James_Scholes, Matt_King, howard-e, Sam_Shaw, Sam_PAC, Hadi
21:08:59 <Zakim> RRSAgent, please draft minutes
21:09:00 <RRSAgent> I have made the request to generate https://www.w3.org/2024/02/22-aria-at-minutes.html Zakim
21:09:07 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
21:09:07 <Zakim> Zakim has left #aria-at
21:09:14 <jugglinmike> RRSAgent, leave
21:09:14 <RRSAgent> I see no action items