19:03:31 RRSAgent has joined #aria-at 19:03:36 logging to https://www.w3.org/2025/05/29-aria-at-irc 19:03:37 RRSAgent, make logs Public 19:03:38 please title this meeting ("meeting: ..."), jugglinmike 19:03:41 present+ 19:04:01 meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference 19:04:36 present+ jugglinmike 19:04:38 scribe+ jugglinmike 19:04:59 present+ 19:05:20 Isa has joined #aria-at 19:05:31 present+ 19:07:03 Topic: Review agenda and next meeting dates 19:08:30 https://github.com/w3c/aria-at/wiki/May-29%2C-2025-Agenda 19:08:36 Matt_King: Requests for changes to agenda? 19:08:45 Matt_King: Hearing none, we'll stick with the agenda as scheduled 19:08:58 Matt_King: Next meeting: Wednesday June 4 19:09:06 Matt_King: Next AT Driver Subgroup meeting: Monday June 9 19:09:15 Topic: Current status 19:09:26 Matt_King: We still have 15 plans in candidate review 19:09:42 Matt_King: Vispero has approved one more of them. I believe they have approved a total of 8 at this point 19:10:07 Matt_King: We have two new things coming up for draft review soon: accordion (which will come up today), and Isa is working on spin button 19:10:21 Isa: That should be ready by next week 19:10:47 present+ Joe_Humbert 19:10:57 Joe_Humbert: The test queue has a lot more in it than it did a couple days ago 19:11:21 Matt_King: Right. Some of those are temporarily on hold. Four of the items on today's agenda are related to what's in the test queue right now 19:11:29 Topic: App release 1.15.0 19:11:35 Matt_King: There was a release this week 19:11:47 Matt_King: Change log is here https://github.com/w3c/aria-at-app/blob/development/CHANGELOG.md 19:11:52 Matt_King: This is a milestone release 19:12:06 Matt_King: The test queue has two tabs, now. One for manual testing, and one for automated reports 19:12:41 Matt_King: If we add a new version of the screen reader bot to the system, then when that version becomes available, we can have that bot automatically re-run all of the reports which were generated with an earlier version of that screen reader (and to regenerate the reports) 19:12:48 Matt_King: This is a really big deal! 19:13:14 Matt_King: When we get the recent bug with macOS worked out, this will be even more important 19:13:23 Matt_King: Props to everyone at the Bocoup team who made this happen 19:13:37 Isa: I used this feature to kick off some test plan runs. It's running, now 19:14:21 Isa: I have some feedback. There is no visible progress on what the bot is doing. There is a table with a message that says, "created 10 re-run collection jobs", but there is no progress that I can perceive with a screen reader 19:14:47 Isa: If I press "refresh," there is only a message at the bottom of the screen which reads, "refreshing object events" but nothing further 19:14:59 Matt_King: Is that expected? Is there any way to get additional information? 19:15:27 ChrisCuellar: I think we would need to file that as a new issue for a feature enhancement 19:16:08 ChrisCuellar: We can check in with Carmen and the folks who implemented this (howard-e and Stalgia) to learn the strategy behind the implementation schedule 19:16:51 Matt_King: Also, a thing that's important to me is that on the candidate review page, the tables are filtered, and we can set priorities with "Target dates" with screen reader implementers 19:17:07 Matt_King: Right now, we have the four that we want Apple to approve at the very top of the list 19:17:16 Matt_King: Overall, this will simplify communications 19:17:25 Topic: Issue 1298 - Inconsistent VO behavior 19:17:31 github: https://github.com/w3c/aria-at-app/issues/1298 19:17:44 ChrisCuellar: I wanted to bring this to the community group because this may be a process question 19:18:14 ChrisCuellar: Bocoup built a consistency report for our own use. We're kind of automating the automation to help improve it by catching bugs earlier 19:18:28 ChrisCuellar: Thanks to folks who have been reporting bugs alongside this effort 19:19:30 ChrisCuellar: I was looking into one of the bugs we reported through the consistency checker, and I found that I could manually reproduce it. With the quicknav commands on this particular test, I was replicating the inconsistency that we were observing in the automated bot runs 19:19:42 ChrisCuellar: Other Bocoupers were able to reproduce it manually, as well 19:20:13 ChrisCuellar: It looks to me that within this specific test with Safari and VoiceOver, there's a little bit of unpredictability as to what you'll get as a manual tester 19:20:30 ChrisCuellar: The published reports indicate that this test passes 19:20:44 ChrisCuellar: But when I test on macOS 15.3.2, it wasn't passing consistently 19:21:22 ChrisCuellar: My question for the group is: what should we do in these situations, where the bot is finding some contradictions to what manual testers have found, and that the contradiction is legitimate? 19:21:49 Matt_King: It sounds like, in the attempt to identify bot flakiness, we have found VoiceOver flakiness (in some cases, at least) 19:22:01 ChrisCuellar: That's right 19:22:32 Matt_King: It seems like then it becomes a little bit of a random thing what the AT Interop report says. We don't know if the report is accurate or inaccurate 19:22:45 Matt_King: I think that when we discover something like this, we should raise a bug with VoiceOver 19:23:13 ChrisCuellar: I think it's specifically a WebKit bug in this place 19:23:38 ChrisCuellar: We at Bocoup could be filing these bugs, but do we want confirmation from this Community Group? 19:23:52 ChrisCuellar: Especially when the bug may contradict the published report 19:24:28 Matt_King: I made notes to myself about raising two more issues related to the app. One is related to an easier way for us to edit a report. If we had that, then when we found this flakiness, then maybe we would change it from a "fail" to a "pass". 19:24:40 s/"fail" to a "pass"/"pass" to a "fail"/ 19:24:51 Matt_King: We also need a better way to link failures to AT bugs so that we can track them 19:25:01 Matt_King: So I've been thinking of ways to do that at scale 19:25:33 Matt_King: As we go through Candidate Review, when the AT developer is reviewing a test plan, when there's test failures, part of the Candidate Review process is about determining whether the problem is with the test plan or the AT 19:26:00 Matt_King: If the test plan is at fault, then we can update the test plan. But if it's with the AT, then the AT developer can address that, but we currently don't tie that work back to the report 19:26:23 Matt_King: I think that linkage is something we need in order to say, "yes, we have addressed everything that we think is a problem with the test plan" 19:26:37 ChrisCuellar: I think that will make the app more valuable to AT developers 19:27:32 Matt_King: The process could be that, when we discover flakiness, we raise a bug with the developer. In that bug, we describe the flakiness. We would edit the report to show a failure and link the report to the bug. It would then be clear to the person reading the report 19:27:47 Matt_King: I doubt that we would want to come up with a new assertion verdict of "sometimes fails" 19:27:55 ChrisCuellar: Probably not, no 19:28:39 jugglinmike: Or it could be the forthcoming "untestable" designation, because it is a sort of ambiguity 19:28:46 Matt_King: Yeah, that could work 19:29:09 Matt_King: Okay, I'm going to raise a few more issues for the app 19:29:33 Matt_King: After we have a plan, we'll probably want to circle back to this issue and write down what the actual process will be for addressing it 19:29:40 Matt_King: So we can leave the issue open for now 19:30:10 ChrisCuellar: There have been a couple version bumps in macOS, so we should also run the tests again to determine whether the problem is still occurring 19:30:20 Topic: Re-run JAWS report for color viewer slider 19:30:31 Matt_King: This is the first item in the test queue, the color viewer slider 19:30:37 Isa: No updates, yet 19:30:56 Isa: But we'll get there 19:31:03 Topic: Issue 1246 - Arrow keys in NVDA test plans 19:31:13 github: https://github.com/w3c/aria-at/issues/1246 19:32:11 Matt_King: I think the Bot is adding a bunch of things to the queue because maybe some of the results don't match...? 19:32:22 Isa: On my tab, I can't see any progress 19:33:07 Matt_King: Let's see. "Action Menu Button"... "NVDA"... The bot has 11 of 11 responses recorded 19:33:29 Matt_King: Wasn't it going to be the case that these would only show up in the manual test queue if the responses didn't match prior responses and there were conflicts? 19:33:53 ChrisCuellar: I think that was the intention. But we're seeing all of them in the queue, already, but it looks like verdicts have not yet been recorded 19:34:01 Isa: It appears as though you have to assign testers 19:34:08 Matt_King: We have a little bit of a mess, but we can work around it 19:34:17 Matt_King: This might be what Joe_Humbert was talking about 19:34:39 Matt_King: So "radio group example using active descendant" is the one that I wanted to talk about here 19:35:02 Matt_King: We were going to re-run this with NVDA, and we have Isa and Louis assigned. 19:35:15 Matt_King: But it doesn't look like you've started on this 19:36:18 Matt_King: The goal here was, in particular, to look at the output for "insert+up arrow" because we think the output for "insert + up arrow" should always be giving information for all the radio buttons if the defaults are truly being used 19:36:30 Matt_King: At least that's what James was asserting when we last discussed the issue 19:36:52 Matt_King: We wanted to get two people to verify whether that is the case. And then we wanted to decide whether to remove "insert+up" 19:37:28 Matt_King: When we looked at the reports for "Active descendant", it looked like the arrow keys were fine. But James said, "I don't think they should be passing" 19:37:43 Matt_King: That's why we wanted to re-run--to see if other people get the same thing as James 19:37:52 Matt_King: We decided to just do the "Active descendant" first 19:38:14 Matt_King: We have Isa and Loius assigned to this "Active descendant" one 19:38:33 Isa: I think we should wait for the bot to finish whatever it's doing because it's currently very confusing 19:38:37 Matt_King: Agreed 19:38:59 Isa: So we don't modify anything, yet, right? 19:39:02 Matt_King: Right 19:39:20 Matt_King: I found out that sometimes I thought I was using the defaults with NVDA, but it wasn't actually using the defaults 19:39:37 Matt_King: By the way, I guess we should be looking at the instructions in the test plans 19:39:58 Matt_King: So the next step here is for Isa and Louis. 19:40:20 Isa: It's confusing right now, but we can try to work around it. If not, we'll ask for help on the mailing list 19:40:32 Isa: It would be helpful if we could have the process documented to know what to expect 19:40:45 Isa: The process with the bot and the automatic reports 19:40:55 Matt_King: I think it's behaving unexpectedly, so we want to correct that 19:41:16 Matt_King: We expected it to automatically publish reports if the new version of the screen reader generate the same output as the prior version 19:41:39 Matt_King: If the output was different,then a human would need to step in and review 19:41:46 ChrisCuellar: That's my understanding, as well 19:41:59 ChrisCuellar: I'm currently communicating with howard-e to understand what's gone wrong 19:42:09 Topic: Re-run of JAWS report for Radio Group Example Using Roving tabindex 19:42:38 Matt_King: We were in a meeting with Vispero. On the first "radio group" (the "active descendant" one), they had made bug fixes and reached 100% pass rate 19:43:19 Matt_King: They expected the same for "roving tab index", but only 92% were passing instead of 100%. We looked at the failures and tried to reproduce them with the latest release of JAWS (the May release) and found that there were no longer repoducible 19:43:37 present+ Louis 19:43:41 Louis: I can take that on 19:44:04 Matt_King: Radio group example using roving tab index. I just assigned myself 19:44:17 Isa: Anybody else? Would you like to run JAWS? 19:44:24 Joe_Humbert: I can do that if needed 19:44:31 Isa: Thank you, Joe_Humbert! 19:44:36 Joe_Humbert: I will assign myself 19:45:00 Matt_King: With JAWS and Chrome, please 19:45:12 Matt_King: Fabulous! Now, we can make progress on that 19:45:36 Matt_King: Our next meeting with Vispero is two weeks from yesterday, so it would be great if we could get this done within the next two weeks. Ideally, by Tuesday June 10 19:45:55 Louis: I think I can commit to that. I will let James and Isa know if anything comes up 19:46:04 Topic: Run of accordion test plan 19:46:18 Matt_King: I guess we're not ready to assign people to this because we still have some technical details to figure out 19:46:59 Matt_King: It's pull request number 1245 19:47:11 accordion PR 19:47:13 https://github.com/w3c/aria-at/pull/1245 19:47:29 Matt_King: We've had this strange problem come up on a few pull requests 19:47:58 Matt_King: There are sixty files changed. One of the problems for those of us using a screen readers. When that number grows very large, it tanks performance of the screen reader 19:48:19 Matt_King: You have to sit and wait for many minutes, and navigating the tree--if possible at all--takes a very long time 19:48:54 Matt_King: I was able to do this to see that there are files present in the "build" directory even though that directory has been explicitly excluded from the repository via the ".gitignore" file 19:49:09 ChrisCuellar: I remember howard-e bringing this up 19:50:38 Matt_King: I think this is only happening with Isa. I haven't tried to author a pull request recently. I could do an experiment to see if it happens to me 19:51:33 mfairchild__: Do you have the .gitignore file, Isa? 19:51:41 Isa: I can check. Or I can just re-clone the repository 19:51:51 ChrisCuellar: Do we have an issue for this? 19:51:55 Matt_King: I don't think so, no 19:52:19 Matt_King: It would be nice to get some help. For screen reader users, it's hard to know whether we should delete the file 19:53:33 jugglinmike: I can push a commit to Isa's pull request branch that deletes the "build" directory 19:55:16 jugglinmike1 has joined #aria-at 19:55:20 scribe+ 19:56:09 Isa: I confirm that I have .gitgnore listing the build directory in it 19:57:39 jugglinmike: it's possible that the global git configuration could be forcing the addition of files by default. I'll look into that. 19:58:25 jscholes: Maybe someone else can try making a pull request to test. 19:59:12 jugglinmike: If you review commits, it looks like a gh action bot is adding the build files in somehow 20:00:10 ... I'll fix this branch as it stands and file a separate issue to investigate the root cause. It's probably related to a gh actions bot. 20:00:57 Matt_King: Ok we'll close the meeting. Thanks everyone! 20:01:31 Zakim, end the meeting 20:01:31 As of this point the attendees have been ChrisCuellar, jugglinmike, Joe_Humbert, mfairchild__, Louis 20:01:33 RRSAgent, please draft minutes 20:01:34 I have made the request to generate https://www.w3.org/2025/05/29-aria-at-minutes.html Zakim 20:01:41 I am happy to have been of service, ChrisCuellar; please remember to excuse RRSAgent. Goodbye 20:01:41 Zakim has left #aria-at 20:02:31 RRSagent, leave 20:02:31 I see no action items