Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: Requests for changes to agenda?
Matt_King: Hearing none, we'll stick with the agenda as scheduled
Matt_King: Next meeting: Wednesday June 4
Matt_King: Next AT Driver Subgroup meeting: Monday June 9
Current status
Matt_King: We still have 15 plans in candidate review
Matt_King: Vispero has approved one more of them. I believe they have approved a total of 8 at this point
Matt_King: We have two new things coming up for draft review soon: accordion (which will come up today), and Isa is working on spin button
Isa: That should be ready by next week
Joe_Humbert: The test queue has a lot more in it than it did a couple days ago
Matt_King: Right. Some of those are temporarily on hold. Four of the items on today's agenda are related to what's in the test queue right now
App release 1.15.0
Matt_King: There was a release this week
Matt_King: Change log is here https://
Matt_King: This is a milestone release
Matt_King: The test queue has two tabs, now. One for manual testing, and one for automated reports
Matt_King: If we add a new version of the screen reader bot to the system, then when that version becomes available, we can have that bot automatically re-run all of the reports which were generated with an earlier version of that screen reader (and to regenerate the reports)
Matt_King: This is a really big deal!
Matt_King: When we get the recent bug with macOS worked out, this will be even more important
Matt_King: Props to everyone at the Bocoup team who made this happen
Isa: I used this feature to kick off some test plan runs. It's running, now
Isa: I have some feedback. There is no visible progress on what the bot is doing. There is a table with a message that says, "created 10 re-run collection jobs", but there is no progress that I can perceive with a screen reader
Isa: If I press "refresh," there is only a message at the bottom of the screen which reads, "refreshing object events" but nothing further
Matt_King: Is that expected? Is there any way to get additional information?
ChrisCuellar: I think we would need to file that as a new issue for a feature enhancement
ChrisCuellar: We can check in with Carmen and the folks who implemented this (howard-e and Stalgia) to learn the strategy behind the implementation schedule
Matt_King: Also, a thing that's important to me is that on the candidate review page, the tables are filtered, and we can set priorities with "Target dates" with screen reader implementers
Matt_King: Right now, we have the four that we want Apple to approve at the very top of the list
Matt_King: Overall, this will simplify communications
Issue 1298 - Inconsistent VO behavior
github: w3c/
ChrisCuellar: I wanted to bring this to the community group because this may be a process question
ChrisCuellar: Bocoup built a consistency report for our own use. We're kind of automating the automation to help improve it by catching bugs earlier
ChrisCuellar: Thanks to folks who have been reporting bugs alongside this effort
ChrisCuellar: I was looking into one of the bugs we reported through the consistency checker, and I found that I could manually reproduce it. With the quicknav commands on this particular test, I was replicating the inconsistency that we were observing in the automated bot runs
ChrisCuellar: Other Bocoupers were able to reproduce it manually, as well
ChrisCuellar: It looks to me that within this specific test with Safari and VoiceOver, there's a little bit of unpredictability as to what you'll get as a manual tester
ChrisCuellar: The published reports indicate that this test passes
ChrisCuellar: But when I test on macOS 15.3.2, it wasn't passing consistently
ChrisCuellar: My question for the group is: what should we do in these situations, where the bot is finding some contradictions to what manual testers have found, and that the contradiction is legitimate?
Matt_King: It sounds like, in the attempt to identify bot flakiness, we have found VoiceOver flakiness (in some cases, at least)
ChrisCuellar: That's right
Matt_King: It seems like then it becomes a little bit of a random thing what the AT Interop report says. We don't know if the report is accurate or inaccurate
Matt_King: I think that when we discover something like this, we should raise a bug with VoiceOver
ChrisCuellar: I think it's specifically a WebKit bug in this place
ChrisCuellar: We at Bocoup could be filing these bugs, but do we want confirmation from this Community Group?
ChrisCuellar: Especially when the bug may contradict the published report
Matt_King: I made notes to myself about raising two more issues related to the app. One is related to an easier way for us to edit a report. If we had that, then when we found this flakiness, then maybe we would change it from a "pass" to a "fail".
Matt_King: We also need a better way to link failures to AT bugs so that we can track them
Matt_King: So I've been thinking of ways to do that at scale
Matt_King: As we go through Candidate Review, when the AT developer is reviewing a test plan, when there's test failures, part of the Candidate Review process is about determining whether the problem is with the test plan or the AT
Matt_King: If the test plan is at fault, then we can update the test plan. But if it's with the AT, then the AT developer can address that, but we currently don't tie that work back to the report
Matt_King: I think that linkage is something we need in order to say, "yes, we have addressed everything that we think is a problem with the test plan"
ChrisCuellar: I think that will make the app more valuable to AT developers
Matt_King: The process could be that, when we discover flakiness, we raise a bug with the developer. In that bug, we describe the flakiness. We would edit the report to show a failure and link the report to the bug. It would then be clear to the person reading the report
Matt_King: I doubt that we would want to come up with a new assertion verdict of "sometimes fails"
ChrisCuellar: Probably not, no
jugglinmike: Or it could be the forthcoming "untestable" designation, because it is a sort of ambiguity
Matt_King: Yeah, that could work
Matt_King: Okay, I'm going to raise a few more issues for the app
Matt_King: After we have a plan, we'll probably want to circle back to this issue and write down what the actual process will be for addressing it
Matt_King: So we can leave the issue open for now
ChrisCuellar: There have been a couple version bumps in macOS, so we should also run the tests again to determine whether the problem is still occurring
Re-run JAWS report for color viewer slider
Matt_King: This is the first item in the test queue, the color viewer slider
Isa: No updates, yet
Isa: But we'll get there
Issue 1246 - Arrow keys in NVDA test plans
github: w3c/
Matt_King: I think the Bot is adding a bunch of things to the queue because maybe some of the results don't match...?
Isa: On my tab, I can't see any progress
Matt_King: Let's see. "Action Menu Button"... "NVDA"... The bot has 11 of 11 responses recorded
Matt_King: Wasn't it going to be the case that these would only show up in the manual test queue if the responses didn't match prior responses and there were conflicts?
ChrisCuellar: I think that was the intention. But we're seeing all of them in the queue, already, but it looks like verdicts have not yet been recorded
Isa: It appears as though you have to assign testers
Matt_King: We have a little bit of a mess, but we can work around it
Matt_King: This might be what Joe_Humbert was talking about
Matt_King: So "radio group example using active descendant" is the one that I wanted to talk about here
Matt_King: We were going to re-run this with NVDA, and we have Isa and Louis assigned.
Matt_King: But it doesn't look like you've started on this
Matt_King: The goal here was, in particular, to look at the output for "insert+up arrow" because we think the output for "insert + up arrow" should always be giving information for all the radio buttons if the defaults are truly being used
Matt_King: At least that's what James was asserting when we last discussed the issue
Matt_King: We wanted to get two people to verify whether that is the case. And then we wanted to decide whether to remove "insert+up"
Matt_King: When we looked at the reports for "Active descendant", it looked like the arrow keys were fine. But James said, "I don't think they should be passing"
Matt_King: That's why we wanted to re-run--to see if other people get the same thing as James
Matt_King: We decided to just do the "Active descendant" first
Matt_King: We have Isa and Loius assigned to this "Active descendant" one
Isa: I think we should wait for the bot to finish whatever it's doing because it's currently very confusing
Matt_King: Agreed
Isa: So we don't modify anything, yet, right?
Matt_King: Right
Matt_King: I found out that sometimes I thought I was using the defaults with NVDA, but it wasn't actually using the defaults
Matt_King: By the way, I guess we should be looking at the instructions in the test plans
Matt_King: So the next step here is for Isa and Louis.
Isa: It's confusing right now, but we can try to work around it. If not, we'll ask for help on the mailing list
Isa: It would be helpful if we could have the process documented to know what to expect
Isa: The process with the bot and the automatic reports
Matt_King: I think it's behaving unexpectedly, so we want to correct that
Matt_King: We expected it to automatically publish reports if the new version of the screen reader generate the same output as the prior version
Matt_King: If the output was different,then a human would need to step in and review
ChrisCuellar: That's my understanding, as well
ChrisCuellar: I'm currently communicating with howard-e to understand what's gone wrong
Re-run of JAWS report for Radio Group Example Using Roving tabindex
Matt_King: We were in a meeting with Vispero. On the first "radio group" (the "active descendant" one), they had made bug fixes and reached 100% pass rate
Matt_King: They expected the same for "roving tab index", but only 92% were passing instead of 100%. We looked at the failures and tried to reproduce them with the latest release of JAWS (the May release) and found that there were no longer repoducible
Louis: I can take that on
Matt_King: Radio group example using roving tab index. I just assigned myself
Isa: Anybody else? Would you like to run JAWS?
Joe_Humbert: I can do that if needed
Isa: Thank you, Joe_Humbert!
Joe_Humbert: I will assign myself
Matt_King: With JAWS and Chrome, please
Matt_King: Fabulous! Now, we can make progress on that
Matt_King: Our next meeting with Vispero is two weeks from yesterday, so it would be great if we could get this done within the next two weeks. Ideally, by Tuesday June 10
Louis: I think I can commit to that. I will let James and Isa know if anything comes up
Run of accordion test plan
Matt_King: I guess we're not ready to assign people to this because we still have some technical details to figure out
Matt_King: It's pull request number 1245
<Matt_King> accordion PR
<Matt_King> w3c/
Matt_King: We've had this strange problem come up on a few pull requests
Matt_King: There are sixty files changed. One of the problems for those of us using a screen readers. When that number grows very large, it tanks performance of the screen reader
Matt_King: You have to sit and wait for many minutes, and navigating the tree--if possible at all--takes a very long time
Matt_King: I was able to do this to see that there are files present in the "build" directory even though that directory has been explicitly excluded from the repository via the ".gitignore" file
ChrisCuellar: I remember howard-e bringing this up
Matt_King: I think this is only happening with Isa. I haven't tried to author a pull request recently. I could do an experiment to see if it happens to me
mfairchild__: Do you have the .gitignore file, Isa?
Isa: I can check. Or I can just re-clone the repository
ChrisCuellar: Do we have an issue for this?
Matt_King: I don't think so, no
Matt_King: It would be nice to get some help. For screen reader users, it's hard to know whether we should delete the file
jugglinmike: I can push a commit to Isa's pull request branch that deletes the "build" directory
Isa: I confirm that I have .gitgnore listing the build directory in it
jugglinmike: it's possible that the global git configuration could be forcing the addition of files by default. I'll look into that.
jscholes: Maybe someone else can try making a pull request to test.
jugglinmike: If you review commits, it looks like a gh action bot is adding the build files in somehow
… I'll fix this branch as it stands and file a separate issue to investigate the root cause. It's probably related to a gh actions bot.
Matt_King: Ok we'll close the meeting. Thanks everyone!