Meeting minutes
Review agenda and next meeting dates
Matt_King: lola isn't present today, so we may need to skip her pull request for the HTML button tests
Matt_King: We'll see if she joins in the mean time
Matt_King: Requests for changes to agenda?
Matt_King: Hearing none, we'll stick with the agenda as published
Matt_King: Next CG meeting: Wednesday September 9
Current interop reporting status
Matt_King: There are no changes in the test plan phases. We're working on getting the test plans ready, so we still have 2 in draft review (one with active work ongoing, one stuck due to an application bug)
Matt_King: Lola's button test plans will probably be merged and in the system and advanced to "draft review" by the next meeting
Matt_King: I haven't made any progress in the stuff that's in "candidate review" in the last couple months with the screen reader developers.
Tri-State Checkbox Test Plan
Matt_King: Quick reminder (especially since I'm the only one who was working on the pull request): the more people who are paying attention to the details of the test plan, the better (to make sure there are no mistakes)
Matt_King: To that end, thanks to Joe_Humbert for surfacing some issues that I have since resolved
Matt_King: I've advanced the new version of the test plan to "draft review", and the system did a really good job of keeping all of the work that's been done in place
Matt_King: It looks as though in the JAWS run that Joe_Humbert completed, it only removed two verdicts, so Joe_Humbert only has to re-do those two verdicts
Matt_King: And then it's the same thing with the test plan run for NVDA
Joe_Humbert: I noticed that in the JAWS test, so I guess I glossed over it in the NVDA test
Joe_Humbert: In one of the earlier test, I believe it was "navigating backwards" with JAWS, it read out more information than there were assertions for. That was captured by the bot and that I confirmed through manual testing. The assertions were about the name and state of the checkbox, but when navigating backwards, it also gave the group label
Joe_Humbert: Is that expected or something new? (It was just for JAWS)
Matt_King: I'm taking a look at the test plan, now...
Joe_Humbert: Unfortunately, since it's just me assigned, I can't actually review the results
Matt_King: Ah, yes, "shift + x" announces the "all condiments" group
Matt_King: So that's not necessarily expected because the "navigate back from here" is outside of that group
Joe_Humbert: Right. It's just that generally, the assertions kind of match the shape of what you get because that's what you're expecting. I don't know if this has changed between versions of JAWS. I don't think it's a major issue; I just wanted to make note of it
Matt_King: Actually, the "mixed" checkbox is indeed inside the group. The label on the checkbox is "All condiments" and the label on the group is "sandwich condiments"
Matt_King: You wouldn't necessarily expect the group label because you aren't transiting the group boundary, I wouldn't mark this as excess verbosity because you are inside the group
Joe_Humbert: Agreed. I did not
Matt_King: Okay, I think this is just a case of acceptable extra information. A good call-out, though!
Matt_King: As for what to do about VoiceOver
Matt_King: Are you set up to capture the output though that VoiceOver capture program?
Joe_Humbert: I believe so. I have two different macs: one with 15 and the other with 26. Should I test with the latest version of macOS?
Matt_King: I think we should use the latest version because it will probably be difficult to find someone who is running the older version
Matt_King: macOS 15 is so old now that I would be shocked if we could find two people to run it. macOS 26 will be old soon, itself.
Matt_King: mmoss, if you have the time to invest, it would be more beneficial for us to get the macOS bot upgraded
mmoss: Sounds good!
App issue 1666: Can't advance tabs test plan
github: w3c/
Matt_King: This issue is keeping the "tabs" test plan in "draft review" even though all the testing is done. We have some problem where the system errors when I try to advance it
Matt_King: elizabeth has volunteered to take a look!
elizabeth: I sent a message to the Slack channel on Monday. I received ChrisC's message
elizabeth: I was able to run it locally, but I noticed that on my side, it is in the "draft preview" but on production, it is in "r&d"
Matt_King: "r&d" is before draft. If it's in "draft", then the next step is "candidate"
Matt_King: In the production deployment of the app, on the "data management" page and the row for "tabs with automatic activation". It's at version 25.0.8.21
Matt_King: I think the best thing to do here, without knowing much about how to do it, is to get a snapshot of the current production data
Matt_King: In order to go from R&D to draft to candidate: it can't advance to candidate until all of the results are available and the test plan run is marked "final"
Matt_King: If your version of that test plan is still in the "r&d" phase, that means that you don't have any of those test results in your local database. It would be a lot of work for you to regenerate all of those. It would be faster and easier if you were able to get a snapshot of the production data on your local machine
elizabeth: I'm not sure how to do that. I will wait for chrisc's message to see if they are able to help me with that
Matt_King: That will be the next step. I would assume there is some documentation available
jugglinmike: I can look for documentation on that procedure. It's especially important to try that because the problem is due to a subtle problem with the production database itself. Recreating that exact state manually may be difficult
PR 1400: Add HTML Button Test Plan
Matt_King: We're skipping this topic because Lola is absent
Revised how to test wiki page
Matt_King: I spent some time on a page that is linked in the agenda
Matt_King: https://
Matt_King: The title is "Running a Test Plan"
Matt_King: There is one section that have not finished writing, yet
Matt_King: I tried to start from the very beginning, assuming very little. I imagined a new tester who has just joined the group
Matt_King: I started with an overview of the testing process and tried to keep it as simple as possible. I reviewed the five steps that we have, trying to explain the basics of the language (test plan, test queue, etc), explaining the assignment of test plans to people, explaining the goals, explaining the process for correcting mistakes in the plan and differences between results, and finally getting it to the point where an admin can
finalize the plan
Matt_King: My goal for a new tester like tugce is to give an overall idea of what we're doing when we run these tests
Matt_King: The rest of the page is all about getting into the nitty-gritty of executing a test plan
Matt_King: I talk a little bit about the instructions and then go through dealing with the nuance of commands with multiple key presses, when to use the "no output" check box, the "untestable" check box
Matt_King: I haven't written about the "negative side effects" checkbox
Matt_King: And the details of recording screen reader output. I provided instructions and links
Matt_King: I'm intending for someone to walk through this and follow the instructions in a test plan to confidently execute a test plan for the first time
Matt_King: I'm looking for feedback on anything in here that isn't clear or confusing
Matt_King: In particular from Joe_Humbert or mmoss, if there's something you have encountered when you've been running test plans in the past, if there have been any problems that raise doubts
Matt_King: It occurs to me now that one thing I didn't talk about at all is the process of raising issues. I wonder if that should be included
Matt_King: I could provide some guidance about when and how to raise an issue...
elizabeth: I think so. I think it could also be helpful to discuss the bots
Matt_King: Ah, do you mean more explicit instructions about verifying the bot output?
elizabeth: Yes
Matt_King: I wrote a little about that [reads the current text]. In the second paragraph, under "recording screen reader output", I think I talk about that [reads the current text]
elizabeth: Yes, that's what I was talking about
Matt_King: I think, if someone discovers errors or omissions here, you could e-mail me directly or raise an issue in the repository. Either way is fine
Joe_Humbert: The vast majority looks good to me so far, but I haven't reviewed all of it, yet
Joe_Humbert: Somewhat related: do we know if the VoiceOver recorder application will work with macOS 26?
mmoss: It does work on macOS 26
Joe_Humbert: Good to know! The GitHub repository hasn't been updated in a very long time
mmoss: There is a bug where it will occasionally forcibly relaunch VoiceOver
Joe_Humbert: That's a bug in macOS 15, as well
Joe_Humbert: The biggest thing for novice users of that tool is in getting Homebrew set up.
Joe_Humbert: I've experienced strange installation errors with Homebrew, and I think those could trip up contributors without systems knowledge
Joe_Humbert: If it were instead packaged in a standard macOS installation procedure, I think that would be more robust
Joe_Humbert: I assume that is related to how strictly Apple polices third-party software installation--that it requires signing certificates for security purposes
mmoss: We ought to be able to generate a pkg file regardless of whether we sign it; it will just warn the end-user when they go to install it
Matt_King: What are the hurdles to securing a signing certificate?
Joe_Humbert: You have to join Apple's developer program, pay a yearly fee, and submit the application for review from Apple
mmoss: That's only needed if you don't want the user to have to go through any hoops to decrease security
Matt_King: Maybe the most practical solution is through an organization who has people who are already Apple developers. If they were to join ARIA-AT, they could help here. It could still be an open-source application, right? I wonder...
mmoss: You just need to be a member of the Apple Developer program. Technically, I am a member
mmoss: It does cost money, though, and there is identity verification
Matt_King: I wonder if Dequeue has built any software for macOS
Joe_Humbert: It would be lovely if we could find someone at Apple to help get us internal support
Matt_King: I can raise that with James Craig. Why not have a participant from Apple in ARIA-AT?
Matt_King: Meta had tons of open-source software (like React). I haven't noticed whether Apple does any open-source software work
mmoss: They do it. It's listed on https://
Matt_King: Oh, thank you! I'll research that