Meeting minutes
Review agenda and next meeting dates
Matt_King: Starting in January, we will have to use a different Zoom account for this meeting
Matt_King: We're still deciding which account to use, but we'll update the meeting invitation once we do
jugglinmike: I will lead a meeting of the ARIA-AT Automation on this Monday, December 18
Matt_King: Great! I can't tell you how excited I am to get this into production. It's so cool to tell the Bot to go collect responses, and then to verify that the data it reported is accurate
Matt_King: We have a meeting for this group scheduled for next Wednesday, December 20. Should we cancel? I know that PAC will be on holiday at that time.
Matt_King: Hearing no response, I will cancel.
Matt_King: This means that the next meeting of this group will be Thursday, January 11, 2024. The W3C calendar will reflect this.
Matt_King: It's possible that Vispero will approve some test plans before the end of the year. That would be extremely helpful. However, there's no hope of getting any feedback from Apple by the end of the year
Manual testing progress check-in
Matt_King: If we could get these four test plans into "candidate review" by the end of the year, that would be great: Alert, Command Button, Link, and Toggle Button
IsaDC: I'm unable to access all of the data
Matt_King: What we have right now with Toggle Button is an instance of the bug that's later in the agenda
Matt_King: But for Alert, if you access the Test Plan through the "Data Management" page, then in the "draft review" column, you'll find that the required reports are in progress
Matt_King: Clicking through to that, it appears that testing for VoiceOver is 100% complete by two testers. That's surprising
murray_moss: Oh, that was me--I just finished testing today
IsaDC: That's great!
IsaDC: I think we can mark "Alert" as final, now
Matt_King: I will do that, now...
Matt_King: It should say "Required reports complete" on the "Data Management" page, now
Joe_Humbert: It does!
Matt_King: Great, then we can advance "Alert" to the "candidate" phase
Matt_King: Okay, so, "Command Button" for JAWS has a problem. The version that's in the test queue, and the one that folks recently ran, is a deprecrated version of the plan
Matt_King: I don't know what to do about this
Matt_King: There's a bug: the deprecated version of the test plan should not show up in the test queue
Matt_King: More immediately, though, we should figure out what our options are
Matt_King: We could add the December 13th version of the plan to the test queue. Then both versions will be there, but the December 13th version will be empty
Matt_King: Nothing changed about the JAWS tests for that version
Matt_King: howard-e, is there a better way to manage this state rather than just throwing away IsaDC's work and asking her to re-do it
howard-e: Without explicit assistance from the Bocoup team, the so-called "Brute force" solution that you just described is the only option
howard-e: I think the reason why this didn't work is a recent settings change affects a top-level value in the test
howard-e: I think we may have a backup of the verison before; we may be able to roll back, but it makes me uneasy to manipulate the data in that way
Matt_King: If there's no straightfoward way to do this on the backend, then IsaDC: how long would it take to re-run the test plan?
IsaDC: The issue is not only Toggle Button. That's just the one that triggered the bug. We made changes to Link, Command, and Toggle Button
Matt_King: You could just copy your output manually. You don't have to re-run the test; all you have to do is look at your results, copy the output strings, and select "pass" for the rest
Matt_King: Because JAWS passed everything in your report and because nothing has changed
IsaDC: I'm concerned about "link" because I marked as final a deprecated version for which we had results
IsaDC: This was for NVDA
IsaDC: the changes you made to the setting are not present in the VoiceOver test plan run
Matt_King: I don't see "toggle button" with VoiceOver in the test queue right now
IsaDC: It seems like the test queue is behaving strangely today
Matt_King: If I go to "toggle", it shows the December 13th version in "Draft review". It shows in the "report status" dialog, "VoiceOver for macOS" is just empty
Matt_King: I'm going to add that to the test queue. Now, we can look at it in the test queue to see if the new instructions show up
Matt_King: It should be on the first test (for navigating to the toggle button under "J"--it should say "j command with single quick key"
IsaDC: Nope, the runner is not rendering the instructions
Matt_King: Okay, I guess we're blocked, then
<Joe_Humbert> Just FYI. I filed a new issue this morning: w3c/
<Joe_Humbert> Issue # 1024 since the chat blocked my link
Matt_King: I'm going to close issue 1019
IsaDC: Yeah, that was mine. And you created issue 1023
howard-e: Except 2023 is a change to Link, and 1021 is a change to "Command button"
IsaDC: It appears that I missed merging a pull request.
Matt_King: Ahah, that may explain the problem we've been discussing. That would be nice because it would mean there is no software bug to fix
Matt_King: There are no conflicts in "Toggle Button", so I should just merge this pull request...
Matt_King: We have to merge the "toggle button" pull request, it has to show up in the app, we have to add the newer version to the test queue, and then it needs to be run
Matt_King: For the folks who have already run the test, I recommend copying the output you've already collected and pasting it into the new version (rather than manually re-running the test)
App gbug impacting manual test running
github: w3c/
Matt_King: We have the December 6th verison of a test plan in "draft review". We have results for some of those tests. While we were running the test, we found a problem. So we made a new version, and that new version shows up as an "r&d" version
Matt_King: So we have three active versions of this test plan. We advanced the r&d version to draft review.
Matt_King: The december 13th version is now in "Draft" and the december 6th version becomes "deprecated"
Matt_King: But the december 6th version is still in the test queue and it has results
Matt_King: The December 13th version is not in the test queue at all
Matt_King: This is not the situation we want, because if people continue testing, we'll have to throw their work away
Matt_King: The bug is that whenever a test plan is in the test queue and we advance a new version of that plan to the same phase as the version that is in the test queue--well, what should happen?
Matt_King: In this bug report, I proposed what should happen
Matt_King: The first thing is that if a plan becomes deprecated, any runs that exist in the test queue for the deprecated plan should be remoeved from the test queue (not thrown away, just removed)
Matt_King: Then, since there were runs, the runs should still be there--we should add a new test run for the new version of the test plan, retaining the metadata from the prior version (e.g. AT version, browser version). Then, for any test for which the results would still be valid, copy the results into the new test plan run
Matt_King: Is this what people would expect?
IsaDC: That's what I already expected would happen, especially because we didn't change anything for VoiceOver
howard-e: I do agree with the scenario that you've outlined. That's what should have happened
howard-e: The issue is that I think there was a failed comparison. After these new changes, there's a part of that test which should be excluded during the comparison, but it isn't being excluded
Matt_King: Let's say all the tests changes and none of the data is transferrable, should we still do everything else? The deprecated version should definitely be removed from the queue, right?
Matt_King: It seems like there are two different kinds of failure here
howard-e: If every test changed, then if you transitioned a test plan version to that phase, then every tester would have zero results
Matt_King: But the deprecated version should also be replaced by the newer version, right?
howard-e: Yes, that's right
Matt_King: The new V2 format separates one version of a screen reader with another version very cleanly
Matt_King: We have an open issue related to updating the comparison algorithm...
Matt_King: There's another scenario, here. Let's say we had marked "final" the JAWS test for this plan.
Matt_King: So it's not in the test queue. But now that test plan is deprecated, and we advance a test plan from one phase to the next phase.
Matt_King: Should we automatically create a new draft report for any report that was previously marked "final" for that plan?
Matt_King: This is a harder question, especially in the world of automation
Matt_King: If it's in draft review, maybe this is an easier question to answer
IsaDC: I think the data should be copied into the new version, if there are no other changes.
Matt_King: If we had made a change to something that affected JAWS, should we not just keep it and also automatically add it to the test queue...?
howard-e: Only if it's in draft
[the group discusses further edge cases and potential remediations]
Matt_King: I'm getting a sense for what we need to do, here. I'll follow up with some more information on the bug report itself