17:03:31 <RRSAgent> RRSAgent has joined #aria-at
17:03:35 <RRSAgent> logging to https://www.w3.org/2025/04/09-aria-at-irc
17:03:35 <Zakim> RRSAgent, make logs Public
17:03:36 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
17:03:44 <ChrisCuellar> present+
17:03:44 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
17:03:48 <jugglinmike> present+ jugglinmike
17:03:50 <jugglinmike> scribe+ jugglinmike
17:03:52 <Joe_Humbert> present+
17:04:10 <jugglinmike> present+ mmoss
17:04:17 <jugglinmike> present+ Matt_King
17:04:30 <jugglinmike> Topic: Review agenda and next meeting dates
17:04:38 <jugglinmike> Matt_King: Next AT Driver Subgroup meeting: Monday April 14
17:04:42 <jugglinmike> Matt_King: Next Community Group Meeting: Thursday April 17
17:04:50 <jugglinmike> https://github.com/w3c/aria-at/wiki/April-9%2C-2025-Agenda
17:05:13 <jugglinmike> Matt_King: Requests for changes to agenda?
17:07:02 <jugglinmike> Topic: Current status
17:07:08 <jugglinmike> Matt_King: We have 14 plans in candidate review
17:07:21 <Carmen> Carmen has joined #aria-at
17:07:22 <jugglinmike> Matt_King: We have two in the test queue, and then the "disclosure" plan is on hold while we wait for the JAWS update
17:07:40 <jugglinmike> Matt_King: We'll talk about the other two plans which are currently in the queue
17:08:08 <jugglinmike> Matt_King: The agenda has a link to a spreadsheet we are building to make a schedule for the upcoming work https://docs.google.com/spreadsheets/d/14QIhQB9ufdUdZNhb3TNZzoZkGBq0oHawft-S5pm0ioY/edit?gid=0#gid=0
17:08:27 <jugglinmike> Topic: Testing Radio group with roving tab index
17:08:40 <jugglinmike> Matt_King: In this test plan, we only have one conflict with JAWS. Everything else is done
17:09:03 <jugglinmike> Matt_King: I think we may have narrowed this down to Windows 10 and Windows 11 giving different results
17:09:03 <jugglinmike> present+ James
17:09:38 <jugglinmike> James: I think we have two testers on Windows 10 who have produced on set of results consistently, and two testers on Windows 11 who have produced a different set of results consistently
17:09:42 <jugglinmike> present+ IsaDC
17:09:52 <jugglinmike> IsaDC: Are you sure we have two testers on Windows 11?
17:10:02 <jugglinmike> Matt_King: Well, we know IsaDC is on Windows 11
17:10:13 <jugglinmike> Matt_King: And we know James and Joe_Humbert have Windows 10
17:10:37 <jugglinmike> present+ Hadi
17:11:41 <jugglinmike> Matt_King: I am on Windows 11. It seems like I should check this one out
17:12:00 <jugglinmike> Matt_King: If I get results like James and Joe_Humbert, then IsaDC's machine would be the outlier, and we'd be wondering what's going on
17:12:10 <jugglinmike> Joe_Humbert: Is this something we should ask Vispero about?
17:12:28 <jugglinmike> Matt_King: Absolutely! If I observe what IsaDC has reported, then I will reach out to Brett at Vispero
17:12:57 <jugglinmike> Matt_King: In that case, we might have to collaborate with Vispero about what JAWS ought to say. That would be a new circumstance for us
17:13:06 <jugglinmike> Matt_King: Anyway, I'll take this as an action item for the next meeting
17:13:17 <jugglinmike> Topic: Testing vertical temperature slider
17:13:25 <jugglinmike> Matt_King: We have multiple situations with this test plan
17:13:38 <jugglinmike> Matt_King: Let's start with the issue James raised yesterday. It's an APG issue
17:13:57 <jugglinmike> Matt_King: The way the test plan is currently written, it's not possible for people to get the same output for the down arrow for navigating to the slider with NVDA
17:14:43 <jugglinmike> Joe_Humbert: I noticed that with JAWS, you get to the slider because you down arrow three times. With NVDA, you don't get to the slider because you only down arrow twice
17:14:48 <jugglinmike> Matt_King: Yes
17:14:56 <jugglinmike> Matt_King: I think we should update this test plan
17:15:02 <jugglinmike> Matt_King: James raised an issue against APG
17:15:26 <jugglinmike> Matt_King: I think the arrow is due to a bug in the test case. The "25 degrees C" that appears should not be there because it is duplicative
17:15:39 <jugglinmike> Matt_King: I don't know why it's there; we'll discuss that in the next APG Task Force meeting
17:15:55 <jugglinmike> Matt_King: To work around the bug, I think we should just update the test to change the command with another arrow key press
17:16:08 <jugglinmike> IsaDC: I have the pull request for that ready to go. I wanted to discuss it here, first
17:16:41 <jugglinmike> Matt_King: I'm suggesting that the APG remove the text label, but that will come later.
17:17:02 <jugglinmike> Joe_Humbert: They would have to remove it completely because it's text. You don't want to hide text from a screen reader user.
17:17:25 <jugglinmike> James: It would be acceptable to hide the text because it is duplicated in the aria-value text.
17:17:34 <jugglinmike> Joe_Humbert: I still think it's generally a bad practice
17:17:45 <jugglinmike> Matt_King: Yeah, the APG might come back and say that it's fine the way it is
17:18:32 <jugglinmike> Joe_Humbert: I can see the text rendered on the screen twice. One is above, and one is next to the slider "thumb"
17:18:39 <jugglinmike> James: That seems visually duplicative
17:18:46 <jugglinmike> ChrisCuellar: I agree
17:19:14 <jugglinmike> James: Hiding text can indeed set a dangerous precedent, but so does enunciating text twice. I think it should just be in once place for everyone
17:19:31 <jugglinmike> Matt_King: For context, we've had many requests to add "aria-hidden" to the text "temperature"
17:19:39 <jugglinmike> Matt_King: The Task Force has pushed back on that
17:19:59 <jugglinmike> Joe_Humbert: I think the number on the top is better because it is larger and easier to read and because it doesn't move with the "thumb"
17:20:12 <jugglinmike> Matt_King: Maybe you don't need the one on the rail
17:20:54 <jugglinmike> Matt_King: Well, we'll see what the Task Force says. For now, I'm glad IsaDC has a patch ready
17:21:05 <jugglinmike> Matt_King: Applying that will mean running that one test again
17:21:17 <jugglinmike> James: Do we need to alter the "navigate backwards" tests?
17:21:23 <jugglinmike> Matt_King: Nope
17:21:58 <jugglinmike> Matt_King: Were there any conflicts which were not related to that "moving forward" test?
17:22:17 <jugglinmike> IsaDC: The negative assertions, what should we do, for now, when we find negative assertions that would technically pass?
17:22:45 <jugglinmike> Matt_King: This is an issue which appears later on the agenda
17:22:56 <jugglinmike> Matt_King: James raised an issue for this, and then I have worked on a solution.
17:23:30 <jugglinmike> Matt_King: The testing for this--and I think even the prior slider stuff that we've done for NVDA--I think we'll need to revisit. I think this is blocked until we can generate an accurate report. What I mean by that will become more clear in a minute
17:23:56 <jugglinmike> Topic: Re-run of JAWS for color viewer slider
17:24:10 <jugglinmike> Matt_King: JAWS has fixed some bugs, so it would be advantageous if we could re-run these tests
17:25:02 <jugglinmike> present+ Hadi
17:25:06 <jugglinmike> Hadi: I can help
17:25:10 <jugglinmike> Joe_Humbert: I can, as well
17:25:40 <jugglinmike> Matt_King: Great! We want to use the latest JAWS. The version which was released in March of this year
17:25:50 <jugglinmike> IsaDC: I will assign you both
17:26:01 <jugglinmike> Matt_King: Great!
17:26:04 <jugglinmike> Topic: App issue 1352 - Changes for negative side effects and untestable assertions
17:26:14 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1352
17:26:28 <jugglinmike> Matt_King: I put together a detailed plan that talks about the test runner and the reports
17:27:11 <jugglinmike> Matt_King: When you encounter a test like this (where it technically passed an assertion but for an inappropriate reason)
17:27:31 <jugglinmike> Matt_King: ...there would be a checkbox which indicates whether the assertions were testable
17:28:13 <jugglinmike> Joe_Humbert: So if it doesn't work as we think it should (it skips over or it says nothing at all, for example), there will be a test box for us to use which says, "we can't apply these assertions"
17:29:05 <jugglinmike> Matt_King: [summarizes the full UI and workflow as proposed in the issue]
17:30:11 <jugglinmike> Matt_King: I would like this to be expedited as quickly as possible so that we can get accurate reports on all of the sliders. I think we may even need to re-run a few VoiceOver tests because we encountered this problem and the way we reported them to Apple was confusing
17:30:17 <jugglinmike> present+ Carmen
17:30:31 <jugglinmike> Carmen: Sounds good. We have a planning meeting tomorrow, so we can prioritize this work accordingly
17:30:42 <jugglinmike> Hadi: How often is this condition?
17:31:07 <jugglinmike> Matt_King: It occurred on any test plan with Shift+J for one screen reader. We also just found it in a class of tests for NVDA
17:31:33 <jugglinmike> Matt_King: So far, it's happened in probably seven or eight test plans and with two screen readers
17:32:16 <jugglinmike> Matt_King: We discussed about what to do without this new feature. We could just mark assertions as failing, but that gives a misleading picture of what is wrong. It produces confusing data, and I don't think we want that
17:33:16 <jugglinmike> Topic: App issue 1162 - Automated updating of reports for new AT releases
17:33:22 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1162
17:33:36 <jugglinmike> Matt_King: This feature affects everyone, but it's really only used by admins
17:33:52 <jugglinmike> Matt_King: When a new version of JAWS comes out (one is due in May)--or NVDA for that matter
17:34:13 <jugglinmike> Matt_King: We would like to be able to re-run all the test plans in "candidate review" using the bots
17:34:57 <jugglinmike> Matt_King: It won't automatically add the bot to the system and start running the tests. It will require some administrative input. This interface is for performing that task. It is deployed to the "sandbox" server right now, and it is ready for testing
17:35:17 <jugglinmike> Matt_King: In the issue I linked to in Carmen's latest comment, there are instructions for testing
17:35:47 <jugglinmike> Matt_King: As soon as we have approval from IsaDC and me, we'll have this pushed out
17:36:05 <jugglinmike> Matt_King: This will be a big deal for us, especially when JAWS releases the next version
17:36:50 <jugglinmike> Joe_Humbert: With this kind of automation, will it be possible to get results for previous versions, or will that require manual testing?
17:37:02 <jugglinmike> Matt_King: I think we could add older versions of screen readers to the bots and use those
17:38:44 <jugglinmike> jugglinmike: For NVDA and VoiceOver, yes. For JAWS, we may need to do some extra work (depending on whether Vispero hosts previous releases)
17:38:48 <ChrisCuellar> ChrisCuellar has joined #aria-at
17:39:14 <jugglinmike> jugglinmike: So to support older versions of JAWS, we may need to keep those versions on hand ourselves
17:39:23 <jugglinmike> Topic: Issue 1213 - Minimum version requirements
17:39:31 <jugglinmike> github: https://github.com/w3c/aria-at/issues/1213
17:39:50 <jugglinmike> Matt_King: I don't know if we want a policy on this issue or if we just want it to be an admin decision every time you add something to the test queue
17:40:35 <jugglinmike> Matt_King: Depending on how the run is added, the app may or may not present a user interface for setting the version
17:40:53 <jugglinmike> Matt_King: I think that may be an omission in the design. I think addressing that omission may resolve the issue
17:40:59 <jugglinmike> IsaDC: That would definitely resolve it
17:41:19 <jugglinmike> Matt_King: We want to control the minimum version of the AT when adding a test plan to the test queue to the report status dialog
17:41:51 <jugglinmike> Matt_King: Carmen, can you create an issue for that in the aria-at-app repository?
17:41:53 <jugglinmike> Carmen: Sure
17:42:04 <jugglinmike> Topic: Issue 1211 - Reliability of app support for advancing new test plan versions
17:42:13 <jugglinmike> github: https://github.com/w3c/aria-at/issues/1211
17:42:36 <jugglinmike> Matt_King: we expect results to be copied into the draft for the report of the new test run. We've seen some inconsistent behavior on this, though
17:43:03 <jugglinmike> Matt_King: James filed an issue, and howard-e shared a very detailed response. Have you had a chance to review howard-e's response, James?
17:43:31 <jugglinmike> James: I did read this when it was first posted; I will have to refresh my memory
17:43:59 <jugglinmike> Matt_King: I don't think that we have a current behavior failure
17:44:35 <jugglinmike> Matt_King: We did have an example, but we destroyed it when we fixed the problem
17:45:12 <jugglinmike> Matt_King: We're going to have an opportunity coming up. IsaDC is working on a change to the slider. We'll see if that one works correctly. It might have something to do with which specific things get changed in the test plan. We can just leave this issue open until we see a problematic behavior again
17:45:27 <jugglinmike> James: We're missing the ability to update the reference without changing anything in the test plan itself.
17:45:58 <jugglinmike> James: Some change would warrant changing the reference date. But sometimes we have to make a small change to make settings work. What we don't have in the app is to essentially take notice of that
17:46:27 <jugglinmike> James: from howard-e's response, it seems as though the app is only aware of a command being change, an assertion being changed, or a change to an assertion ID
17:46:45 <jugglinmike> James: ...but we also want the app to take notice if we change the reference or the setup script
17:46:56 <jugglinmike> James: So right now, we've pushed a new test plan, and it doesn't get re-imported
17:47:05 <jugglinmike> Matt_King: That's a different problem, them. This is about copying results
17:47:24 <jugglinmike> Matt_King: If, for example, the assertions change, then you don't copy the results from the prior into the new
17:47:41 <jugglinmike> Matt_King: If the setup script changed, is that another one that should void prior results? What about the reference?
17:47:50 <jugglinmike> James: It's tricky to say because that's on a "test" level
17:47:52 <jugglinmike> Matt_King: Right
17:48:15 <jugglinmike> Matt_King: One of the side-effects of maintaining who the tester is, is that we currently don't have a function for the tester to be changed from one person to another
17:48:58 <jugglinmike> Matt_King: It would be really nice if, when something was assigned to me and I did half the work, if I could re-assign it to Joe_Humbert. Then Joe_Humbert would assume responsibility for everything I've done, and he could finish the rest of the work
17:49:43 <jugglinmike> IsaDC: With the bot, it would be really useful to have that because sometimes we have the bot collect responses, then we assign to a tester, and then that tester can't help, but we aren't able to re-assign the run to another tester
17:49:54 <jugglinmike> Matt_King: That sounds like another feature request
17:50:05 <jugglinmike> Matt_King: A button for "change assignee"
17:50:23 <jugglinmike> Matt_King: We could even make the person's name into that button. Right now, it's a link to their GitHub profile
17:50:30 <jugglinmike> Matt_King: You can propose something
17:50:39 <jugglinmike> Matt_King: Right now, I would prioritize this as "P2"
17:50:42 <jugglinmike> Carmen: Got it!
17:51:15 <jugglinmike> Matt_King: If a copy is in prior results that aren't value, it's up to someone to re-run those results or make sure the previously-copied results are valid
17:51:37 <jugglinmike> Matt_King: Do we want to err on the side of over-copying (copying things that may have been voided), or under-copying?
17:51:51 <jugglinmike> James: I would like to test things like these before they go into the main app
17:52:33 <jugglinmike> James: I think that, regardless of the route we take, it needs to be possible for us to--when we make a change to the test plan, run it through a separate environment which is a copy of production, in order to review the actual change
17:52:49 <jugglinmike> James: Then we can immediately halt and not deploy to production because something unexpected happened
17:52:57 <jugglinmike> Matt_King: Essentially testing the test plan itself
17:53:00 <jugglinmike> IsaDC: Yes!
17:53:13 <jugglinmike> Matt_King: Okay, that is a separate issue. It's on the agenda, though we won't get there today
17:53:33 <jugglinmike> Matt_King: I think it might not be a massive piece of work to make it happen. We'll save the discussion for when we get to that issue
17:53:46 <jugglinmike> Matt_King: But in the mean time, if you can reflect on how safe we want to play it, I think that would be helpful
17:54:14 <jugglinmike> James: I would also love the ability to "roll back" anything that happened. Whether due to a bug or an expected-but-hard-to-predict behavior, I would love to be able to revert
17:54:46 <jugglinmike> IsaDC: I'm pushing some changes, and I would like to know if the results we have now--will we have a way to get them back?
17:55:16 <jugglinmike> James: We're making a change to a test plan, and it's possible that the same issue in the app will occur. Do we have a strategy to address it if we lose the results?
17:55:25 <jugglinmike> Carmen: I can ask howard-e tomorrow
17:55:49 <jugglinmike> Matt_King: Let's do it today and pay attention to what happened. If something goes wrong, we can send howard-e a detailed e-mail with what happened
17:56:43 <jugglinmike> Carmen: directly after this call, I will see if we can do a database dump. I'll reach out to you soon, IsaDC
17:56:50 <jugglinmike> Topic: App issue 1365 - Bot reliability reporting
17:57:00 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1365
17:57:17 <jugglinmike> Matt_King: Bocoup has come up with a testing methodology to test the reliability of the bots
17:57:34 <jugglinmike> Matt_King: I included a link to the report in the agenda
17:57:52 <jugglinmike> s/the report/a recent report/
17:58:14 <jugglinmike> Carmen: We are testing consistency by running each test five times and determining whether there were different responses in each "Trial"
17:58:53 <jugglinmike> Carmen: You will see that NVDA is over 99% consistent. Our focus this year is on VoiceOver--it is currently at 91%, and we would like to raise it to at least 95%
17:59:05 <jugglinmike> ChrisCuellar: Right now, it is triggered completely manually
17:59:17 <jugglinmike> ChrisCuellar: We would like to run it as a service on a regular schedule
17:59:31 <jugglinmike> Matt_King: I wonder if this will change with different versions of screen readers and bots
18:00:41 <jugglinmike> ChrisCuellar: Now that we're letting the CG know about the reports, they are almost like a feature that we can iterate on and that we can improve in response to external input
18:00:55 <jugglinmike> Matt_King: Okay, this is great. I'm really happy to have these metrics!
18:01:17 <jugglinmike> Zakim, end the meeting
18:01:17 <Zakim> As of this point the attendees have been ChrisCuellar, jugglinmike, Joe_Humbert, mmoss, Matt_King, James, IsaDC, Hadi, Carmen
18:01:20 <Zakim> RRSAgent, please draft minutes
18:01:21 <RRSAgent> I have made the request to generate https://www.w3.org/2025/04/09-aria-at-minutes.html Zakim
18:01:28 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
18:01:28 <Zakim> Zakim has left #aria-at
18:01:34 <jugglinmike> RRSAgent, leave
18:01:34 <RRSAgent> I see no action items