19:04:28 <RRSAgent> RRSAgent has joined #aria-at
19:04:32 <RRSAgent> logging to https://www.w3.org/2025/04/17-aria-at-irc
19:04:32 <Zakim> RRSAgent, make logs Public
19:04:33 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
19:04:55 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
19:05:01 <jugglinmike> present+ jugglinmike
19:05:04 <jugglinmike> scribe+ jugglinmike
19:05:15 <jugglinmike> present+ Joe_Humbert
19:05:22 <Carmen> Carmen has joined #aria-at
19:05:28 <jugglinmike> present+ mmoss
19:05:31 <jugglinmike> present+ Carmen
19:05:32 <howard-e> howard-e has joined #aria-at
19:05:32 <ChrisCuellar> present+
19:05:37 <jugglinmike> present+ howard-e
19:05:41 <jugglinmike> present+ dean
19:05:45 <jugglinmike> present+ james
19:05:54 <jugglinmike> present+ IsaDC
19:06:51 <jugglinmike> present+ Matt_King
19:07:00 <jugglinmike> Topic: Review agenda and next meeting dates
19:07:01 <Isa> Isa has joined #aria-at
19:07:34 <mmoss> mmoss has joined #aria-at
19:08:50 <jugglinmike> present+ Isa
19:09:07 <jugglinmike> Matt_King: Requests for changes to agenda?
19:09:24 <mmoss> +present
19:09:33 <jugglinmike> ChrisCuellar: We filed an issue related to consistency reports for disclosure
19:09:51 <jugglinmike> ChrisCuellar: It's not higher priority than anything on the agenda today, though
19:10:01 <jugglinmike> Matt_King: Then let's plan to discuss it next week
19:10:11 <jugglinmike> Matt_Link: Next Community Group Meeting: Wednesday April 23
19:10:19 <jugglinmike> Matt_King: Next AT Driver Subgroup meeting: Monday May 12
19:10:50 <jugglinmike> s/Matt_Link/Matt_King/
19:11:12 <Matt_King> Matt_King has joined #aria-at
19:11:43 <jugglinmike> Topic: Current status
19:11:54 <ChrisCuellar> The disclosure nav related issue raised by Bocoup to be discussed next week: https://github.com/w3c/aria-at-app/issues/1298
19:12:04 <jugglinmike> Matt_King: We're shooting for a new test plan roughly every week. We'll see how that goes
19:13:04 <jugglinmike> Matt_King: The stuff that we have in draft review right now--we will get a new one rolling today (the "rating radio group")
19:13:18 <jugglinmike> Matt_King: And we have one that's currently blocked by conflicts in JAWS results--"radio group with roving tab index"
19:13:26 <jugglinmike> Matt_King: I still haven't done anything on that to work on breaking the tie
19:13:47 <jugglinmike> Matt_King: Though we did raise it with Vispero yesterday, so we may get additional information on unblocking it when we talk to them next
19:13:58 <jugglinmike> Matt_King: Vertical temperature slider is on hold for the moment due to a couple issues
19:14:05 <jugglinmike> Matt_King: As is "disclosure navigation"
19:14:21 <jugglinmike> Matt_King: I think that for JAWS, we have to re-run the test plan because the output will be significantly different
19:14:43 <jugglinmike> Matt_King: For the other screen readers, we can probably leave the reports as-is because we don't anticipate changes (that is, for NVDA and Voiceover)
19:15:16 <jugglinmike> Matt_King: Coming up soon, will be the other disclosure navigation plan
19:15:32 <jugglinmike> Matt_King: Or maybe we should do the other disclosure first, and move that one down two weeks
19:15:39 <jugglinmike> Isa: Sounds good
19:15:58 <jugglinmike> Matt_King: So this will come up in about three weeks, instead. I will update the spreadsheet after this meeting
19:16:26 <jugglinmike> Speadsheet: https://docs.google.com/spreadsheets/d/14QIhQB9ufdUdZNhb3TNZzoZkGBq0oHawft-S5pm0ioY/edit?gid=0#gid=0
19:16:41 <jugglinmike> s/Spreadsheet:/The spreadsheet is available here:/
19:17:06 <jugglinmike> Isa: The versioning stuff with VoiceOver is set from 11-point-something to begin with
19:17:32 <jugglinmike> Matt_King: Oh, that's because I added it from the "report status" dialog. That's the only way to add the bot, and we have an open issue for being able to set the version number there
19:17:48 <jugglinmike> Matt_King: I think we need some other ways to run the bot, but we can discuss that elsewhere
19:18:24 <jugglinmike> Isa: It would be good to have a way of editing the AT version without deleting the report and re-adding the test plan for that specific AT from the Test Queue
19:18:49 <jugglinmike> Matt_King: If we were able to add it with the correct minimum version, then we wouldn't need to edit it
19:18:56 <jugglinmike> Isa: I agree
19:19:15 <jugglinmike> Matt_King: We need some volunteers for all three screen readers
19:19:25 <jugglinmike> Joe_Humbert: I can run it for JAWS
19:19:40 <jugglinmike> Matt_King: Yay! Can you assign yourself in the Test Queue?
19:19:43 <jugglinmike> Joe_Humbert: Sure
19:20:11 <jugglinmike> Matt_King: Next on the list is NVDA. The bot has recorded responses
19:20:23 <jugglinmike> s/has/has successfully/
19:20:29 <jugglinmike> Joe_Humbert: I can do all three if necessary
19:20:44 <jugglinmike> Isa: Thank you! I will assign you the bots' results for both NVDA and VoiceOver
19:20:50 <jugglinmike> Matt_King: Wow, Joe_Humbert's killing it
19:21:13 <jugglinmike> Joe_Humbert: I had a question about the vertical temperature slider
19:21:15 <jugglinmike> Matt_King: I left the vertical temperature slider off the agenda
19:21:21 <jugglinmike> Isa: We changed one test
19:21:33 <jugglinmike> Joe_Humbert: Do I need to go back and re-do the test for that one result?
19:21:45 <jugglinmike> Isa: It's not urgent because that test plan is blocked for now.
19:21:56 <jugglinmike> Matt_King: Before we finish the NVDA one, we're waiting on issue 1352
19:22:15 <jugglinmike> Matt_King: And for JAWS, we have a conflict that's very similar to the one we have for the "roving tab index radio group"
19:22:24 <jugglinmike> Matt_King: You can do the JAWS one, but hold off on the NVDA one
19:22:27 <jugglinmike> Joe_Humbert: Got it
19:22:58 <jugglinmike> mmoss: I can also run it for VoiceOver
19:23:18 <jugglinmike> Isa: I will run the bot, so you should have the responses available to you after the end of this meeting
19:23:29 <jugglinmike> Topic: Re-run of JAWS for color viewer slider
19:24:13 <jugglinmike> Matt_King: How did we get two rows in the table for the same minimum version of JAWS and the same browser? Is that a bug in the app?
19:24:30 <jugglinmike> Joe_Humbert: I also see those two rows
19:24:38 <jugglinmike> m
19:25:04 <jugglinmike> Matt_King: Is it possible to add the same one twice and create two rows?
19:25:27 <jugglinmike> howard-e: It may have stemmed from the functionality that was added to allow "exact" versus "minimum." It shouldn't be possible
19:25:45 <jugglinmike> Matt_King: I'm hesitant to hit "delete" because I don't want it to accidentally delete the row above
19:25:56 <jugglinmike> howard-e: I can't tell when it was introduced immediately.
19:26:10 <jugglinmike> howard-e: But I reproduced it locally right now, and I can confirm that it is safe to delete
19:26:30 <jugglinmike> Matt_King: Okay, I'm hitting "delete report" right now
19:27:04 <jugglinmike> Joe_Humbert: It's gone and the prior work is still there
19:27:14 <jugglinmike> Matt_King: Yay! We're good
19:27:32 <jugglinmike> Carmen: I will raise an issue related to that.
19:27:38 <jugglinmike> Joe_Humbert: It reported the same version in both of them
19:28:16 <jugglinmike> Joe_Humbert: Since I reloaded the page, I see myself assigned to the "rating radio group" ones, but NVDA and VoiceOver says "0 of 15 tests completed"
19:28:29 <jugglinmike> Isa: I assigned the bot's results to Joe_Humbert, so it should show them
19:29:01 <jugglinmike> Matt_King: "Evaluated" means you just have to assign the verdicts
19:29:14 <jugglinmike> s/completed"/evaluated"/
19:29:31 <jugglinmike> Joe_Humbert: Ah, yes. It says "evaluated" while the row for JAWS says "completed"
19:29:46 <jugglinmike> Isa: We need a second person for NVDA
19:30:04 <jugglinmike> dean: I can do the "rating radio group" for NVDA
19:30:17 <jugglinmike> Isa: I will run the bot and assign the result to dean
19:30:21 <jugglinmike> Matt_King: Awesome. Thank you
19:30:56 <jugglinmike> Topic: App issue 1367 - Prototyping decisions for manual testing on mobile
19:31:04 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1367
19:31:49 <jugglinmike> howard-e: Initially, when thinking about the manual tester's collection, we've always wanted to do it on-device or as close to the device as possible
19:32:04 <jugglinmike> howard-e: Perhaps using system utterances in the same way we're doing with NVDA and VoiceOver
19:32:15 <jugglinmike> howard-e: iOS's security precautions make that difficult
19:32:53 <jugglinmike> howard-e: For Android, we're considering an approach of taking video recordings and extracting text data from that video stream. But that's very much a fallback solution
19:33:13 <jugglinmike> howard-e: What we have built today, however, is a set of scripts that allow one to collect the utterances from TalkBack
19:33:35 <jugglinmike> howard-e: One can press the "run test" button and begin recording utterances
19:33:44 <jugglinmike> howard-e: We've confirmed that we can get system-level utterances from Android
19:33:56 <jugglinmike> howard-e: We've started work on this for iOS, but the feasibility is less clear
19:34:07 <jugglinmike> howard-e: That's all presented in this issue, and at the end, I've included four questions
19:34:32 <jugglinmike> howard-e: First: should we move forward with building an interface to make it easier for others to evaluate the Android-specific solution that I just described?
19:35:14 <jugglinmike> howard-e: Second: should we halt the approach for VoiceOver (because there is so much uncertainty) and move forward with the fallback video-recording approach?
19:35:34 <jugglinmike> howard-e: So it's really just the two questions
19:36:17 <jugglinmike> Matt_King: I'm personally very reluctant to do anything that relies on OCR, even as a fallback, because it seems very unreliable, potentially super-flaky, compute-heavy, and not very scalable
19:36:29 <jugglinmike> Matt_King: We can think of it as a fallback, but in my mind, it's a very distant fallback
19:37:17 <jugglinmike> Matt_King: And given where we're at right now, I think even a prototype on Android (one that just experiments with different ways to make the experience viable) could be really useful because we don't know what ANY method of doing this could feel like for human testers
19:38:18 <jugglinmike> Matt_King: We have humans who will manually run the tests on Android devices. Right now, doing that on a mobile device would be ridiculously time-intensive because it would involve human transcription (running the test runner on both the mobile device and a laptop)
19:38:35 <jugglinmike> Matt_King: The question is, what do we actually want the tester to perform on their device? Which steps?
19:38:55 <jugglinmike> Matt_King: Let's imagine how we eliminate the horror of this work?
19:39:44 <jugglinmike> james: If I could open a web page on a mobile device and sign in, it's a painful experience, sure. But if you're running the test plan on your laptop, it would be good if you could press a button on the page on your laptop and have that open the page on your phone
19:40:56 <jugglinmike> Matt_King: A dream user-experience would be if, somehow, your Android device is connected to what you're doing on your laptop in such a way that you would be able to run through the test plan on your laptop, but when you press "open test plan" on the laptop, it actually opens in your Android device. When you execute the prescribed gestures on the Android device, the computer is automatically collecting the AT response into the appropria
19:40:56 <jugglinmike> te field on the laptop
19:41:37 <jugglinmike> howard-e: I left out two details here. The setup scripts allow you to directly open the test page. At the end of the utterance collection (The end of the script or a closed process), the AT responses are available on the clipboard
19:42:06 <jugglinmike> james: Just to be clear, though, it's a non-starter to expect anyone to input assertion verdicts on the mobile device
19:42:41 <jugglinmike> james: Any mobile-oriented testing needs to assume that the primary driver is a computer connected to a mobile device (because form entry is unavoidable painful on mobile)
19:43:03 <jugglinmike> howard-e: I am not proposing a mobile-first experience. I want to limit the interaction with the Android device as much as possible--just to direct interaction with the test page
19:43:28 <jugglinmike> dean: Is there something like the VoiceOver recorder that will simply record everything and transcribe it--that could then be cut-and-pasted into the results?
19:43:44 <jugglinmike> Joe_Humbert: Possibly on Android. ADB is very powerful, but it's also very fickle
19:44:09 <jugglinmike> howard-e: That's the Android Debug Bridge. It's powering some of the scripts I'm presenting today
19:44:34 <jugglinmike> dean: If something could record what I'm doing with VoiceOver on my iPhone and just put it in Notes, I could open Notes on my laptop and just paste it into my test results
19:44:56 <jugglinmike> dean: It's not any more work than what I'm doing now on my desktop with the VoiceOver recorder
19:45:27 <jugglinmike> james: The VoiceOver testing experience, even with the recorder, is extremely inefficient. So using it as a baseline for our mobile testing solution doesn't seem ideal
19:45:32 <jugglinmike> dean: Agreed
19:46:22 <jugglinmike> howard-e: To your point, dean, I took a look at that for iOS. Even in generating the transcript which is provided by Apple for its own recording--that transcript is significantly incorrect
19:46:34 <jugglinmike> Matt_King: I don't want to worry too much about those kind of problems at this point
19:47:04 <github-bot> github-bot has joined #aria-at
19:47:08 <Joe_Humbert> I have to drop. I will try to get all the work done by next wednesday
19:47:10 <jugglinmike> Matt_King: If it turns out that we can do this well on Android but that problems with Apple products make it so that we can't do it well on Apple products, let's bring that as feedback to Apple and point to our work with Android as a baseline for what is reasonable or good
19:47:30 <jugglinmike> Matt_King: But it's really hard to have any conversation without having some kind of concrete experience
19:47:57 <jugglinmike> Matt_King: james described an ideal workflow, but a prototype isn't about implementing an idea. It's about taking a first step in that direction
19:48:19 <jugglinmike> Matt_King: So if howard-e could bring us a functional "step one", we can review and discuss next steps
19:48:28 <jugglinmike> howard-e: Got it
19:49:07 <jugglinmike> howard-e: This issue is now slightly outdated. The team at Bocoup has begun investigation at doing a similar approach on iOS. It's certainly less documented, so it's requiring additional research, but we're hopeful
19:49:15 <jugglinmike> james: What does the approach look like?
19:49:42 <jugglinmike> howard-e: It's using system-level intercepts of the voice. We wouldn't expect testers to use it on their personal devices. It probably gets us to a position that's better for automation
19:49:51 <jugglinmike> james: With a custom voice like we're using on macOS?
19:50:09 <jugglinmike> howard-e: No, that's not necessarily possible on iOS
19:50:28 <jugglinmike> ChrisCuellar: We're taking inspiration from security researchers
19:50:49 <jugglinmike> james: That may raise issues with Apple's acceptance of results collected in that manner
19:51:14 <jugglinmike> james: It's worth noting that eSpeak is available on iOS and it is open source
19:51:36 <jugglinmike> james: It's available in the iOS app store
19:51:50 <jugglinmike> Matt_King: That must mean that custom synthesizers are already supported
19:52:02 <jugglinmike> james: Yes. It's a universal macOS and iOS implementation
19:52:55 <jugglinmike> Matt_King: Anyway, let's start there with some "step 1" prototype for Android and see where it takes us
19:53:00 <jugglinmike> howard-e: Sounds good! Thanks!
19:53:03 <jugglinmike> Topic: Capturing lessons learned from JAWS AT driver experience
19:53:10 <dbaron> dbaron has joined #aria-at
19:53:27 <jugglinmike> Matt_King: I don't necessarily want to talk about the lessons learned right now, but I do want to discuss how we capture the lessons and respond to them
19:53:41 <jugglinmike> Matt_King: We could do a one-off meeting with some of the members here
19:54:16 <dbaron> dbaron has left #aria-at
19:54:37 <jugglinmike> Matt_King: We have such a small attendance on the automation subgroup meeting, it's realistic to move it
19:54:44 <jugglinmike> Matt_King: Maybe we want to do a poll to reschedule
19:55:00 <jugglinmike> Matt_King: It seemed like james wanted to do this sooner rather than later
19:55:18 <jugglinmike> james: It's not urgent. It just depends on whether Vispero would benefit from anything happening sooner. And also whether they can join that call
19:55:28 <jugglinmike> james: I'm happy for that to happen in the next automation meeting
19:55:48 <jugglinmike> Matt_King: So even if we had it in the first half of that meeting, you would be able to attend that first half?
19:55:58 <jugglinmike> james: Actually, I can join for the full hour on May 12
19:56:12 <jugglinmike> james: Okay, let's see if we can get Brett and Glenn to join that one
19:56:24 <jugglinmike> jugglinmike: I've been working with them directly, so I'll reach out
19:56:33 <jugglinmike> Topic: Issue 1355 - Version of Bot used for a run
19:56:47 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1355
19:57:25 <jugglinmike> Matt_King: As I understand the issue, it's if we say that the minimum version of NVDA requires the latest version of NVDA, then if we run it with the bot, this pull request ensures that it only runs if that bot has that version available... Is that right?
19:58:03 <jugglinmike> howard-e: For the longest time, it has been prioritizing the release date when it came to choosing the "latest". There was some oversight when it comes to patch release of older versions
19:58:21 <jugglinmike> howard-e: For instance, first version 13 is released, then 14 is released, then 13.1 is released
19:58:45 <jugglinmike> howard-e: One would expect version 14 to be the latest version, but the system was selecting 13.1
19:59:11 <jugglinmike> howard-e: We saw this mainly with NVDA versions and with VoiceOver versions. Mainly VoiceOver versions
19:59:26 <jugglinmike> Matt_King: We always have the option of not adding a patched version into the system at all
20:00:29 <jugglinmike> Matt_King: Is this primarily effecting the order that versions are presented in the drop down?
20:00:49 <jugglinmike> howard-e: Yes, and that order was also being implicitly used in bot selection. And that is where the true concern popped up
20:01:24 <jugglinmike> Matt_King: We have a challenge right now where, when we run with the bot, we aren't able to choose a specific version. We only have one way of kicking off the bot right now.
20:01:40 <jugglinmike> Matt_King: I'm going to raise an issue about reconsidering the use-cases for starting bot runs
20:03:07 <jugglinmike> Zakim, end the meeting
20:03:07 <Zakim> As of this point the attendees have been jugglinmike, Joe_Humbert, mmoss, Carmen, ChrisCuellar, howard-e, dean, james, IsaDC, Matt_King, present
20:03:09 <Zakim> RRSAgent, please draft minutes
20:03:11 <RRSAgent> I have made the request to generate https://www.w3.org/2025/04/17-aria-at-minutes.html Zakim
20:03:18 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
20:03:18 <Zakim> Zakim has left #aria-at
20:23:09 <jugglinmike> RRSAgent, leave
20:23:09 <RRSAgent> I see no action items