17:02:17 <RRSAgent> RRSAgent has joined #aria-at
17:02:21 <RRSAgent> logging to https://www.w3.org/2025/10/08-aria-at-irc
17:02:21 <Zakim> RRSAgent, make logs Public
17:02:22 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
17:02:23 <jugglinmike> present+ jugglinmike
17:02:25 <jugglinmike> scribe+ jugglinmike
17:02:26 <ChrisCuellar> ChrisCuellar has joined #aria-at
17:02:30 <jugglinmike> present+ ChrisCuellar
17:02:35 <jugglinmike> present+ howard-e
17:02:40 <jugglinmike> present+ Matt_King
17:02:47 <jugglinmike> present+ mfairchild
17:02:49 <jugglinmike> present+ dean
17:02:56 <jugglinmike> present+ IsaDC
17:03:08 <jugglinmike> present+ elizabeth
17:03:57 <jugglinmike> present+ mmoss
17:04:05 <mmoss> mmoss has joined #aria-at
17:04:09 <jugglinmike> present+ mmoss
17:04:33 <thelounge> thelounge has joined #aria-at
17:04:36 <jugglinmike> present+ carmen
17:04:44 <Carmen> Carmen has joined #aria-at
17:05:24 <spectranaut_> spectranaut_ has joined #aria-at
17:05:25 <jugglinmike> topic: Review agenda and next meeting dates
17:06:40 <jugglinmike> Matt_King: Requests for changes to agenda?
17:07:41 <howard-e> howard-e has joined #aria-at
17:07:44 <howard-e> present+
17:08:12 <Carmen> present+
17:08:55 <Matt_King> Matt_King has joined #aria-at
17:09:01 <jugglinmike> Matt_King: hearing none, we'll stick with the agenda as scheduled
17:09:16 <jugglinmike> Matt_King: Next CG meeting: Thursday October 16
17:09:22 <jugglinmike> Matt_King: Next AT Driver Subgroup meeting: Monday October 13
17:10:03 <jugglinmike> jugglinmike: October 13 is a holiday in the US, so we will have to reschedule the AT Driver Subgroup meeting
17:10:17 <jugglinmike> IsaDC: PAC is also closed on that day
17:10:50 <jugglinmike> jugglinmike: Let's push it to the same time on following week--Monday, October 20
17:10:59 <jugglinmike> Topic: Current status
17:11:14 <jugglinmike> Matt_King: We changed "switch made from HTML checkbox" to "draft review"
17:11:20 <jugglinmike> Matt_King: Everything else is the same
17:11:31 <jugglinmike> Matt_King: We'll probably move on to the two-state checkbox example today
17:11:40 <jugglinmike> Matt_King: And soon after that: the quantity spin button
17:11:56 <jugglinmike> Topic: Issue 1521: Rename test plan directories
17:12:03 <jugglinmike> github: https://github.com/w3c/aria-at-app/issues/1521
17:12:13 <jugglinmike> Matt_King: This is about the repository where we write all of our tests
17:12:33 <jugglinmike> Matt_King: I am asking what it would take to possibly adopt a naming convention for the directories where the test plans lives
17:13:26 <jugglinmike> Matt_King: That's because I am preparing for the time when we have a lot more test plans that are not coming from the APG--test plans where people write the test case itself directly in the repository. The code, example, etc. will all be sourced from this repository. And the tests are more atomic: a specific ARIA feature in a much simpler context
17:13:31 <jugglinmike> Matt_King: We're planning to teach people how to do that at TPAC
17:13:46 <jugglinmike> Matt_King: I'm going to come up with an example of a simple test plan that they can essentially copy and then modify
17:14:07 <jugglinmike> Matt_King: Essentially, we have a few kinds of tests: you "navigate to", "get information about", and "operate"
17:14:33 <jugglinmike> Matt_King: Imagine we have a test plan for "aria-details" or "aria-errormessage". The directory could just be named according to the name of the property
17:15:01 <jugglinmike> Matt_King: I'm not sure what to do for ARIA roles because it could become confusing when an element and property have the same name
17:15:21 <jugglinmike> Matt_King: Someone might see a directory like "aria-button" and get confused, thinking that there is a property named "aria-button"
17:15:46 <jugglinmike> Matt_King: One approach is that we could put "apg-" as a prefix to the directories that originated from the APG
17:15:58 <jugglinmike> Matt_King: The plan is to have a lot more test plans in the repository which come from different places
17:16:23 <jugglinmike> Matt_King: In this issue, I list some of the downstream impact of changing directory names
17:16:47 <jugglinmike> ChrisCuellar: I think that captures it accurately!
17:17:26 <jugglinmike> Matt_King: Would prepending all the APG tests with "apg-" be a good approach?
17:17:37 <jugglinmike> mfairchild: It sounds fine to me, but is there a reason we couldn't use sub-folders?
17:17:52 <jugglinmike> Matt_King: I would love to do that. It adds other kinds of complexity, but I don't know if it's more or less complex
17:18:33 <jugglinmike> Matt_King: The biggest problem would be that we would have to decide ahead of time what all of the "sources" are, and the code in the system would have to account for that... Or I guess, maybe under the "test" directory, the system just sweeps all sub-folders.
17:19:27 <jugglinmike> ChrisCuellar: I'm thinking about precedence in the Web Platform Tests project, which makes heavy use of hierarchical directory structure
17:20:07 <jugglinmike> ChrisCuellar: There's no way to anticipate future entries, just due to the nature of the web platform. And I think building to support that open-endedness makes sense
17:20:40 <ChrisCuellar> scribe+
17:21:09 <ChrisCuellar> Matt_King: Did anyone have pushback against the idea of a "screen-reader" folder?
17:23:17 <ChrisCuellar> jugglinmike: I think having a directory structure by AT doesn't give us much flexibility, since we might be replicating tests across different AT folders. We might also run into version parity issues. Namespacing by AT seems not ideal.
17:24:44 <ChrisCuellar> Matt_King: Maybe that's not necessarily a knock on subdirectories but more about how we work with different AT's, which is a future-looking question. But the general question of subdirectories is still open I think if we segment by things like ARIA, HTML-AAM, APG, etc.
17:25:35 <jugglinmike> ChrisCuellar: I think subdirectory structure according to the type of feature under test makes sense. Unlike segmenting by AT, segmenting by feature will probably not involve test plans replicated across subdirectories
17:25:39 <jugglinmike> Matt_King: Oh yes, for sure
17:26:55 <ChrisCuellar> jugglinmike: But there's not a one-to-one mapping of test to feature, tho.
17:28:00 <ChrisCuellar> Matt_King: But the subdirectory structure could be reflect the scope of the test at least. It tells us the focus of the test even if it's not exactly testing one feature.
17:28:39 <ChrisCuellar> ... I think we have the same issues in WPT. Look at how accessibility is tested broadly across WPT.
17:29:11 <ChrisCuellar> jugglinmike: Bocoup is currently engaged in this problem with WPT specifically. We're trying to classify tests by web feature. It's a pretty complex problem.
17:30:11 <ChrisCuellar> Matt_King: Accessibility bugs are rarely traceable back to isolated features. Bugs are often contextual.
17:31:55 <jugglinmike> Matt_King: I would prefer subdirectories--it seems more in-line with what other repositories do, and it is generally easier for humans to parse. It's also harder to get wrong (especially since naming involves typing strings, but if a subdirectory already exists, then there is less of a hazard)
17:32:22 <jugglinmike> howard-e: It's a similar level-of-effort whether we use sub-directories or directory name prefixes
17:32:45 <jugglinmike> Matt_King: Based on what we've learned from the reference work so far is that we would have three sub-directories: html, aria, and apg
17:33:04 <jugglinmike> Matt_King: Okay, this is really helpful. In terms of making a plan for this issue, do you think we're better-positioned, now?
17:33:45 <jugglinmike> ChrisCuellar: I think so! We're going to move everything except for support.json and commands.json into a new sub-directory named "apg"
17:33:47 <jugglinmike> Matt_King: Correct
17:34:12 <jugglinmike> Matt_King: And we'll have a very crisp plan for when we actually enact this change
17:34:37 <jugglinmike> IsaDC: As the person who writes the test plans, the way I organize them is by sub-directories. So for me, that would be the best approach, if possible
17:34:42 <jugglinmike> Matt_King: Good!
17:35:04 <jugglinmike> Topic: Question about Disclosure Navigation Menu Example
17:35:13 <jugglinmike> Matt_King: This showed up in the test queue again
17:35:27 <jugglinmike> Matt_King: I was scratching my head because I was quite certain it wasn't present last week
17:35:36 <jugglinmike> Matt_King: It has a run by Joe_Humbert and a run by JAWS Bot
17:37:18 <jugglinmike> ChrisCuellar: Before we rolled out a bunch of fixes last week (for the last bits of the updates to the bot-running), we fixed an issue where some bot runs were getting misclassified as automated updates. That's where we're trying to collect everything when we're doing an automated re-run. There were just some issues in how we were filtering bot runs which we corrected in the previous release. I think that test plan was hidden previously
17:37:18 <jugglinmike> , and now it's back where it belongs
17:37:36 <jugglinmike> Matt_King: We had previously published this, I thought. I may be mistaken about that, though
17:38:04 <jugglinmike> IsaDC: We had "rating radio", but not disclosure
17:38:16 <jugglinmike> Matt_King: We were re-running disclosure for the latest version of JAWS.
17:38:36 <jugglinmike> Matt_King: Maybe I'm not remembering correctly, and it got hidden by mistake, and I just didn't notice that we never published it
17:39:23 <jugglinmike> ChrisCuellar: We can look into this. I want to verify my hypothesis. It sounds like you folks don't recognize this run, though
17:39:36 <jugglinmike> Matt_King: It says here that Joe_Humbert's run has zero verdicts
17:39:49 <jugglinmike> ChrisCuellar: It looks like it might be a bot run that got re-assigned but never finishd
17:40:11 <jugglinmike> Matt_King: Yeah, and the other one looks like an automated bot run where almost all the verdict got assigned...
17:40:36 <jugglinmike> Matt_King: If you could do a little investigation to determine why this happened
17:41:04 <jugglinmike> Matt_King: It's odd that all the responses matched but the verdicts were not assigned
17:42:23 <jugglinmike> ChrisCuellar: It appears as though all the tests were run by Hadi and IsaDC
17:42:28 <jugglinmike> Carmen: I can raise an issue for this
17:42:32 <jugglinmike> Matt_King: Alright, thanks
17:42:38 <jugglinmike> Topic: Running Switch test plan
17:42:59 <jugglinmike> Matt_King: I commented on your issue, Elizabeth
17:43:17 <jugglinmike> Matt_King: I think it's possible that the cause of your problem is that the function keys aren't doing what they're supposed to do
17:43:36 <jugglinmike> Matt_King: I found an article
17:43:49 <Matt_King> https://support.apple.com/guide/voiceover/change-function-key-behavior-mchlp2685/10/mac/26
17:44:26 <jugglinmike> Matt_King: This is an Apple article about how you change the "function" behavior so that you don't have to press the "Fn" key
17:44:42 <jugglinmike> Elizabeth: I'm getting the correct response, now, actually
17:44:56 <jugglinmike> IsaDC: The conflicts are gone, now, so this Test Plan is done
17:45:14 <jugglinmike> IsaDC: I've just now marked it as such
17:45:31 <jugglinmike> Matt_King: Awesome. We can advance that test plan to "candidate" because we have all three ATs done
17:45:40 <jugglinmike> Topic: Running test plan for Tabs with Automatic Activation
17:45:53 <jugglinmike> Matt_King: We had nine conflicting results with Hadi, but Hadi isn't present today
17:45:57 <jugglinmike> IsaDC: I can e-mail him
17:46:10 <jugglinmike> Topic: Running test plan for Tabs with Manual Activation
17:46:36 <jugglinmike> Matt_King: IsaDC iss still working on this one in JAWS
17:46:47 <jugglinmike> IsaDC: I'm going to run that test plan for JAWS
17:46:52 <jugglinmike> Matt_King: For NVDA, we need one more tester
17:47:20 <jugglinmike> Matt_King: Do we have somebody who is available to take on the NVDA run?
17:47:40 <jugglinmike> Elizabeth: I'm done with "switch", but I don't have access to a PC at the moment
17:48:32 <jugglinmike> dean: I can test "Tabs with Manual Activation" with NVDA. That was my intention--to do that once I finished it with VoiceOver, but I'll just do that, now.
17:48:36 <jugglinmike> Matt_King: Cool!
17:48:46 <jugglinmike> Topic: Updating reports to latest screen reader versions
17:49:57 <jugglinmike> Matt_King: We have this feature in the system for when a new version of a screen reader comes out. We can have the bot run the test plans for all the published reports. The bot goes through and runs the test plan with a new version of the screen reader. If it gets the same response, then it assigns the same verdicts
17:50:40 <jugglinmike> Matt_King: Bocoup folks: does the bot also carry forward the designation of "untestable" in this case?
17:50:53 <jugglinmike> Matt_King: We should verify what it does with untestables and side-effects
17:51:07 <jugglinmike> Matt_King: Anyway, back to the context of this topic
17:51:34 <jugglinmike> Matt_King: If the bot can re-assign all the verdicts based on matching with historic AT responses, then it immediately publishes a new report
17:51:41 <jugglinmike> Matt_King: Otherwise, we need to have a human review
17:52:13 <jugglinmike> Matt_King: Because of the differences in the way the bots record output and the way that humans report output, the distinction between the AT responses can be trivial
17:52:17 <jugglinmike> Matt_King: But other times, it is not
17:52:40 <jugglinmike> Matt_King: for instance, the presence or absence of hint text can interfere with the matching heuristic
17:53:54 <jugglinmike> Matt_King: We can find the tests requiring oversight in the test queue via a button at the top. There are filters there: one for manual test runs (that's the default), and one for automated updates. If you press the filter for "automated updates", then right now, you will see six entries.
17:54:37 <jugglinmike> Matt_King: So the job is to just go through and look at these responses and determine whether or not the new response passes or fails the verdict.
17:54:54 <jugglinmike> Matt_King: So we have six of these ready to go. Is there anybody who wants to take on some of this work?
17:55:04 <jugglinmike> Elizabeth: Yes, I have some time for this, actually
17:55:18 <jugglinmike> Matt_King: Okay, I can assign the first one to you right now (the "action menu button")
17:55:41 <jugglinmike> ChrisCuellar: It should stay in the "automated" section even after you assign it
17:56:16 <jugglinmike> Matt_King: I assigned the first one to you, Elizabeth
17:57:00 <jugglinmike> ChrisCuellar: Yup, I see it assigned to Elizabeth, and it is indeed still in the "automated updates" section
17:57:19 <jugglinmike> Matt_King: This is a pretty good case for why we might want a new filter for "My Test Runs"
17:57:39 <jugglinmike> Matt_King: Elizabeth, when you open that, you'll see it looks like a little bit different
17:58:53 <jugglinmike> Matt_King: For each command, you'll see what the response was for the previous report, and in the "edit" box, you'll see what the response was in the latest automated run. We don't want you to actually edit that response; we just want you to report whether the new response satisfies the assertions
18:01:26 <jugglinmike> Zakim, end the meeting
18:01:26 <Zakim> As of this point the attendees have been jugglinmike, ChrisCuellar, howard-e, Matt_King, mfairchild, dean, IsaDC, elizabeth, mmoss, carmen
18:01:28 <Zakim> RRSAgent, please draft minutes
18:01:29 <RRSAgent> I have made the request to generate https://www.w3.org/2025/10/08-aria-at-minutes.html Zakim
18:01:36 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
18:01:36 <Zakim> Zakim has left #aria-at
18:01:59 <jugglinmike> RRSAgent, leave
18:01:59 <RRSAgent> I see no action items