19:04:25 <RRSAgent> RRSAgent has joined #aria-at
19:04:29 <RRSAgent> logging to https://www.w3.org/2024/10/17-aria-at-irc
19:04:29 <Zakim> RRSAgent, make logs Public
19:04:30 <Zakim> please title this meeting ("meeting: ..."), jugglinmike
19:04:36 <jugglinmike> present+ jugglinmike
19:04:38 <jugglinmike> scribe+ jugglinmike
19:05:09 <jugglinmike> meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference
19:05:13 <Joe_Humbert> present+
19:05:16 <Matt_King> Matt_King has joined #aria-at
19:05:21 <jugglinmike> present+ Matt_King
19:05:25 <jugglinmike> present+ Joe_Humbert
19:05:26 <Matt_King> present+
19:06:10 <jugglinmike> Matt_King: Today, we'll just walk through the test queue
19:06:23 <jugglinmike> Matt_King: Would anyone like to request any other items for the agenda?
19:06:27 <jugglinmike> present+ IsaDC
19:06:33 <jugglinmike> IsaDC: Not from us
19:06:54 <jugglinmike> Matt_King: Okay. I can also give an update on my latest conversation with Vispero
19:07:00 <jugglinmike> Topic: Status update
19:07:42 <jugglinmike> Matt_King: Our goal is to reach 10 recommended test plans and 10 more in candidate review. This is starting to feel unrealistic as time goes on
19:07:59 <jugglinmike> Matt_King: We currently have eight in "candidate review" and 2 that we're working to get there
19:08:19 <jugglinmike> Matt_King: We have feedback from both Apple and Vispero about changes to Toggle and Radio
19:08:46 <jugglinmike> Matt_King: On "radio," I still need to find (or create) a GitHub issue for the change. That's a priority for me this weekend
19:09:03 <jugglinmike> Matt_King: I'll probably work on those changes myself to speed things up so you can focus on testing, IsaDC
19:09:26 <jugglinmike> Matt_King: Once we change those, I think we can make solid progress towards approval for several of the plans from Apple
19:09:40 <jugglinmike> Matt_King: We already have approval from Vispero on four of the plans
19:09:53 <jugglinmike> Matt_King: That's pretty good for getting those eight plans to "recommended"
19:10:14 <jugglinmike> Matt_King: NVAccess hasn't been able to participate fully, though, so we'll be moving forward without explicit approval from that organization
19:10:31 <jugglinmike> Matt_King: There are two more in the test plan right now. The goal is to get those into "Candidate review" as soon as possible
19:10:47 <ChrisCuellar> ChrisCuellar has joined #aria-at
19:10:56 <jugglinmike> Matt_King: Based on the feedback we've received from Vispero and Apple, I think those are pretty low-risk; I think we know what needs to be done
19:11:28 <jugglinmike> Matt_King: Getting 10 more into "candidate review" though--that's a lot of testing, especially for the end-of-year deadline
19:11:42 <jugglinmike> IsaDC: "Action menu button" is ready for review in a pull request
19:12:00 <jugglinmike> Matt_King: I'll get the "Radio" changes cleared up this weekend so that you can move that forward
19:12:07 <jugglinmike> IsaDC: There are two more "links"
19:12:11 <jugglinmike> Matt_King: Those will be fast
19:12:18 <jugglinmike> IsaDC: Then there are sliders and tabs
19:12:38 <jugglinmike> Matt_King: I think the testing on "radio" not long and that "action menu button" is familiar
19:12:51 <jugglinmike> Matt_King: We have two months, roughly speaking, so it may still be possible
19:13:02 <jugglinmike> Matt_King: We'll take any minute of testing time we can get!
19:13:16 <jugglinmike> present+ ChrisCuellar
19:13:50 <jugglinmike> Topic: Test queue review
19:13:57 <jugglinmike> IsaDC: I sent an update to the community group list
19:14:33 <jugglinmike> IsaDC: Still pending for "action menu button" with "active descendant"
19:14:58 <jugglinmike> Matt_King: Can we re-assign Murray's task since we haven't heard from them in so long?
19:15:47 <jugglinmike> Matt_King: It seems as though I'm the only person besides IsaDC that's available for JAWS testing right now
19:19:11 <jugglinmike> IsaDC: It looks like my message to the mailing list didn't go through. I will re-send it soon
19:20:20 <jugglinmike> Matt_King: Could you also request help for testing with JAWS? Hadi or Alyssa aren't present today, but they might see a request there
19:20:36 <jugglinmike> Matt_King: As for disclosure, do we have any questions or problems?
19:21:06 <jugglinmike> IsaDC: The update for that one is out
19:21:15 <jugglinmike> IsaDC: I have assigned the NVDA and VoiceOver bots just to check on the results and correct them if necessary
19:21:31 <jugglinmike> IsaDC: I think we have Joe_Humbert assigned on NVDA, Dean on VoiceOver, and Hadi for JAWS
19:21:45 <jugglinmike> IsaDC: That means we have all testers assigned; we just have to run them
19:22:05 <jugglinmike> Matt_King: It looks like the bot ran for you folks; I think you just have to review the bot results and verify that they're accurate
19:22:10 <jugglinmike> present+ Dean
19:22:19 <jugglinmike> Dean: I can't see the bot
19:22:37 <jugglinmike> Joe_Humbert: I see partially-completed test results from the bot
19:22:48 <jugglinmike> Dean: I'm not seeing them today, though I did see them before
19:23:40 <jugglinmike> IsaDC: Click on "start testing" for the test plan itself
19:24:04 <jugglinmike> Dean: Ah, I understand. It has stuff filled in. So it's my job to go through and verify that the responses are accurate
19:24:34 <jugglinmike> Matt_King: Right. You don't have to copy the text into the text box. You do have to run the tests to verify, though--and report on any unexpected behaviors
19:24:40 <jugglinmike> Dean: Got it
19:25:00 <jugglinmike> Matt_King: It looks like Hadi has completed six of the tests. Is that from a prior version?
19:25:22 <jugglinmike> IsaDC: For us and the bots, I deleted the report and re-added it from scratch. Should I do the same for Hadi?
19:25:45 <jugglinmike> Matt_King: Well, if the system believes six results are transferable, then I'm inclined to trust it
19:26:00 <jugglinmike> IsaDC: We only deleted assertions and changed priorities, so I think it should be fine
19:26:10 <jugglinmike> Matt_King: Yes, I believe that could be accurate, too
19:26:37 <jugglinmike> Matt_King: We'll see if you have any conflicts with Hadi. If you don't, that will give us even more confidence in those six results that Hadi currently has
19:26:54 <jugglinmike> s/with Hadi/with Hadi, IsaDC/
19:27:21 <jugglinmike> IsaDC: We have conflicts with "navigation menu button"
19:27:55 <jugglinmike> Matt_King: On this one, we're still missing NVDA
19:28:04 <jugglinmike> IsaDC: I called for NVDA testers in my e-mail for that one
19:28:13 <jugglinmike> Matt_King: Oh my! Sixteen conflicts!
19:28:47 <jugglinmike> IsaDC: It's not as complicated as it seems. It's just that some of them are repetitive. For instance, "request information about menu button and menu item"
19:28:57 <jugglinmike> IsaDC: There, Dean didn't get any response
19:29:09 <jugglinmike> Matt_King: I love the new user experience for reviewing the conflicts!
19:29:32 <jugglinmike> Dean: When I run those keyboard sequences, I just get a little "bing" noise and that's it--nothing else happens
19:29:48 <jugglinmike> IsaDC: I thought you had the bot's responses recorded for that plan, but it looks like you didn't
19:29:57 <jugglinmike> Dean: No, there was nothing pre-filled when I ran those tests
19:30:13 <jugglinmike> Matt_King: Should we just go in order of the tests, then?
19:30:50 <jugglinmike> Matt_King: So test number three... You literally are not getting any output, Dean.
19:31:05 <jugglinmike> Matt_King: This is where you press ctrl-option-f3. Could it be a problem with the setup script?
19:31:06 <Joe_Humbert> Joe_Humbert has joined #aria-at
19:31:13 <jugglinmike> IsaDC: No, because the bot and I got output
19:31:33 <jugglinmike> Dean: The only thing that I can think of is that my Mac keyboard. That would be bizarre
19:31:52 <jugglinmike> Matt_King: Oh! Does Ctrl-option-f3 give you output for anything?
19:32:04 <jugglinmike> Matt_King: Maybe the function keys aren't mapped to work as function keys on your system
19:32:13 <jugglinmike> Matt_King: Do you have a touch bar?
19:32:36 <jugglinmike> Dean: Yes, but I'm using the Apple Bluetooth external keyboard--the big one with all the keys
19:32:56 <jugglinmike> Matt_King: If you press ctrl-option-f8 on that keyboard, does it open the VoiceOver utility?
19:33:43 <jugglinmike> Dean: It does not
19:34:03 <jugglinmike> Matt_King: What if you press fn+ctrl+option+f8?
19:34:54 <jugglinmike> Matt_King: On the MacBook, at least, there is a setting for this. It might be on by default if you have VoiceOver on. But it might be off by default if you're using an external keyboard...
19:35:39 <jugglinmike> Matt_King: It's interesting that Command+f5 actually works for you, though, not fn+command+f5
19:36:51 <jugglinmike> Matt_King: In keyboard help mode, what if you press fn+ctrl+option+f3?
19:37:53 <jugglinmike> Dean: That's difficult to do on my external keyboard because it's the model with the number pad. The "fn" key is way on the right.
19:38:08 <jugglinmike> Matt_King: Well, at least now, we know that this is a hardware problem
19:38:28 <jugglinmike> Dean: Yeah, now that I know that pressing the "fn" key changes everything, I'm inclined to defer to IsaDC and the bot
19:38:54 <jugglinmike> Matt_King: On some keyboards, the fn+escape changes the function of the function keys
19:39:18 <jugglinmike> Matt_King: What happens if you press "fn+escape"
19:39:40 <jugglinmike> s/ape"/ape"?/
19:39:53 <jugglinmike> Dean: I'll have to get back to you on that
19:40:04 <jugglinmike> IsaDC: There are some other conflicts which are unrelated to this problem
19:41:50 <jugglinmike> IsaDC: We're looking at several tests that have to do with "role menu item" (including 5, 6 and 7). Dean has marked an assertion as "fail"--"convey position of the focused item in the menu: 4"
19:42:53 <jugglinmike> Dean: It doesn't say "four". I stand by my verdict
19:43:06 <jugglinmike> IsaDC: I think VoiceOver makes this confusing because of the way it speaks
19:43:34 <jugglinmike> Dean: But we reported the same AT response, and nowhere in there does it say "four"
19:43:56 <jugglinmike> IsaDC: There are several assertions which we don't agree on. Another is "role menu item", which Dean passed and I failed
19:46:00 <jugglinmike> Matt_King: It says "menu", but it doesn't say "menu item"
19:46:24 <jugglinmike> Dean: It says "you are currently on a menu item"
19:46:32 <jugglinmike> IsaDC: Ah, that's a hint. We don't test those
19:46:50 <jugglinmike> Dean: but if it says "menu item," I think I should pass it for "role menu item"
19:46:55 <jugglinmike> Matt_King: Interesting
19:47:01 <jugglinmike> present+ James_Scholes
19:47:43 <jugglinmike> James_Scholes: James Craig at Apple has said that a majority of VoiceOver users turn off hints, so that they shouldn't be considered when testing
19:48:04 <jugglinmike> Matt_King: If the hint said something incorrect, we would record that as a problem
19:48:38 <jugglinmike> Matt_King: But if the extra text says something helpful, it might make sense to count it in favor of the assertion
19:49:03 <jugglinmike> IsaDC: I remember making this decision about not considering them for testing. There was something similar for JAWS where we decided to ignore it
19:49:28 <jugglinmike> IsaDC: We don't include "help" text from JAWS, e.g. "press arrow key to move by [...]"
19:49:45 <jugglinmike> Matt_King: I don't think we exclude that. The Bot certainly won't exclude it
19:50:06 <jugglinmike> Joe_Humbert: I've excluded text like that from VoiceOver in the past, but then you folks told me to include it, so I started including it after that
19:50:20 <jugglinmike> Matt_King: I think we did say that we need to include it. I think we need to be consistent here
19:50:40 <jugglinmike> Matt_King: The "hint" text is on by default in most of these screen readers
19:50:54 <jugglinmike> Dean: I am running with default standings
19:51:30 <jugglinmike> Matt_King: I don't remember us deciding to fail a test because some information appeared only in the hint
19:52:25 <jugglinmike> Matt_King: If I recall correctly, perhaps in "slider", the "hint" text had incorrect information about the orientation, and we failed the test because of that
19:52:53 <jugglinmike> Joe_Humbert: Would we not fail it if key information to pass an assertion is only in the hint text?
19:53:15 <jugglinmike> Matt_King: That's what we're talking about. In this, the information about the "menu item" is in a hint
19:53:22 <jugglinmike> Matt_King: I'm leaning towards saying that it passes
19:53:47 <jugglinmike> IsaDC: If we do, we'll have to review at least "aria activeDescendant" and possibly others as well
19:54:21 <jugglinmike> Joe_Humbert: There are failures about this in "action menu button"
19:54:43 <jugglinmike> IsaDC: If we pass them, then it would be inconsistent with the test plan that we marked as "final"
19:55:15 <jugglinmike> Joe_Humbert: To be clear: we're considering optional assertions here
19:55:40 <jugglinmike> Matt_King: I'm reviewing "aria activeDescendent", now
19:55:47 <jugglinmike> IsaDC: test number 4
19:56:25 <jugglinmike> Matt_King: We have fifteen failures... There is no hint here. It just says "menu actions you are currently in a menu"
19:56:35 <jugglinmike> James_Scholes: the hint is "you are currently in a menu"
19:56:42 <jugglinmike> IsaDC: Ah, that hint is different, then
19:58:13 <jugglinmike> Matt_King: In test 4, VoiceOver did pretty poorly. In test 5, seven passed and one failed.
19:58:49 <jugglinmike> Matt_King: Test 5 looks very similar to the one we're talking about. It does say "menu item". The only one that failed is "position in the menu is conveyed"
19:59:06 <jugglinmike> Matt_King: So it seems we passed the corresponding assertion in this test plan
20:00:36 <jugglinmike> Matt_King: Why is the corresponding assertion a "should" assertion in this test plan, though? That seems like an unintentional inconsistency
20:00:56 <jugglinmike> Matt_King: "role menu is conveyed" is a "must" assertion in this one
20:01:02 <jugglinmike> IsaDC: It should be a "may"
20:01:40 <jugglinmike> Matt_King: Oh, I'm looking at an old test plan!
20:01:52 <jugglinmike> Matt_King: We're out of time, though, so I'll look into this after the meeting
20:05:03 <jugglinmike> Matt_King: Our next meeting is on Wednesday, October 23
20:06:48 <jugglinmike> Matt_King: Thanks everyone!
20:06:58 <jugglinmike> Zakim, end the meeting
20:06:58 <Zakim> As of this point the attendees have been jugglinmike, Joe_Humbert, Matt_King, IsaDC, ChrisCuellar, Dean, James_Scholes
20:07:00 <Zakim> RRSAgent, please draft minutes
20:07:01 <RRSAgent> I have made the request to generate https://www.w3.org/2024/10/17-aria-at-minutes.html Zakim
20:07:08 <Zakim> I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent.  Goodbye
20:07:08 <Zakim> Zakim has left #aria-at
20:08:45 <jugglinmike> RRSAgent, leave
20:08:45 <RRSAgent> I see no action items