Meeting minutes
Matt_King: Today, we'll just walk through the test queue
Matt_King: Would anyone like to request any other items for the agenda?
IsaDC: Not from us
Matt_King: Okay. I can also give an update on my latest conversation with Vispero
Status update
Matt_King: Our goal is to reach 10 recommended test plans and 10 more in candidate review. This is starting to feel unrealistic as time goes on
Matt_King: We currently have eight in "candidate review" and 2 that we're working to get there
Matt_King: We have feedback from both Apple and Vispero about changes to Toggle and Radio
Matt_King: On "radio," I still need to find (or create) a GitHub issue for the change. That's a priority for me this weekend
Matt_King: I'll probably work on those changes myself to speed things up so you can focus on testing, IsaDC
Matt_King: Once we change those, I think we can make solid progress towards approval for several of the plans from Apple
Matt_King: We already have approval from Vispero on four of the plans
Matt_King: That's pretty good for getting those eight plans to "recommended"
Matt_King: NVAccess hasn't been able to participate fully, though, so we'll be moving forward without explicit approval from that organization
Matt_King: There are two more in the test plan right now. The goal is to get those into "Candidate review" as soon as possible
Matt_King: Based on the feedback we've received from Vispero and Apple, I think those are pretty low-risk; I think we know what needs to be done
Matt_King: Getting 10 more into "candidate review" though--that's a lot of testing, especially for the end-of-year deadline
IsaDC: "Action menu button" is ready for review in a pull request
Matt_King: I'll get the "Radio" changes cleared up this weekend so that you can move that forward
IsaDC: There are two more "links"
Matt_King: Those will be fast
IsaDC: Then there are sliders and tabs
Matt_King: I think the testing on "radio" not long and that "action menu button" is familiar
Matt_King: We have two months, roughly speaking, so it may still be possible
Matt_King: We'll take any minute of testing time we can get!
Test queue review
IsaDC: I sent an update to the community group list
IsaDC: Still pending for "action menu button" with "active descendant"
Matt_King: Can we re-assign Murray's task since we haven't heard from them in so long?
Matt_King: It seems as though I'm the only person besides IsaDC that's available for JAWS testing right now
IsaDC: It looks like my message to the mailing list didn't go through. I will re-send it soon
Matt_King: Could you also request help for testing with JAWS? Hadi or Alyssa aren't present today, but they might see a request there
Matt_King: As for disclosure, do we have any questions or problems?
IsaDC: The update for that one is out
IsaDC: I have assigned the NVDA and VoiceOver bots just to check on the results and correct them if necessary
IsaDC: I think we have Joe_Humbert assigned on NVDA, Dean on VoiceOver, and Hadi for JAWS
IsaDC: That means we have all testers assigned; we just have to run them
Matt_King: It looks like the bot ran for you folks; I think you just have to review the bot results and verify that they're accurate
Dean: I can't see the bot
Joe_Humbert: I see partially-completed test results from the bot
Dean: I'm not seeing them today, though I did see them before
IsaDC: Click on "start testing" for the test plan itself
Dean: Ah, I understand. It has stuff filled in. So it's my job to go through and verify that the responses are accurate
Matt_King: Right. You don't have to copy the text into the text box. You do have to run the tests to verify, though--and report on any unexpected behaviors
Dean: Got it
Matt_King: It looks like Hadi has completed six of the tests. Is that from a prior version?
IsaDC: For us and the bots, I deleted the report and re-added it from scratch. Should I do the same for Hadi?
Matt_King: Well, if the system believes six results are transferable, then I'm inclined to trust it
IsaDC: We only deleted assertions and changed priorities, so I think it should be fine
Matt_King: Yes, I believe that could be accurate, too
Matt_King: We'll see if you have any conflicts with Hadi, IsaDC. If you don't, that will give us even more confidence in those six results that Hadi currently has
IsaDC: We have conflicts with "navigation menu button"
Matt_King: On this one, we're still missing NVDA
IsaDC: I called for NVDA testers in my e-mail for that one
Matt_King: Oh my! Sixteen conflicts!
IsaDC: It's not as complicated as it seems. It's just that some of them are repetitive. For instance, "request information about menu button and menu item"
IsaDC: There, Dean didn't get any response
Matt_King: I love the new user experience for reviewing the conflicts!
Dean: When I run those keyboard sequences, I just get a little "bing" noise and that's it--nothing else happens
IsaDC: I thought you had the bot's responses recorded for that plan, but it looks like you didn't
Dean: No, there was nothing pre-filled when I ran those tests
Matt_King: Should we just go in order of the tests, then?
Matt_King: So test number three... You literally are not getting any output, Dean.
Matt_King: This is where you press ctrl-option-f3. Could it be a problem with the setup script?
IsaDC: No, because the bot and I got output
Dean: The only thing that I can think of is that my Mac keyboard. That would be bizarre
Matt_King: Oh! Does Ctrl-option-f3 give you output for anything?
Matt_King: Maybe the function keys aren't mapped to work as function keys on your system
Matt_King: Do you have a touch bar?
Dean: Yes, but I'm using the Apple Bluetooth external keyboard--the big one with all the keys
Matt_King: If you press ctrl-option-f8 on that keyboard, does it open the VoiceOver utility?
Dean: It does not
Matt_King: What if you press fn+ctrl+option+f8?
Matt_King: On the MacBook, at least, there is a setting for this. It might be on by default if you have VoiceOver on. But it might be off by default if you're using an external keyboard...
Matt_King: It's interesting that Command+f5 actually works for you, though, not fn+command+f5
Matt_King: In keyboard help mode, what if you press fn+ctrl+option+f3?
Dean: That's difficult to do on my external keyboard because it's the model with the number pad. The "fn" key is way on the right.
Matt_King: Well, at least now, we know that this is a hardware problem
Dean: Yeah, now that I know that pressing the "fn" key changes everything, I'm inclined to defer to IsaDC and the bot
Matt_King: On some keyboards, the fn+escape changes the function of the function keys
Matt_King: What happens if you press "fn+escape"?
Dean: I'll have to get back to you on that
IsaDC: There are some other conflicts which are unrelated to this problem
IsaDC: We're looking at several tests that have to do with "role menu item" (including 5, 6 and 7). Dean has marked an assertion as "fail"--"convey position of the focused item in the menu: 4"
Dean: It doesn't say "four". I stand by my verdict
IsaDC: I think VoiceOver makes this confusing because of the way it speaks
Dean: But we reported the same AT response, and nowhere in there does it say "four"
IsaDC: There are several assertions which we don't agree on. Another is "role menu item", which Dean passed and I failed
Matt_King: It says "menu", but it doesn't say "menu item"
Dean: It says "you are currently on a menu item"
IsaDC: Ah, that's a hint. We don't test those
Dean: but if it says "menu item," I think I should pass it for "role menu item"
Matt_King: Interesting
James_Scholes: James Craig at Apple has said that a majority of VoiceOver users turn off hints, so that they shouldn't be considered when testing
Matt_King: If the hint said something incorrect, we would record that as a problem
Matt_King: But if the extra text says something helpful, it might make sense to count it in favor of the assertion
IsaDC: I remember making this decision about not considering them for testing. There was something similar for JAWS where we decided to ignore it
IsaDC: We don't include "help" text from JAWS, e.g. "press arrow key to move by [...]"
Matt_King: I don't think we exclude that. The Bot certainly won't exclude it
Joe_Humbert: I've excluded text like that from VoiceOver in the past, but then you folks told me to include it, so I started including it after that
Matt_King: I think we did say that we need to include it. I think we need to be consistent here
Matt_King: The "hint" text is on by default in most of these screen readers
Dean: I am running with default standings
Matt_King: I don't remember us deciding to fail a test because some information appeared only in the hint
Matt_King: If I recall correctly, perhaps in "slider", the "hint" text had incorrect information about the orientation, and we failed the test because of that
Joe_Humbert: Would we not fail it if key information to pass an assertion is only in the hint text?
Matt_King: That's what we're talking about. In this, the information about the "menu item" is in a hint
Matt_King: I'm leaning towards saying that it passes
IsaDC: If we do, we'll have to review at least "aria activeDescendant" and possibly others as well
Joe_Humbert: There are failures about this in "action menu button"
IsaDC: If we pass them, then it would be inconsistent with the test plan that we marked as "final"
Joe_Humbert: To be clear: we're considering optional assertions here
Matt_King: I'm reviewing "aria activeDescendent", now
IsaDC: test number 4
Matt_King: We have fifteen failures... There is no hint here. It just says "menu actions you are currently in a menu"
James_Scholes: the hint is "you are currently in a menu"
IsaDC: Ah, that hint is different, then
Matt_King: In test 4, VoiceOver did pretty poorly. In test 5, seven passed and one failed.
Matt_King: Test 5 looks very similar to the one we're talking about. It does say "menu item". The only one that failed is "position in the menu is conveyed"
Matt_King: So it seems we passed the corresponding assertion in this test plan
Matt_King: Why is the corresponding assertion a "should" assertion in this test plan, though? That seems like an unintentional inconsistency
Matt_King: "role menu is conveyed" is a "must" assertion in this one
IsaDC: It should be a "may"
Matt_King: Oh, I'm looking at an old test plan!
Matt_King: We're out of time, though, so I'll look into this after the meeting
Matt_King: Our next meeting is on Wednesday, October 23
Matt_King: Thanks everyone!