Meeting minutes
Matt_King: We'll meet on Thursday of next week
Matt_King: However, we will NOT meet the following week (when we would normally meet on Wednesday the 24)
Matt_King: I didn't have time to create an agenda for today's meeting; we'll simply review the test plans in the test queue
Disclosure of Answers to Frequently Asked Questions Example
Matt_King: The only screen reader we have left to work on is VoiceOver
Matt_King: It looks like Joe_Humbert is all the way done
mmoss: I'll have my part done by the end of the day, for sure
Matt_King: Awesome!
Joe_Humbert: I haven't raised any issues in my latest round of testing
Rating Radio Group
Matt_King: We're skipping this for now; it's still on hold
Switch Example
Matt_King: We're only working on VoiceOver on this Test Plan, as well
Matt_King: This one doesn't have any conflicts, either
Matt_King: Elizabeth and Joe_Humbert ran this one
Joe_Humbert: I ran this a couple days ago. Again, I reported no issues
Matt_King: Elizabeth hasn't been here for a little while. I think IsaDC reached out to Elizabeth before IsaDC left, but I don't know the outcome of that
Matt_King: We may have to re-assign Elizabeth's work
Joe_Humbert: I will double-check the commands that I ran to be sure that I accurately reported the "untestable" condition
Switch Example Using HTML Button
Matt_King: In this one, we have five conflicts for VoiceOver
Matt_King: Joe_Humbert's results and dean's results are conflicting
Joe_Humbert: I marked the first as "unexpected behavior" and dean marked it as nothing
Matt_King: This is test 2, which is a "navigate backwards" test. Also 7 and 8
Joe_Humbert: in test 7, I listed unexpected behaviors, but dean did not. We reported the same AT responses and assigned the same assertion verdicts
dean: I think Joe_Humbert is correct on test 7
Matt_King: so on test 1, though, it did find the button?
Joe_Humbert: No, on test 1, since its VoiceOver, instead of saying "no buttons," it goes to the "run test setup" button. We both indicated that
dean: I'm looking at all these, and like Joe_Humbert said, the only difference is that I didn't mark anything as "unexpected behavior." I have no problem changing to what Joe_Humbert reported
Joe_Humbert: When I checked "untestable", it automatically changed the radio button to "unexpected behaviors" to "yes"
dean: I don't think it did that for me
mmoss: I observed the same behavior as what Joe_Humbert just described
Matt_King: And that's the way it's supposed to function
Joe_Humbert: Maybe dean was able to get around that by testing with an earlier version of the application
dean: Or I may have manually changed "unexpected behaviors" from "yes" to "no"
jugglinmike: The "no" radio button should be disabled when "untestable" is checked
Matt_King: Well now, I'm questioning whether or not we should have the "b" command in the test plan...
Matt_King: The reason we included "b" in the first place is because we thought two of the three screen readers were treating switches a buttons.
Matt_King: We were going to make the case that, "hey, it's called a switch button, so shouldn't it be included?"
Matt_King: But at least when it's coded this way, it seems to be working
Joe_Humbert: It looks like it goes to the "run test setup" button
Joe_Humbert: So apparently VoiceOver and JAWS don't work with a switch when you use the "b" key
Matt_King: Maybe we should remove that test
Joe_Humbert: I wondered about this, but I wasn't sure because I haven't done enough testing of switching in HTML contexts to know which buttons work and which do not
Matt_King: I wonder what iOS does when you switch to buttons on the rotor...
Joe_Humbert: You can't use button in native; it only works in HTML
Joe_Humbert: Only a subset of the rotor is available in native
Matt_King: If we remove the "b" command, then the conflicts go away
Joe_Humbert: Some of the conflicts go away. Tests 7 and 8 have different conflicts
Matt_King: I talked to James about this, and we decided to collect the data and decide based on that. Maybe we already have enough data to make a decision
Matt_King: I thought I had tested the "b" key with at least two screen readers and found it to work. Now, I'm questioning my memory
Matt_King: What did NVDA do in test 1?
Joe_Humbert: It worked just fine in browse mode
Matt_King: So NVDA is the only one...
Matt_King: Any change to what Apple does in this space would really be swimming upstream
Matt_King: When IsaDC returns, we'll talk though this
Matt_King: I think we know what the path forward for fixing the conflicts is right now
dean: I've already fixed most of them; I'm fixing the final one was we speak
Matt_King: Okay, as for test 7...
Joe_Humbert: dean and I got the same output. It's very strange--the screen reader says "no action available off"
Joe_Humbert: The "off" is confusing. This comes up in another test. It says "no action available off", but the visible switch updates to the correct state
Joe_Humbert: The visual updates but the screen reader is not reporting that
Joe_Humbert: This is operating a switch that is off to turn it on
dean: You can toggle that thing back and forth (off and on), and you can visually perceive the change, but it will not be announced by VoiceOver
Joe_Humbert: Right
Joe_Humbert: This occurs in test 8. There, dean marked it as "pass" because it technically said "off", but I don't think that "off" is related to any state. Because no matter what you do, it reports "off"
Joe_Humbert: It does report the state when you ask for it. It's this command (and I'm 95% sure on this), it updates the state of the button visually changes, but all VoiceOver says is "no action available off" regardless of the state of the switch
Joe_Humbert: For this test, you get no output when you press "space" or "enter"
Joe_Humbert: It seems that, at least with a switch using an HTML button, there are significant problems with VoiceOver
Joe_Humbert: that's why I marked "unexpected behavior" even though dean did not--I wanted to have this conversation
Joe_Humbert: I don't think I've ever heard the screen reader say "no action available"
Matt_King: It normally says when actions ARE available, but it normally wouldn't say "no actions available" unless you are trying to execute an action
Joe_Humbert: Right. But I've never heard that output from VoiceOver, so I was very confused by this
Matt_King: Isn't there a side-effect in the form that is related to "incorrect or misleading output" or something like that?
Matt_King: The option named "other" would be fine, too
Matt_King: I do think we should be reporting a side-effect, especially when the switch appears to be on but it says "off"
Joe_Humbert: What should we say for the details?
Joe_Humbert: "VoiceOver is announcing incorrect information, but the visual state changes"
Matt_King: That's fine
dean: I wanted to make sure that we had a note somewhere that these things visually change but are not announced
Joe_Humbert: There may be other conflicts in JAWS and NVDA. I'll double-check
dean: I'm going to finish my tests today/tomorrow because I will be out for about a week
Joe_Humbert: So there may be other conflicts that are not this (because we know how to resolve these conflicts, now)
Tabs with Automatic Activation
Matt_King: For this one, we have JAWS and VoiceOver in progress
Matt_King: VoiceOver is done. It says "99% done". I'm not sure why
Matt_King: There are four conflicts--they are in test 2 and test 4
Matt_King: Joe_Humbert and mmoss have different output
Matt_King: You're both using the same versions of the software
mmoss: We have the same output; I just didn't include the entire text
Matt_King: In test 2, mmoss says "you are currently on a text element." I don't see that on Joe_Humbert's output
Joe_Humbert: It is the last line
Joe_Humbert: mmoss just didn't include all of the text output for the two commands--he just included the final line
Matt_King: So for both of you, it read the entire tab panel
mmoss: Correct
Joe_Humbert: I'm happy to update
Matt_King: Oh, right. Now, I understand
mmoss: Instead of navigating backwards into the tab list and onto a tab, it navigated to the tab panel
Matt_King: When you navigate backwards--how in the world did it read that whole panel content...?
mmoss: It didn't read the tab
Matt_King: Did the focus ever reach the tab on "Peter Muller"?
Joe_Humbert: I can't remember
Matt_King: Why does it say "you are on a text input"?
Joe_Humbert: It actually says "you are on a text element", which makes sense because it is a paragraph
Matt_King: Ah. That's a weird thing to say, though
Joe_Humbert: I imagine most people turn that off
Joe_Humbert: Do I need to add an unexpected behavior, or does mmoss need to remove the unexpected behavior?
Matt_King: We should be capturing the whole output, though
Matt_King: There is an unexpected behavior, here. I think the way that mmoss described it looks pretty accurate
Joe_Humbert: Should we change it to "the cursor moved unexpectedly"?
Matt_King: I think that is very much the case. The reading cursor moved in an unexpected manner
mmoss: I'll update accordingly
Joe_Humbert: And once you do that, mmoss, I will update mine to match yours
Matt_King: That was for test 2
Joe_Humbert: There was a similar problem with "shift+j"
Joe_Humbert: It said "form element not found"
Matt_King: Is that also what it did for "j"?
mmoss: No, it was only "shift+j"
Joe_Humbert: I'm happy to update my unexpected behaviors if that's what we need to do
Matt_King: It just says, "no found". Does the reading cursor rectangle move at all? Or does it just stay put?
Matt_King: Actually, what is the reading cursor visually placed on?
Joe_Humbert: I would have to revisit this to say
mmoss: I just tested real quick and confirmed that the visual cursor doesn't move at all
Matt_King: This sounds like an untestable situation similar to the other "shift+j" ones
Joe_Humbert: In the other cases, we were saying just that the cursor didn't move
Matt_King: Okay, yes, I think that's the way forward here, too
mmoss: I'll update accordingly
Joe_Humbert: Me, too
Joe_Humbert: Same conflicts for "navigating backwards", so we can resolve them in the same way
Joe_Humbert: JAWS is unfinished because Hadi hasn't finished his testing. Do we need a third Tester?
Matt_King: We can give it some more time
Joe_Humbert: Hadi and Louis are assigned
Matt_King: Rather than having you do that right now, I think I'd rather add the "tabs with manual activation" to the test queue, then run the bots, then have you get started on that
Matt_King: In the case of Hadi and Elizabeth, I'm going to give them an opportunity to finish. Hadi usually pulls through given enough time--it's just a matter of lining up with his schedule
Matt_King: I'll add "tabs with manual activation" to the queue and start the bots and then re-assign to you, Joe_Humbert
Matt_King: I'll work on getting to that today
Matt_King: Okay, that's all for today. Thanks to everyone for the great testing!
mmoss: I will be out all of next week, but I will be back after that, and I will complete all the testing which is currently assigned to me by the end of the day