Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: Any requests for changes to the agenda?
Matt_King: Hearing none, we'll stick with the agenda as planned
Matt_King: Next CG meeting: Thursday September 4
Matt_King: Next AT Driver Subgroup meeting: Monday September 8
Current status
Matt_King: We now have seven plans in draft review
Matt_King: That's the same as last week, but it's different plans
Matt_King: Vertical temperature slider advanced to "candidate review"
Matt_King: That's an awesome accomplishment
Matt_King: So we have five active test plans. "Switch with HTML button" is a brand-new one that was just added this week, along with another disclosure plan
Matt_King: Coming at the end of this meeting, if everything goes well, are two more test plans
Matt_King: IsaDC is working on both of those right now
Matt_King: By the time of the next meeting, we could have nine things to work on in the queue. That's a bunch of work and sets us up well for James' and IsaDC's upcoming time off
IsaDC: Matt_King, the switch with the checkbox is ready for your review
Matt_King: Got it
Running accordion test plan
Matt_King: Last week, we had 42 conflicting results. Now, we're down to 21 conflicting results
Matt_King: We have conflicts in test 2. Are they all in test 2?
Dean: Yes. I resolved a bunch of them yesterday. Some of it was about not clearing out the result when marking stuff as "untestable"
Dean: It works now, and that fixed a bunch of them
Dean: The other one was a user error--I didn't set my keyboard back to "use function keys as function keys"
Dean: The only thing we have left is that stuff where it jumps to a form field, you press "b", and it enters the "b" value instead of running a command
Dean: I still need to investigate that further to understand what's going on there
Dean: I sent mmoss a note about this very recently
mmoss: I'm not observing that behavior
Matt_King: That's interesting.
Matt_King: You are each on different versions--18.5 and 18.6. That could possibly explain the incongruence
Matt_King: It's also possible that you've configured your browsers differently
Dean: I think it's a forms-mode-versus-browse-mode problem
Matt_King: Or it could be browser version differences
mmoss: And I don't have any "auto-fill web form" settings checked in Safari
Dean: I use that all the time. I don't remember turning it on, but I bet that's it
Dean: I'll update everything and go back and look for that setting
James: I think the problem is that mmoss has never saved any data, so it's not going to offer mmoss the option to fill out any data
IsaDC: What did the bot report?
James: I will note (for completeness) that while we want people to test with defaults, we never actually tell people to set their browser to default settings
James: You can't easily do this in incognito because you have to be logged in
Dean: When it comes to something like autofill, can't we assume that most people will be saving their data?
Matt_King: Yeah, this is a can of worms I was hoping not to get in to
Dean: I'll try setting it back, seeing if I still get the same response. If I don't, then we can report a bug to Apple saying that "this doesn't work if you have autofill on"
Dean: I've got another Mac that is totally wiped that I can try this with. I'll update everything on it
Matt_King: Sounds like we have next steps defined for this one. Cool!
mmoss: We had one test case where the reading cursor doesn't move to the expected element, but it remains on the existing element (which does have the same role as the element that it's moving to). I passed the assertion about the role.
Matt_King: That should be marked as "untestable"
mmoss: Ah, yes. That was the one that I missed.
Dean: and that was the only one that I wanted you to look at
mmoss: Great catch--thank you
Dean: I may be able to get this one closed out today. I'll e-mail you with whatever I find
IsaDC: It will be amazing if we can mark this as final
Matt_King: Now we're done to just VoiceOver conflicts in one single test
Running test plan for Tabs with Automatic Activation
Matt_King: we have conflicts in JAWS and NVDA
Matt_King: None of the results are complete, yet. But I figure it's still worth reviewing to get a sense of what kind of trouble we're running in to
Matt_King: In JAWS, we have conflicts in tests 5, 7, 8, 9, and 10
IsaDC: I will clear Louis' conflicts
Matt_King: This is the "Selected one of four selected" thing
IsaDC: We agreed to mark this as overly verbose
James: If we had Hadi's agreement last week, then is it appropriate to do that for him?
Matt_King: I think so, yes
Matt_King: Maybe the most appropriate thing at this point is to just e-mail Hadi
Matt_King: As for NVDA...
IsaDC: I can see six conflicts
Matt_King: Test 1, 2, and 12
Matt_King: In test 1, Joe_Humbert and Louis are getting the same output
mmoss: We are getting the same output. Louis marked the "not selected" assertion as "passing," and I marked it as "failing"
Matt_King: Maybe this is a mistake in the test plan
Matt_King: In the "radio button" one, they do the same thing. They don't convey the "not selected" state.
Matt_King: We got alignment with both Apple and Vispero on this: in certain situations, it's required to convey the "selected" state, but it's completely optional to convey the "not selected" state
James: I don't think that applies across-the-board on Windows
Joe_Humbert: I failed it because it's not conveyed
Matt_King: I think you're right
IsaDC: We should just change Louis's result
James: for the second conflict, I think Louis interpreted the absence of "selected" to mean "not selected"
Matt_King: That's practically what it means, but it's not what we're specifically looking for here
Joe_Humbert: For test 12, I passed the assertion that the "selected" state is conveyed, but Louis failed it. Our output is the same
James: That one, I have no explanation for
IsaDC: I'll correct his results to match yours
Joe_Humbert: Unfortunately, we have different results on test 13
Joe_Humbert: I get the name of the tab panel (I think) and that it's a "property page" which is weird. But it also reads out the contents of the tab panel for me, but it didn't do that for Louis
James: If it reads out the content, it may be switching to browse mode
Matt_King: It probably will switch to browse mode if it was in focus mode
James: If a user has explicitly switch modes, then the auto-switching behavior is different
Matt_King: As a Tester, you want to make sure that you manually don't change it so that it does do the automatic switch
Matt_King: This is a little bit of an interesting twist in our test. We say that, to do it in focus mode, I always thought that means "the test starts in focus mode" not that "you force it to remain in focus mode after the command is pressed"
Matt_King: We should let the AT do its automatic switching
James: The thing with NVDA is that there is no way to determine that you are starting in focus mode without toggling it
James: We instruct testers to "ensure" that they are in focus mode--not just to "observe"
James: If there was a keystroke that they could press to just query the mode and report that (without toggling it), then you would be achieving the goal of "ensuring" you are in the correct mode
Dean: I sent out an e-mail asking at what point you are switching modes when you start a test. This is the problem I had
Matt_King: I wonder if we should change those instructions a little bit. Ideally, "assuring" would actually be "listening to what happens when you press setup"
Joe_Humbert: I just re-tested, taking care not to switch modes, and it does read out the contents of the tab panel
Joe_Humbert: If I force it out of focus mode and back in, it does not read the contents of the tab panel
IsaDC: So do we always trigger it?
James: We need to think about our approach to how we instruct people
Joe_Humbert: There are some tests where it's in browse mode, and you have to manually toggle it into focus mode. Like where you're manually getting the information about something
James: That's fine. In NVDA, that should be consistent regardless
Joe_Humbert: So if it's asking for focus mode, you shouldn't be toggling it on manually
James: But unless you do the toggling, you don't know for certain
James: If you don't verify something yourself, you don't know it to be true
Matt_King: We have a little bit of the problem in the instructions right now because we say that setup is automatically running the script--it doesn't
Matt_King: Right now, we have three steps when we should have four
James: Would if help, given that most Testers have the "speech history" add-on, to also have an add-on to report the current mode
Joe_Humbert: I will say that I don't test with that tool; I just test with straight NVDA
Matt_King: I don't know if we want to make it a requirement to have an add-on. We could, but...
Matt_King: It's not an add-on to change how NVDA behaves but just an add-on to assist in testing. I think it's legit
IsaDC: For this test case, what do we do to resolve the conflict?
Matt_King: I think we should report it in the way that Louis did. We should not be manually toggling out and back in
James: Right, because realistically, we cannot really test this behavior of NVDA which may or may not be intentionally. Realistically, most of the time, people are going to be experiencing the mode-switching behavior that is the default rather than what we're seeing here
Matt_King: Okay, so we have the next step here. Joe_Humbert should modify his results
IsaDC: It looks like we have to copy Joe_Humbert's results to match Louis'
Matt_King: Oh, okay, we're not changing Joe_Humbert's results. We're changing Louis's results
IsaDC: Right
Matt_King: Okay, good good. Then we know the answer to all of these (basically: modify Louis' results)
Running Disclosure of Answers to Frequently Asked Questions test plan
Matt_King: Louis raised issue #1282
Matt_King: That could be a problem with the JAWS Bot
Matt_King: We should manually verify whether or not that happens
Matt_King: There are five conflicts in the JAWS results
Matt_King: Ah, and NVDA is done, and there are no conflicts
IsaDC: There's an issue with the NVDA run
IsaDC: Joe_Humbert says it should say to press the "up arrow" key three times instead of two times
Matt_King: And this is related to NVDA?
Joe_Humbert: I believe so
IsaDC: It's about "navigate backwards from here". With NVDA, we have it marked as two presses of the "up arrow" key
Joe_Humbert: There are no conflicts because apparently Louis and I got the same output. But pressing the "up arrow" twice gets you to the expanded list item, but it doesn't actually get back to the button (which is what the test is trying to do)
Joe_Humbert: We both failed the assertions for "name of button", "role of button" and "expanded"
Joe_Humbert: There are no conflicts because we both got the same output and both failed the assertions
Joe_Humbert: I wondered if there is an extra arrow in there, if we wouldn't fail the assertions
Joe_Humbert: I assumed that the assertions from all the other ones is going back to the button. That's what it's attempting to do. But because this has the list boundary
IsaDC: Ah, yes, so we have this wrong. We need three presses of the "up arrow" key
IsaDC: We should change this
Joe_Humbert: I only did this because I re-tested the example, trying with the third up-arrow press, and it got to the button
Matt_King: I wonder if we should even bother with the "up arrow" command at all for this. I'm almost more tempted to remove it from all the screen readers
Matt_King: We have that text in between
Matt_King: If the link was right after the button for this test, that would be better, because that would only require one press of the "up arrow" key
Joe_Humbert: Wouldn't that break the structure of the other tests if there's a button in the middle of the disclosure?
Matt_King: Ah, so we have the "navigate forward" and "navigate backwards" in the same place for all the tests
Matt_King: I was thinking we could change it
James: The setup script can do anything. It can add controls to the page. I don't think we should go down that route, though
Matt_King: I feel like this is one of these weird situations where if they change something else in the screen reader (like the line length), then we get into this situation. The fact that there are three key presses required is not very deterministic
Matt_King: I think it's better to remove the "up arrow" command in this case
Joe_Humbert: IsaDC, if you want to assign me to a few of the test plans which have bot runs, you can do that. Any ones you want
IsaDC: Thank you!
Dean: And give me anything other than JAWS