Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: Requests for changes to agenda?
Matt_King: Next CG meeting: Wednesday July 30
Matt_King: Next AT Driver Subgroup meeting: Monday August 11
Current status
Matt_King: we have accordion in the words and three other plans in "draft review"
IsaDC: I opened a pull request for the vertical temperature slider
Matt_King: We're going to be ready next week, then--for people to take a look
IsaDC: We need to mark it as draft
IsaDC: ...just as soon as it appears in the app
Matt_King: I meant to have the "radio" test plan ready for today, but it's not. Hopefully it will be next week
Matt_King: IsaDC and I have been working on tabs
IsaDC: There are two more tests for me to modify, then it will be ready for your review, Matt_King. Expect that by tomorrow
Re-run JAWS report for color viewer slider
Matt_King: We still have two conflicts
IsaDC: I was working on getting those conflicts resolved
Matt_King: There were originally conflicts related to Hadi's results, and those got resolved
IsaDC: I think I will retest for the remaining conflicts because there was a conflict between Windows 10 and 11, so I need to re-run those two with the latest JAWS version
IsaDC: I can do that after this meeting
Matt_King: Looking at the conflicts that are in here--I don't know if this was related to Windows 10 versus Windows 11.
Matt_King: Joe_Humbert was getting the min/max announcement while Hadi was not. It was just for one command
IsaDC: Yes, we retested this during a meeting, and it was flaky
louis: Weren't we going to wait for the July JAWS update before we re-tested some commands?
Matt_King: I think that was for disclosure, but that makes sense here, too
IsaDC: I could try with the JAWS bot because the JAWS bot is available on the staging server
Matt_King: I think we should proceed by editing what we have here instead of starting over
Matt_King: Hadi was using the May update, and Joe_Humbert was using the March update. Neither one of these was with the July release!
Matt_King: I think IsaDC should edit Hadi's results using the July build
Joe_Humbert: I am currently using the version that Hadi was testing on
Matt_King: can you update to the July build, instead?
Joe_Humbert: I can try!
Matt_King: That would be really good--if you could re-do that one test and use the July update
Joe_Humbert: It looks like the latest public release is from July, so yes, I can run the July build
Joe_Humbert: A couple weeks ago, I posted a question: would it be better if I got an actual PC to do testing (so that I'm not using Windows in a virtual machine that is running on old macOS architecture)
Matt_King: That shouldn't make a difference
Run of accordion test plan
Matt_King: there are three issues--2 related to NVDA conflicts and 1 related to JAWS
louis: We can defer discussion on the JAWS conflicts because Joe_Humbert is going to update to the new release and re-test
Joe_Humbert: I'll make sure I'm using the latest JAWS and the latest Chrome. I'm using Windows 10, though
louis: And I'm using Windows 11
Joe_Humbert: Hopefully that doesn't impact our results, but it could
Issue #1270: "Navigate backwards to an expanded accordion header" (Accordion, Test 2
github: w3c/
louis: I got strange output where it says "heading" but it doesn't say anything about the heading level
Matt_King: I was trying this myself earlier, and I think I was not getting the same result as you...
Matt_King: Joe_Humbert, your output is almost perfect--it's exactly what NVDA should do. We know for sure you're using the correct version of NVDA
Matt_King: For the first time, I took James' suggestion and tested with an NVDA portable copy. The only thing I don't like about that approach is that it takes over a minute to start
Joe_Humbert: I've been using a portable copy, and I always reset the defaults when I'm testing
James: The recommendation was that people make a portable copy that they don't use so that they don't need to reset it
Joe_Humbert: Well, I just download NVDA and use it directly, immediately
James: Yes, I think the recommendation is more useful for people who are using the screen reader under test in their daily lives
Matt_King: I think we're aligned, then--everyone is testing the way they need to be testing. But the two testers are still getting quite different results
Matt_King: Joe_Humbert, is there a chance of transcription error, since you aren't using the speech history?
Joe_Humbert: No. I am very careful and go back and double-check
James: Louis reports that the screen reader says "billing address" twice, but that phrase doesn't appear in the reported output
louis: That's a typo in the error description. It's actually a different piece of information which is duplicated
James: Heading levels never spoken by NVDA when it's in focus mode
James: The fact that the heading level appears in the output reported by Joe_Humbert suggests that NVDA was in browse mode at the time--perhaps it switched to browse mode
James: When I switch to browse mode, it includes a "boop" sound at the end to suggest that it is in browse mode
James: I think the issue is with our instruction to testers to begin by ensuring they are in focus mode. If you switch to focus mode yourself and then press "shift+tab", then it will not switch
James: [demonstrates the behavior of NVDA across different contexts]
Matt_King: This is an interesting thing. Because this is a navigation test, the way it actually switches is what we expect
James: When we're testing specifically, "navigate backwards after switching to focus mode"
James: And there is no way to check your current mode in NVDA. When instructed to verify the mode, testers much toggle back and forth, and that changes NVDA's subsequent behavior
Matt_King: So if you follow our instructions to a "T" in this case, you get the wrong results
Matt_King: But if you test for focus mode in a way that doesn't interfere with the test...
Matt_King: We say "navigate backwards in focus mode". That is the test
Matt_King: Do we have a "mode switch" assertion?
IsaDC: No, because this is a switch in reverse
Matt_King: Ah, that's right, we intentionally don't test for cases like this
Matt_King: Loius, the way Joe_Humbert did it is correct. I think we need to change your results
Matt_King: I don't know if we're going to do something to our testing instructions as a result of this discussion
James: We could make an add-on for NVDA that reports the current mode
jugglinmike: This also impacts automation. In both JAWS and NVDA, we likewise don't have a programmatic way to detect the current mode, so we toggle mode once, and potentially toggle a second time
Matt_King: This may indeed be a relevant problem for automation. Let's add it to the agenda of the next "automation subgroup" meeting
TPAC
Matt_King: We're short on time, so we'll talk about this next week
Output normalization
github: w3c/
jugglinmike we're commiting to automation in the recent weeks and we have expanded it's power and extent in which we are relying on it
It has surfaced some new issues that we need to walk through. One of them is normalization
mattl Do we store a normalized state or do we normalize every time we want to compare ?
And never store the normalized form
jugglinmike we never store the normalized form
I do make the distinction in my issue. We have to discuss when we normalize and what we normalize
The two time we consider normalization is when we display the output and when you compare the output
Maybe we can save the concept of saving as an implementation detail
Folks will interact with this when they are reading it and when they are using it for comparaison
Is it easier to talk about what normalization should happen and then when. I thought they are linked
mattl A part of me thinks we should never display normalized outputs
@james We don't want to see (didnt catch it)
mattl I thought the bot wasn't doing what is was supposed to do
@james Mike has a list on his issue which is asserted that there are discrepency that don't have any bering on the output for the end user. I am not sure I agree. Punction is very important for fluctuation
mattl I agree, specially on VO
@james Unicode group separator would never make it to the speach output so that is quite different to punctuation which does make it to the user
mattl I agree with what you said. The unicode separators fall in their own class
I don't think they show up in screen history. We only got it from the the JAWS bot
jugglinmike that is right, only with JAWS bot
There are some other one too. It's good to know that the punctuation can have bearing. I should ask: How easy is it for you as a tester to consistently report that information. Do you know if there should be a dash betwen ctrl and alt when you listen to it
@james I guess it doesn't matter because that is why we are copy pasting
mattl VO captures all punctuations
jugglinmike I like that but I have to say the two reluctantcies I have is that we are commiting to those tools being necessary part of the manual testing
mattl I guess sighted testeers will se them
@james I think the daily AT users have an intuition for the punctuation but there is a subtle difference between a period and a colon or a semi colon and a coma
From that point of view the answer is that a human intuition about punctuation is much less accurate
mattl So what you are saying is that we def want to capture the speech output faithfully. We want the actual string coming from the AT
jugglinmike That represents is a difference or the way we have talked historically. Maybe we would need to do some documentation. But it is helpful to know our expectations from human testers and the machine
@james you mention empty spaces. IF peole use the speech histry for NVDA they will have two space character between chunks of outputs where chunks could be undefined. But usually means if it says Heading Level 3 there will be two spaces. And so those in the speech by NVDA are separate. When it is rended to text it put two spaces so that you have a
better chance to determine where those gaps were
jugglinmike In terms of the algorithm for normalization we might want to normalize more than two spaces. it might be good to normalize it because we wouldn't want to invalid our algorithm.
@james If a string on a page ended with a space, the speech output will include a space and not say a space
There are some differences
mattl this is why the spaces conversation was important. In JAWS new lines helps understand what was part of the output or the screen that is read or what JAWS is telling you about the test in the screen. So this is where maintaining the original output can be helpful
But when it comes to comaring historicaly because NVDA and JAWS and VO they might make changes that doesn't affect what is spoken but do affect the output. Maybe they use a line instead of two spaces. So I think that when we are doing comparaisons that we would always compare normalized values so any double white space becomes 1 white space and
always the same regardless of the new line
@james I don't think it will contain two outpts. That is an implementation of the speech details. What gets sent to the synthetizer is a series of commands. It s a whole thing
I think I agree that what humans are seeing should reflect what was gathered and internally when you compare that is an implementation output and a lot more normalization is appropriate
Carmen Next step is to outline normalization in the issue and reach out again
We can also send an email to everyone in the CG