Meeting minutes
Current status
Matt_King: Ready to go to candidate: Checkbox Example (Two State) (candidate advancement blocked by app bug), Still working on first drafts of spinner and grid plans
Isa: Spinner and grid should be ready by next week!
Matt_King: I'll have a short week next week because of vacation.
Meeting schedule
Matt_King: Last week we updated to a bi-weekly schedule with Thursday meetings. That will start in the new year. We will have a meeting next Weds on the 18th and then reconvene in the new year on Jan 8. Next automation subgroup meeting is TBD
New release in the app
Matt_King: There's a lot of changes here. Hopefully some performance gains but still some bugs. Full changelog: https://
howard: Some of the bugs have been fixed manually. Carmen will send email updates.
Matt_King: Let's resolve them now if possible!
howard: For the checkbox finalization problem, it's a UI issue. The status in the queue doesn't include submitted state.
Joe_Humbert: Do I need something different?
howard: No, I can just submit the unsubmitted results. It just looks like a minor user error. Do you want to review yourself, first.
Joe_Humbert: Ok, I will review the test. I see all of the tests as being submitted, however.
… Actually, I was looking at the wrong test. Ok, now I will submit the first test there.
Matt_King: Ok, that seems to resolve the first big issue I reported.
howard: There's an issue that's filed on that.
ChrisCuellar: Maybe we can get rid of the submitted state altogether? That might resolve the UI issue.
Matt_King: That could work. OK, for the Voiceover issue. There's conflicts reported but no visible conflicts.
howard: This just looks like a data integrity issue. If you just resubmit the tests that should clear up the conflict reporting. This is just a residual problem.
Matt_King: Ok, we'll try resubmitting to clear this up. I'll try that after the meeting.
… Otherwise, there's a lot of awesome things in this release!
howard: The performance enhancements are probably the biggest thing. There's also various other enhancements, but hopefully the performance fixes will help everyone in this group.
Running tests for Switch Example Using HTML Checkbox Input
Matt_King: We just have two things left here. We're waiting on Murray's results for JAWS. For NVDA, Joe needs to change untestable assertions to failing.
Joe_Humbert: I changed that, but there are still conflicts being reported.
Joe_Humbert: Actually, I only changed the untestables to failing in Voiceover. I still need to do this for NVDA actually.
Matt_King: Ok, that resolves that, so this test plan will also be done soon.
Joe_Humbert: Do I need to remove the negative side effects actually?
Matt_King: Just mark the ones in question as "No Output"
Dean: I agree with Joe that the problem is that there is no response at all, not just missing output.
Matt_King: We need to explicitly say that the state of the switch did not change, then.
Joe_Humbert: I'll double-check this later.
jscholes: I think the issue is that "ENTER" is not supported.
Matt_King: This might be a bug in the APG example actually.
… This might be a bad test. Actually, we might need to remove ENTER from the test plan.
jscholes: APG states that ENTER is optional.
Isa: That's why we included it.
Matt_King: Ok, let's remove that from the test plan, though that's a really confusing case.
jscholes: I don't like the inconsistencies here in the way that NVDA speaks the switch state.
Matt_King: Let's change the test plan to remove ENTER. Yes, it's inconsistent but it's reflecting inconsistency in the APG.
jscholes: This should only be removed in focus mode, to be clear.
Matt_King: Thank you, Joe and Dean for helping us catch this using the negative side effect reporting!
Running test plan for Tabs with Automatic Activation
Matt_King: We'll check in with Hadi but skip for now.
Auto-update report reviews
Matt_King: Here's we need a lot more help from testers. You can view those test plans in the "Automated Update" tab.
Joe_Humbert: I complete all the ones assigned to me, I believe.
… I hope to complete all the JAWS test plans by next week.
… I finished Toggle Button and Modal Dialog for VO, to be clear.
Matt_King: I might review the Toggle Button for VO results as well. There may be some "Untestables" in there.
Joe_Humbert: I was seeing some non-reproducible characters after the content. I was seeing these in JAWS. Some of that content should still be there.
Matt_King: This seems like a JAWS bug in the AT-driver implementation.
ChrisCuellar: This should have been addressed but I will try and find the email chains where we discussed this with Vispero.
… Let's leave that output in there for future bug-reporting.
Matt_King: Joe, how does that JAWS output look generally in comparison to historical test plan runs?
Joe_Humbert: It mostly looks ok, but there were some edge cases where the virtual cursor seemed to be off that took a lot of time. As if the bot run was failing.
… In those cases, I did have to manually re-run the tests. It took a lot longer than just verification of the bot output.
… In those re-run cases, everything was marked as "No output".
Matt_King: It seems like the bot should be able to detect that in itself.
ChrisCuellar: That would be cool, but it might be hard for the bot to actually do anything about the problem other than abort the run.
Matt_King: In the meantime, if we see something like that in the near future, let's just delete the run and start it over.