Meeting minutes
Review agenda and next meeting dates
Matt_King: Next meeting will be February 28
Automation subgroup meeting planning
Matt_King: I saw an e-mail about a poll
Joe_Humbert: Yup, it was sent to this group's e-mail list
jugglinmike: The poll is here: https://
jugglinmike: This poll is open until Tuesday, Feb 27th, at 9:00 AM US Pacific Time.
Matt_King: Anyone who's interested in automation, please submit a response there!
Sam_PAC: I submitted a single response to represent PAC--both James_Scholes and myself
Set schedule for toggle and dialog
Toggle button plan: set date to resolve
github: w3c/
Matt_King: Given the consenses we reached last time, this is a one-word change to one test plan
Matt_King: Because the assertion changes, verdicts for any tests which include role will become invalid
IsaDC: All assertions that say "toggle button", I believe
Matt_King: Let's set a schedule for figuring this out
Matt_King: I will re-assign this from me to IsaDC
James_Scholes: We're doing "toggle button" as two words, correct?
IsaDC: Yes
Matt_King: Yes
Matt_King: So that new version of the "toggle button" test plan won't be available until you make the change, merge it, and advance the test from "R&D" to "review"
Matt_King: The only time we have the option to preserve data is if there is an incomplete run in the test queue
Matt_King: It seems like we'd want a way to copy all data for tests which haven't changed whenever we add a new test plan to the test queue. Then again, that seems like something that ought to be done automatically
James_Scholes: I guess it depends on the complexity of the changes between the two test plan versions
Matt_King: I don't know where that fits in with our roadmap
Matt_King: For now, though, for anybody assigned to NVDA with Chrome, we do have the bot available
Matt_King: And as far as I can tell, it's 100% reliable. I have not found a case where the NVDA bot gives me different output, and I've done this a number of times for three different test plans
jugglinmike: I can't say when Firefox support will be available, but I can see about getting an estimate for next week
IsaDC: Would anyone like to re-run all of "toggle button" from scratch?
Joe_Humbert: I can re-run the ones that I originally submitted--VoiceOver and Safari
Matt_King: A two-week timeframe would be great. That's because my next meeting with Vispero is three weeks from yesterday
Hadi: I volunteer to run in Chrome and JAWS again
[Matt_King reiterates the rationale for re-running the "toggle button" tests]
howard-e: Copying data in that way is currently possible, actually, at least for the Test Plan Reports which have not been marked "final"
Michael_Fairchild: in order to save time, could we ask the database administrators to manually copy the test results?
howard-e: That's feasible, but I'm also saying that there's a high chance that the data for the unchanged tests will be copied automatically
Matt_King: It sounds like the plan--first we get the new version. IsaDC does that.
Matt_King: That gets moved to Draft Review
Matt_King: Actually, howard-e, before she does that, she would deploy the latest changes in staging (following my completing my functional testing)?
Matt_King: If we deployed that, and then IsaDC does her part, then we would see some results carry forward automatically
Matt_King: After that, should we have howard-e try to manually copy some of Joe_Humbert's test results forward?
James_Scholes: I think we should. It would take multiple hours to re-run this test
James_Scholes: Should we add the new version to the queue now? Or should we do that later?
[Matt_King investigates the current behavior of the application]
Matt_King: IsaDC can make the change, and she can merge it, but do not advance it to draft review until howard-e has updated the app!
Matt_King: And howard-e will not update the app until Monday
IsaDC: Got it
James_Scholes: Should I un-assign you from JAWS and NVDA, Matt_King?
Matt_King: Sure
James_Scholes: Got it (I asked to un-assign you because we already have results from IsaDC and Hadi)
Matt_King: There are some problems with this feature, and I've raised an issue about them
Matt_King: It's related to the "trend reporting" project that Bocoup is going to be working on next quarter
James_Scholes: the status says "draft", and that means there are no conflicts (because if there were conflicts, the status would describe them)
Matt_King: The order of events:
Matt_King: 1 - IsaDC is going to merge a new version of the test plan
Matt_King: 2 - I am going to test what's in staging and give the green light to howard-e
Matt_King: 3 - howard-e will release the changes to production
Matt_King: 4 - IsaDC can advance the new version from R&D to Draft Review
James_Scholes: after the new version is advanced, it doesn't automatically appear in the test queue, does it?
Matt_King: Yes, it does
Matt_King: The status of the new (February 23rd) version, will show "5 of 8 complete" (or something similar)
James_Scholes: that all sounds good. It does leave a feature result outstanding, though: to copy results from a test plan with a different status
Matt_King: Yes, that's right
Matt_King: We have an issue open for that, and it's related to the "trend reporting" work
Schedule for refactor and review of dialog test plan
Matt_King: Would it be reasonable for the meeting two weeks from today that we have people sign up for the testing of "dialog"?
IsaDC: I would like to take a look and then discuss with you next Tuesday
Matt_King: That sounds good. We may want several days of reviewing it even before we ask people to run the test
James_Scholes: I would love to see a test plan for the dialog element
Matt_King: I agree, though it's a separate topic
Review proposal for test format V2 enhancement
Matt_King: This is going to get technical, fast
Matt_King: Right now, we have a way of making it so that you can have several assertions assigned to a test (e.g. an assertion for "role" and one for "state"), and then an additional assertion (e.g. for when the AT changed mode)
Matt_King: We can have another assertion on the test, but we only expect that assertion to happen for "tab"
Matt_King: The way we currently specify that, is we put the assertion on the test (the "mode switch assertion") and then place assertion exception on the specific commands where that assertion does not actually apply
Matt_King: The way we specify the assertion is we give it "zero" priority (prefixing it with the characters "0:")
Matt_King: This proposal is to flip that logic on its head
Matt_King: In the test file, we will put the mode switch assertion in the test file, where it will mean, "don't include this for any commands by default"
Matt_King: If you want it on a command, you would then add that assertion identifier for that command only
Matt_King: That means that there are a lot fewer exceptions and that the exceptions are more clear
Matt_King: As a result, you wouldn't see any exceptions at all in VoiceOver (since it doesn't do mode switching)
Hadi: It sounds like we are mixing objective testing and specific screen reader features
James_Scholes: this is very much a test authorship syntax change. As a tester, you will probably never see a difference--it all takes place in the underlying files
James_Scholes: Are you claiming that these two approaches should be mutually exclusive, or should it support both?
Matt_King: I think it should support both
James_Scholes: Good. I think that's important because there are cases where the current syntax is more appropriate
Matt_King: Something else that happens with Radio Groups is that when you navigate backwards with "f" or "r", you always end on the last radio button, but if you navigate backwards with other commands, you land on the *first* item
Matt_King: I've been imagining that this could be another use case for the assertion exceptions
James_Scholes: I think that would be appropriate. I think that it may end up a bit confusing in some test plans. It could turn test plans into a sort of spaghetti
Matt_King: I was just trying to make the test plan simpler for testers
James_Scholes: Absolutely
Matt_King: howard-e I would like to get a date on how fast you could handle this issue--1039
howard-e: I have't been able to triage this, yet, so I'm not prepared to estimate right now
Matt_King: I would like to see this within the next sprint at the latest
Matt_King: Sooner is better
howard-e: Of course
howard-e: I'll share that with Carmen and follow up in the issue