Meeting minutes
<Sam_Shaw> Agenda: 1. The latest iteration of the automation design document. 2. Update from Vispero
BTT Charter
MK: Vispero has verbally said they will have someone nominated to BTT Browser Test and Tools working group
MK: The charter is typically 3 years, Vispero will join and show intent to implement the API
MK: They will plan to implement the API at some point within the 3 year charter
MK: We needed two implementers to join, PAC can join as an implementer or the NVDA Add on
JS: When you say the charter is three years, does that mean that joining this working group that PAC will support the driver of this standard for a three year period?
MK: No. Our goal is for NVAccess to become the supporter over time. The purpose of the implementors is to demonstrate the implementability
MK: The main purpose for signing up, is IP. When you agree to implement a specification, you are basically saying that all knowledge you contribute to developing this specification, is licensed under the W3C license
MK: Mike what do you recommend we do as a next step with BTT?
jugglinmike: I will contact Lola about this, and she will read this update in these notes. She is leading the BTT communication, we have started communication with David Burns about this proposal to get the driver into the spec
jugglinmike: This commitment from Vispero will be helpful, in our communication so far with BTT they said this would be needed.
MK: Steve Faulkner is the Vispero BTT rep, do you have his contact info?
MK: I can write an email to you and steve this afternoon to connect you.
jugglinmike: Yes please address it to Lola and I
MK: I will start an email thread, I will copy Glen on this aswell
MK: We should do something similar for PAC, is Sina the rep?
JS: I assume so
MK: Sina might be a invited expert
JS: I'm not sure PAC is registered in W3C
MK: That would mean a little different process to get PAC set up as a invited expert. Mike can you take the lead on starting that process with an email to Sina?
jugglinmike: Yes I will do that
MK: We need to resolve this in the next few weeks to keep with their timeline for the new charter
The latest iteration of the automation design document
https://
jugglinmike: This is a new version of the document, I've tried to keep the old version accessible to everyone in case we need to reference it
jugglinmike: These are the changes I've made: change to central use-case previously: "collect responses to limit human responsibility to rendering verdicts" in this iteration: "collect responses so that humans only run tests which may have novel results" new section: "Project Goal" new terminology: "historic test results" and "verdict" improved document structure
jugglinmike: Overall, throughout the doc I changed the central use case. Previously this was to collect responses to limit human responsibility and assign verdicts. Now the central use case is to collect responses so that humans only run tests that have novel responses
jugglinmike: The AT will be able to run tests and assign verdicts
jugglinmike: New Project Goal: ARIA-AT uses automated systems to ensure that humans only run tests when the test environment and/or the implementation behavior have not previously been reviewed.
jugglinmike: Does that make sense to folks?
MK: does the test environment include all versions of the AT? So we can retest old version against new releases?
jugglinmike: I think so. Test environment is made up term, but what I think that includes is the widget code itself, assertions and commands, and the browser and versions, and at and versions
jugglinmike: Is that correct? Is there a better way to concisely define that?
MK: If we state that the browser and AT are included, but don't include version numbers, then yes
MK: You may want to call that a test scenario, or test profile instead of environment
MK: Thats something we might want a term for.
MK: We will want to talk about all these tests, minus the versions
jugglinmike: I agree, I've made an action item to modify this
jugglinmike: The new terminology is the other thing I want to mention. In the doc now, each term is a new heading
jugglinmike: I added a definition of the term verdict
jugglinmike: Verdict: a subjective judgment as to whether a given AT response satisfies a given test assertion; verdicts are typically rendered by humans, but ARIA-AT may copy
jugglinmike: verdicts from historic test results to a new Test Plan Run if there is evidence that the test and implementation have not changed
MK: That sounds good to me
michael_fairchild: I agree
jugglinmike: For collection Job status, there is a new value. We've added a new value called skipped, to reflect that something changed in that test so we decided to skip it. It wont run and it will never run. Even if we collected responses there is nothing we would use it for.
MK: Beautiful
jugglinmike: Another new term Historic Test Results: A test has "historic" results for a given browser and AT if that same test has previously been run in that browser and AT at some point in the past and if those results have been included in a Candidate or Recommended Test Plan; any modification to the test (e.g. via a change to the application code or a change to an assertion) disqualifies the previously-reported results from consideratio[CUT]
MK: It would be nice if we make a change to a test, that we don't have to rerun the entire test. I guess now we have a way to do that by just coping the test. I'm talking myself out of a change right now.
jugglinmike: Yes if you change an assertion, you can still use the same test plan. Wouldn't the test case you are considering if you had a new version of the test in draft review and you wanted to test a new version of a browser?
MK: I think if we want to change test plan, we have three ways now to create a new test. We can go grab the results from a previous test, we can also have Humans make a new set of results, and now we can have AT go make a new set of results
jugglinmike: Thats use case is only worthwhile is there is a new version to test right?
MK: Yes
MK: If you have a plan thats in draft, and we haven't worked on it for a while but that is complete but not promoted, and we want to update the test results to see whats changed. Now that the data is out of date, we want to re run this
jugglinmike: We need a different term than "complete" as this currently means its been promoted
MK: Okay hmm
MK: We use approved in a couple other places so we can't use that
MK: In the test queue, when two tests have completed all tests, an admin then finalizes this, can we use the term finalized?
jugglinmike: so the use case is, we have two testers who have finished all tests but the test admin hasn't moved it forward to candidate, why would this be the case?
MK: We might have gone through that process to see how the AT fairs. We may not want to move it forward for political reasons
michael_fairchild: Its less that its finalized, but more that its not ready
MK: We could finalize results for a test plan thats not ready for candidate review. We may have gotten feedback that we need to make a change to an assertion
MK: When we say results, are we saying verdicts plus output?
MK: Yes
michael_fairchild: That term isn't in the doc you created Mike, should we add it or have it somewhere else?
jugglinmike: correct we don't explicitly define what the results are, but its implied by being results.
MK: Lets make sure we don't leave anything implied.
jugglinmike: This is expanding the scope a little more, so I'm hesitant to make it any bigger. I mentioned this to Howard, but as far as I understand, the test plan run results are somewhat redundant after the test plan enters the candidate phase because of this requirement that they are equivalent. I don't know if this is making more work, or if there is something here in which we can deal with this redundancy. Can we promoted one set of re[CUT]
MK: I was wondering about that, I think its valuable to maintains two sets of data that were collected at different times.
MK: However, when we mark them as finalized, they are equal.
MK: We can have equivalent results that are created with two different version of browsers, we don't require they are the same versions. We can still separate the verdicts from the testing profile : Date, time, user, version info
jugglinmike: I can take this back to the folks at Bocoup, it will expand the UI so Issac will want to be aware of this to know how an admin would mark a test plan result as finalized, Howard will want to know how to encode that in the DB.
jugglinmike: I will update the design doc aswell
MK: The functionality is all there right now, its just that button does a little more than it should
MK: Right now the button when theres no conflicts is labeled "Mark as candidate", we need to change that to be named "Mark as final"
MK: I did write this in issue 648
MK: If that plan, depending on the phase of the working mode, in which that test plan version exists. If you have a candidate test plan, and you mark them as final, then the results are published on the reports page. If its a draft plan, then the results don't get published, the test plan just gets removed from the test queue
MK: This is all documented in issue 648
MK: We will need this button fixed fairly soon
JS: I would encourage you Matt to review the feedback and question from Issac. I think its difficult to keep track of who has said what and what has been decided. Issac had some questions, I tried to answer, but at the moment you are the person with the most complete version of the UI we want
MK: When were his questions?
JS: Last week, I mentioned you in my response
MK: I will look for them, it probably won't be before Wednesday Mike, can you give Issac a heads up?
jugglinmike: Will do
jugglinmike: Great, we are almost out of time, I have plenty to work on so lets call it
jugglinmike: Matt did your son pass his test?
MK: Yes! I've been flying with my son twice now. Saturday we are going to take a flight to my hometown in central washington
MK: Pretty cool stuff, I love flying and I love flying with my own personal pilot!