18:51:33 RRSAgent has joined #aria-at 18:51:37 logging to https://www.w3.org/2023/07/17-aria-at-irc 18:51:56 rrsagent, make log public 18:52:04 Zakim, start the meeting 18:52:04 RRSAgent, make logs Public 18:52:06 please title this meeting ("meeting: ..."), jugglinmike 18:52:26 meeting: Bi-Weekly Meeting of Assistive Technology Automation Subgroup of ARIA-AT Community Group 18:52:31 present+ jugglinmike 19:01:45 Sam_Shaw has joined #aria-at 19:05:46 scribe+ 19:06:27 present+ James Scholes 19:06:38 present+ Michael Fairchild 19:07:23 present+ James_Scholes 19:08:05 Matt_King has joined #aria-at 19:08:10 Agenda: 1. The latest iteration of the automation design document. 2. Update from Vispero 19:08:47 present+ Michael_Fairchild 19:08:56 present+ 19:09:11 TOPIC: BTT Charter 19:09:42 MK: Vispero has verbally said they will have someone nominated to BTT Browser Test and Tools working group 19:10:15 MK: The charter is typically 3 years, Vispero will join and show intent to implement the API 19:10:45 MK: They will plan to implement the API at some point within the 3 year charter 19:11:06 MK: We needed two implementers to join, PAC can join as an implementer or the NVDA Add on 19:11:44 JS: When you say the charter is three years, does that mean that joining this working group that PAC will support the driver of this standard for a three year period? 19:12:36 MK: No. Our goal is for NVAccess to become the supporter over time. The purpose of the implementors is to demonstrate the implementability 19:13:29 MK: The main purpose for signing up, is IP. When you agree to implement a specification, you are basically saying that all knowledge you contribute to developing this specification, is licensed under the W3C license 19:14:12 MK: Mike what do you recommend we do as a next step with BTT? 19:15:11 jugglinmike: I will contact Lola about this, and she will read this update in these notes. She is leading the BTT communication, we have started communication with David Burns about this proposal to get the driver into the spec 19:15:50 jugglinmike: This commitment from Vispero will be helpful, in our communication so far with BTT they said this would be needed. 19:16:12 MK: Steve Faulkner is the Vispero BTT rep, do you have his contact info? 19:16:30 MK: I can write an email to you and steve this afternoon to connect you. 19:16:37 jugglinmike: Yes please address it to Lola and I 19:17:10 MK: I will start an email thread, I will copy Glen on this aswell 19:17:35 MK: We should do something similar for PAC, is Sina the rep? 19:17:42 JS: I assume so 19:17:55 MK: Sina might be a invited expert 19:18:18 JS: I'm not sure PAC is registered in W3C 19:19:05 MK: That would mean a little different process to get PAC set up as a invited expert. Mike can you take the lead on starting that process with an email to Sina? 19:19:11 jugglinmike: Yes I will do that 19:19:31 MK: We need to resolve this in the next few weeks to keep with their timeline for the new charter 19:20:08 TOPIC: The latest iteration of the automation design document 19:20:13 https://docs.google.com/document/d/1bjXZ21gGFH_rdP6IGcgSohbSNikpPqntV_wX-5nhUx8/edit 19:21:05 jugglinmike: This is a new version of the document, I've tried to keep the old version accessible to everyone in case we need to reference it 19:21:32 jugglinmike: These are the changes I've made: change to central use-case previously: "collect responses to limit human responsibility to rendering verdicts" in this iteration: "collect responses so that humans only run tests which may have novel results" new section: "Project Goal" new terminology: "historic test results" and "verdict" improved document structure 19:22:45 jugglinmike: Overall, throughout the doc I changed the central use case. Previously this was to collect responses to limit human responsibility and assign verdicts. Now the central use case is to collect responses so that humans only run tests that have novel responses 19:23:04 jugglinmike: The AT will be able to run tests and assign verdicts 19:23:20 jugglinmike: New Project Goal: ARIA-AT uses automated systems to ensure that humans only run tests when the test environment and/or the implementation behavior have not previously been reviewed. 19:23:32 jugglinmike: Does that make sense to folks? 19:24:11 MK: does the test environment include all versions of the AT? So we can retest old version against new releases? 19:24:53 jugglinmike: I think so. Test environment is made up term, but what I think that includes is the widget code itself, assertions and commands, and the browser and versions, and at and versions 19:25:09 jugglinmike: Is that correct? Is there a better way to concisely define that? 19:25:39 MK: If we state that the browser and AT are included, but don't include version numbers, then yes 19:25:46 michael_fairchild has joined #aria-at 19:26:04 MK: You may want to call that a test scenario, or test profile instead of environment 19:26:17 MK: Thats something we might want a term for. 19:26:44 MK: We will want to talk about all these tests, minus the versions 19:27:02 jugglinmike: I agree, I've made an action item to modify this 19:27:37 jugglinmike: The new terminology is the other thing I want to mention. In the doc now, each term is a new heading 19:27:59 jugglinmike: I added a definition of the term verdict 19:28:17 jugglinmike: Verdict: a subjective judgment as to whether a given AT response satisfies a given test assertion; verdicts are typically rendered by humans, but ARIA-AT may copy 19:28:46 jugglinmike: verdicts from historic test results to a new Test Plan Run if there is evidence that the test and implementation have not changed 19:28:52 MK: That sounds good to me 19:28:58 michael_fairchild: I agree 19:30:11 jugglinmike: For collection Job status, there is a new value. We've added a new value called skipped, to reflect that something changed in that test so we decided to skip it. It wont run and it will never run. Even if we collected responses there is nothing we would use it for. 19:30:22 MK: Beautiful 19:31:16 jugglinmike: Another new term Historic Test Results: A test has "historic" results for a given browser and AT if that same test has previously been run in that browser and AT at some point in the past and if those results have been included in a Candidate or Recommended Test Plan; any modification to the test (e.g. via a change to the application code or a change to an assertion) disqualifies the previously-reported results from consideratio[CUT] 19:33:13 MK: It would be nice if we make a change to a test, that we don't have to rerun the entire test. I guess now we have a way to do that by just coping the test. I'm talking myself out of a change right now. 19:34:05 jugglinmike: Yes if you change an assertion, you can still use the same test plan. Wouldn't the test case you are considering if you had a new version of the test in draft review and you wanted to test a new version of a browser? 19:39:29 MK: I think if we want to change test plan, we have three ways now to create a new test. We can go grab the results from a previous test, we can also have Humans make a new set of results, and now we can have AT go make a new set of results 19:39:54 jugglinmike: Thats use case is only worthwhile is there is a new version to test right? 19:40:02 MK: Yes 19:41:47 MK: If you have a plan thats in draft, and we haven't worked on it for a while but that is complete but not promoted, and we want to update the test results to see whats changed. Now that the data is out of date, we want to re run this 19:42:21 jugglinmike: We need a different term than "complete" as this currently means its been promoted 19:42:32 MK: Okay hmm 19:42:53 MK: We use approved in a couple other places so we can't use that 19:43:24 MK: In the test queue, when two tests have completed all tests, an admin then finalizes this, can we use the term finalized? 19:44:00 jugglinmike: so the use case is, we have two testers who have finished all tests but the test admin hasn't moved it forward to candidate, why would this be the case? 19:44:43 MK: We might have gone through that process to see how the AT fairs. We may not want to move it forward for political reasons 19:45:21 michael_fairchild: Its less that its finalized, but more that its not ready 19:45:56 MK: We could finalize results for a test plan thats not ready for candidate review. We may have gotten feedback that we need to make a change to an assertion 19:46:15 MK: When we say results, are we saying verdicts plus output? 19:46:24 MK: Yes 19:47:07 michael_fairchild: That term isn't in the doc you created Mike, should we add it or have it somewhere else? 19:47:34 jugglinmike: correct we don't explicitly define what the results are, but its implied by being results. 19:47:57 MK: Lets make sure we don't leave anything implied. 19:49:37 jugglinmike: This is expanding the scope a little more, so I'm hesitant to make it any bigger. I mentioned this to Howard, but as far as I understand, the test plan run results are somewhat redundant after the test plan enters the candidate phase because of this requirement that they are equivalent. I don't know if this is making more work, or if there is something here in which we can deal with this redundancy. Can we promoted one set of re[CUT] 19:50:09 MK: I was wondering about that, I think its valuable to maintains two sets of data that were collected at different times. 19:50:27 MK: However, when we mark them as finalized, they are equal. 19:51:37 MK: We can have equivalent results that are created with two different version of browsers, we don't require they are the same versions. We can still separate the verdicts from the testing profile : Date, time, user, version info 19:52:32 jugglinmike: I can take this back to the folks at Bocoup, it will expand the UI so Issac will want to be aware of this to know how an admin would mark a test plan result as finalized, Howard will want to know how to encode that in the DB. 19:52:39 jugglinmike: I will update the design doc aswell 19:52:58 MK: The functionality is all there right now, its just that button does a little more than it should 19:53:32 MK: Right now the button when theres no conflicts is labeled "Mark as candidate", we need to change that to be named "Mark as final" 19:53:44 MK: I did write this in issue 648 19:54:56 MK: If that plan, depending on the phase of the working mode, in which that test plan version exists. If you have a candidate test plan, and you mark them as final, then the results are published on the reports page. If its a draft plan, then the results don't get published, the test plan just gets removed from the test queue 19:55:21 MK: This is all documented in issue 648 19:56:14 MK: We will need this button fixed fairly soon 19:57:25 JS: I would encourage you Matt to review the feedback and question from Issac. I think its difficult to keep track of who has said what and what has been decided. Issac had some questions, I tried to answer, but at the moment you are the person with the most complete version of the UI we want 19:57:35 MK: When were his questions? 19:57:44 JS: Last week, I mentioned you in my response 19:58:07 MK: I will look for them, it probably won't be before Wednesday Mike, can you give Issac a heads up? 19:58:10 jugglinmike: Will do 19:58:30 jugglinmike: Great, we are almost out of time, I have plenty to work on so lets call it 19:58:40 jugglinmike: Matt did your son pass his test? 19:59:03 MK: Yes! I've been flying with my son twice now. Saturday we are going to take a flight to my hometown in central washington 19:59:20 MK: Pretty cool stuff, I love flying and I love flying with my own personal pilot! 20:03:06 Zakim, end the meeting 20:03:06 As of this point the attendees have been jugglinmike, James, Scholes, Michael, Fairchild, James_Scholes, Michael_Fairchild, Matt_King 20:03:08 RRSAgent, please draft minutes 20:03:09 I have made the request to generate https://www.w3.org/2023/07/17-aria-at-minutes.html Zakim 20:03:16 I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent. Goodbye 20:03:16 Zakim has left #aria-at