<Jemma__> scribe?

I can scribe

<scribe> scribe: michael_fairchild

documentation and running locally

<Jemma__> present_ rob-fentress

Matt_King: we are at the end of Bocoup's contract
... our plan is to work on a v2 in the second half of the year.
... our goal was to get the runner to a place where the CG can use the app in a minimal capacity

Val: we still need to deploy the reports page, and the tests from ARIA-AT will be imported to the staging and prod servers on a regular basis

<spectranaut> Here is the application: https://github.com/bocoup/aria-at-app

Matt_King: the app itself is still in a bocoup repo. We will work on moving to the w3c org

<spectranaut> https://github.com/bocoup/aria-at-app/blob/main/docs/local-development.md

Val: There is a docs directory, with details on local development and the database

<spectranaut> https://github.com/bocoup/aria-at-app/blob/main/docs/database.md

Val: it's a node react app, using Yarn as a package manager
... to set up and run locally, follow that documentation (details on scripts and yarn commands)
... if there are new tests from ARIA-AT, you will need to manually run a script to get the newest tests. This is in the documentation, and needs to be done for local development

(questions about differences between aria-at-app and aria-at. aria-at-app is the production runner and aria-at is the test repo)

Val: structure of the app: front end is under 'clients', backend and API is under 'server'. We have instructions for managing the database.

<zcorpan> https://aria-at-staging.w3.org/

Val: there is a deploy directory as well with a readme. There is a script for deploying to staging. We also have a vagrant set up to test locally if you don't have access to the actual servers.

Jemma__: what caused the error that I ran into?

zcorpan: that was a bug in the ARIA-AT app

Val: thank you for finding that bug

Seth: In the long run, we want to transfer the ownership of the app from Bocoup to the CG and the W3C.

<Jemma__> app was very stable in core functions.

Matt_King: That's the key, this is a community group owned resource.

<Jemma__> Thanks for brining the fund for this project, Matt

Pilot Findings

<zcorpan> https://github.com/w3c/aria-at/issues/162#issuecomment-642199169

<Jemma__> s/bringing/bringing

zcorpan: today I worked on going through all of the issues that were filed, and closing duplicates. There were a few that were very similar issues, but I didn't close those.
... there are 52 open issues, and I added a label for the tests

Matt_King: that's good, thank you

zcorpan: 47 were about the tests or the test data. 4 issues were about the test runner.

Matt_King: do we want to be able to distinguish if an issue is related to the runner (aria-at-app) or the tests (aria-at)?

zcorpan: I think that makes sense.
... and there is one issue about the process. We should make sure that we respond to issues in the timely manner during a test cycle.
... Seth and I have been working on a document today that goes through our findings for the pilot. It's not quite ready yet.
... I expect more test runner issues to be filed
... any questions?

Matt_King: thank you for going through all of those issues

rob-fentress_: sorry if I logged too many issues

zcorpan: no, not a problem. Thank you for logging them.
... Michael found a bug in one of the Jaws tests that prevented completion of the test and you made a fix with a PR
... the problem is that we won't be able to complete the pilot test in the way the app currently works, since we are fixed to a specific commit from the ARIA-AT (tests) repo

Matt_King: that should be one of our top priorities for v2, to allow us to pull in changes mid cycle

Val: you can still complete the run, but that skipped test will just be missing

rob-fentress_: is the plan to redo all of these tests now that we have done the pilot?

zcorpan: I think it makes more sense to address the issues we found before we start a new cycle

Matt_King: yes, there were a lot of issues in the tests themselves. We should also capture the lessons learned from those, so that as we create and review new tests, we don't repeat those same issues.

zcorpan: I don't think you need to wait for renewed work on the aria-at app to start a new test cycle

Matt_King: right, we didn't run into any show stopper issues in the runner itself
... one show stopper was the limit on assertions, which we already have a PR for

zcorpan: and we have been fixing bugs in the runner which will be deployed soon
... I want to learn about how much time people think they spent working on the pilot tests

jongund: It was slow at first, but it took about 40 to 50% less time after the first cycle. so I think it took about an 60 to 90min per run.

rob-fentress_: that's about what it took me to run the checkbox tests

(that should have been Joe_Humbert_ )

rob-fentress_: it was about 90min per test run for me

Matt_King: (to testers) your experiences are very helpful in knowing how we should approach onboarding and training

Jemma: I did checkbox for nvda/ff and it took me about 2 hours because I had to restart the test. So it only took me 40min for nvda/chrome. Menubar for nvda/chrome took about an hour.

Matt_King: I gave Jemma some feedback about how to interpret the word 'convey'

rob-fentress_: VO provides screen reader instructions for the focused element after pausing for a little bit, and it wasn't clear if I should use that information while determining if assertions were met

Matt_King: we should document and share with all of the testers what our expectations are regarding that additional output

Joe_Humbert_: one of the things that I found difficult was capturing output from JAWS. For one thing, non-printable characters were copied, and switching between the history in JAWS and the browser caused problems sometimes.

<Jemma> That is a good feedback, Joe.

Matt_King: I mentioned to several of the screen reader developers that it would be great to have a command to copy the last utterance and to append the last utterance to the clipboard

Joe_Humbert_: I also had one test that resulted in a conflict. How you navigated to the checkbox (arrow keys vs tab) changed what was conveyed, and could cause issues.

jongund_: this was the command to read the current item, but how you navigated to the form control affected what the read command said

Matt_King: are those conflicts still there?

(the last jongund_ comment should have been Joe_Humbert_)

Joe_Humbert_: yes

<Jemma> One minor feeback for chrome and nvda testing is that once the example was opened and the button is grayed out so I cannot open the example window again. The case of reopening example window is when I lost the example window by clicking external link accidently.

GitHub team transition

zcorpan: after the new deploy that will happen after this meeting, the new teams in the w3c org will need to be used to sign into the app. We will need to invite everyone from the old Bocoup team to the new W3C team

Zakim: that will happen later today

<Jemma> yes. I understand what Simon said.

Matt_King: to make sure that this is very clear to everyone. If you used the app during the pilot, you were added to a Bocup team. That wasn't ideal, so we created similar teams in the W3C org.

This is scribe.perl Revision of Date 
Check for newer version at http://dev.w3.org/cvsweb/~checkout~/2002/scribe/

Guessing input format: Irssi_ISO8601_Log_Text_Format (score 1.00)

FAILED: s/bringing/bringing/
Present: spectranaut Jemma__ Joe_Humbert rob-fentress jongund
Found Scribe: michael_fairchild
Inferring ScribeNick: michael_fairchild

WARNING: No date found!  Assuming today.  (Hint: Specify
the W3C IRC log URL, and the date will be determined from that.)
Or specify the date like this:
<dbooth> Date: 12 Sep 2002

People with action items: 

WARNING: Input appears to use implicit continuation lines.
You may need the "-implicitContinuations" option.


WARNING: IRC log location not specified!  (You can ignore this 
warning if you do not want the generated minutes to contain 
a link to the original IRC log.)