Meeting minutes
Sprint 6 review
<jesdaigle> +present
Seth: sprint 6 completes in a week on May 28
sprint 7 is may 31 to june 11
Added graphQL and started work on control layer
that will have more of the business logic than prev version
Data model has more flexible set of entities to accommodate working mode changes.
Working on test renderer to fix outstanding issues around command sequences and reset button behavior.
Z hasdraft PR open, not quite ready for review.
Cleans up the test harness and documents the methods iwth jsdoc
Should have sig progress by end of sprint 6, but may not be complete depending on our new task we discussed last week related to generated files.
Generated file work is higher pri and might push out the renderer.
James: when to talk about the generated file issue with merge conflicts.
Seth: Z has investigated, we can talk about it during this hour if you like.
mk: better for us to talk about the details of that in this meeting.
seth: just opened issue today related to roles in the app
james: we can put that first on the agenda in the cg meeting next hour
Sprint 7 plan
Seth: highest pri item will be automation work flow and removing generated tests.
close to 10 days of work.
so will start in sprint 6 and finish in sprint 7
hopefully can eleminate the biggest merge conflict issues sooner than end of sprint 7.
Plan to use netifly to provide test preview.
Next on the list is work around aria-at app roles.
Need some change to how we manage roles in github teams.
Adding everyone to w3c org in github adds alot of overhead.
Not sure we want that overhead or that it is appropriate.
Next is work to update the test queue page.
Allow test queue to be managed as individual items; add/remove individual rows.
Right now every plan in the queue needs o be on the same version, which doesn't scale.
New queue would decouple the queue from the github repo version/commit.
Might need to talk about having 2 version of same plan in queue at the same time.
mk: want to talk about how we render versions of tests.
seth: added a version field to the data model
James: could have it be free form field
rather than a number or semantic version.
mk: would be nice to have a number, date, and optional name/description for the version
james: semantic versioning could help with small changes like typo fixes.
seth: as soon as we rely on the version number being human generated, we need some checks to make sure the numbers are updated when they should be
seth: in wpt.fyi, any time a test is changed, it needs to be re-run to ensure it doesn't make material changes
If we use semantic versioning, should it be tied to when a test needs to be re-run
james: any change, even minor, needs to be signed off. We don't necessarily want to make it go all the way back to draft just for a typo fix.
james: right now we don't have a way to name a test plan
it comes from a directory
This should be a test plan level concern
We coudl treat a test plan like a package
seth: It would be good in the future if the app had more of a generica pproach to importing tests
almost like a webhook
So, we could support multiple sources and the changes in the repo could trigger the import.
james: there's iopportunity to do it bit by bit.
Coudl start with splitting up references.csv
We could also moving the naming of plans out of support.json.
seth: much of references.csv is apg specific. Can we make that more generic.
or, if we support multiple sources, could each source have its own metadata
james: I think it should be example driven
let's not try to be excessively generic
seth: there are 2 important links for APG, design pattern and example
seth: let's start an issue and come up with a package description that describes a test plan package with an eye on supporting sources beyond APG.
James: keep in mind all the shared resources
Plan for automated work flow and generated test files
Seth: We discussed how merge conflicts are acute issue.
One problem is that we have generated files in repo and CLI has verbose logging.
Proposal is to first put verbose logging behind a flag and have a more clear summary of errors at the end
james: as long as it is at the end, that is super helpful.
seth: that's quick fix to one problem.
Seth: next is to add a validate flag that doesn't generate any output
z: there is existing script that generate tests for a single direcory
james: there is no npm equivalent for that
seth: next is to turn on netifly deploy previews
Looked at other providers, but netifly is good balance of simplicity and fexibility
can work with branches
it is fast and keep the html files around.
they also partner with OS projects
Turning this on would give every PR a link that shows what the test would look like
seth: the app is loading files through github
In the new model, we would treat netlifly as a build bot
James: if netlify is short term solution, can we make sure the w3.org site is proxied to netlify somehow
z: think we can do that. today we use rawgit proxy so could change over
James: long term, we may not want to reply on outside source for rendered assets
seth: once weremove generated files from repo and serve from netlify, we could start building ability to render directly from the json
james: timeline?
z: Can get CLI work done quickly
Part of this is changing directory structure for some of the files
We need an account with netlify; free service is single user
Think that we can use their OS project support policy.
seth: I'll kick that aspect off with netlify
z: Netlify also has a cli tool that lets us give it a directory
james: as long as pr is free of generated resources, we are good.
seth: Yap, and this is top priority due to impact on your work.
Transition to Community Group Agenda
How should we manage ARIA-AT roles?
https://
james: currently, ARIA-AT App manages roles via GitHub teams under w3c org
james: due to how GitHub teams work, ppl need to be added to w3c org. there is a high overhead to adding testers to the w3c org
james: a few options from seth: 1) create GitHub org for aria-at? 2) create allowlist somewhere else? 3) keep roles under w3c teams, but only for admins
matt: can we keep admins on github w3c team and then add an in-app admin button to authenticate testers?
matt: we don't want to have a free-for-all of anyone submitting results
seth: in terms of lowering barriers to entry, is it really that bad if someone submits results that we later have to delte
james: it's noise that contributes to admin overhead. results take training to record correctly. even pilot test had lots of issues to work through and those contributors were trained and engaged
jes: from a bocoup perspective, we really want to make it easier for people outside this org to get involved. these seems like a crucial question as we talk about wanting greater community support for aria-at
matt: we're really quite committed to training tests and keeping that a careful, skilled process for only certain people. if we need to we can even hire and pay people to submit test results
matt: for short term, let's keep admin roles under w3c teams and add allowlist to repo
james: admins can add people or people can open their own pull request
seth: long term, might still consider in-app, better ux version
matt and james: probably not the highest priority right now
Compositional Test Writing
https://
matt: can we start with a simple toolbar example?
james: what is the value of asking the tester to toggle a toggle button inside a toolbar?
matt: we wouldn't test the functionality of everything in the toolbar, just navigating into and out of a toolbar and navigating between elements within in
*it
james: so we could create a boilerplate that we modify for each element
matt: we do need to check if you're on a radio button inside a toolbar that it still tells you you're within a toolbar
jongund: the code for this hasn't been updated
james: jongund, if you have another example other than toolbar, how much code do you reuse for something like a radio button inside
jongund: we certainly borrowed code from other places to bring into the toolbar. i don't think it's a ton of work, but it's rearranging code, etc. there are probably other efficiencies to look at. is that a priority?
matt: another part of your answer, james is that we don't create dependencies between directories?
james: but for something like grid, there are shared resources
matt: oh yeah, there's an import from util in there. but we don't link between examples
matt: the things that we would *not* put in the apg b/c they're too basic, would be e.g. a toolbar with three buttons
matt: another example would be an oversimplified grid if we needed to test a 2x2 or 3x3 grid... or maybe that's not a great example since the apg already has some simple grids
james: for test plan, can we import the tests for grouping? so we would almost have checkbox and grouping separated... so that you could reuse some aspects of them