Meeting minutes
App Development - Merge conflicts in test format?
s3ththompson: Did we discuss using a dev branch to present merge conflicts?
James: - Not yet
James: Open to alternatives, but part of the reasoning was that status bit was not a test writing concern, it was a community concern.
s3ththompson: Will we reach a state where there are so many tests that we don’t want to change the status bits individual?
James: For a longer non-APG test plan, or a plan where we’re waiting for a more compositional test approach, there might be a test plan with lots and lots of tests
s3ththompson: what about merge-driver that understands our csv format? what about moving to a directory of files? (instead of an array in a csv format)
James: both could work... we definitely need some improvement
James: any short term mitigation would be good. current output from generation script is not very accessible... lots of scrolling to get back to the actual error
James: one big win would be to only generate files for test plans that have changed
s3ththompson: do we need even to have generated files checked in next to source files?
James: not necessarily, could be a feature of the app... whatever it is, we just need a simple way to preview
s3ththompson: given that we're working on both app & test repo infrastructure... we are open to possibilites that shift work from test repo *to* app
James: i agree with the idea to identify the "right" solution (the "real" fix, so to speak), but i think we also want to make sure that we identify short term mitigations to work towards in the process
James: merge conflict, test workflow topic is higher priority than test format for us
s3ththompson: I think broadly speaking we could definitely look at infrastructure stuff before the test format
s3ththompson: we mainly want to make sure that we are aware of upcoming test format changes when we do our database refactor
App Development - Sprint 4
Seth: we've been working on changes to the app data model to support either working mode changes or generally to support greater flexibility
Seth: Right now the biggest constraint on the app is, first, the git sha is an underlying parameter that affects almost all of the views in the app
Seth: RE versioning test plans independently, we are thinking about CRUD operations and views to no longer assume a single version for all test plans
Seth: Another thing that has gradually become a limitation is the fact we currently save runs to a test table, and grouping them was a presentational question only, not specified in a model
How admins determine which tests are active needs to occur as a derived state based on the active test plans and the active AT/Browser configurations and test versions. The entries on the test queue, then, are entirely derived. This, initially, saved us from having to generate a huge matrix of combinations at the outset. However, this is tricky at this point, since it blocks us from changing the global version used by the app'
Seth: to summarize, we are moving towards a model where the test queue exposes normal CRUD operations, allowing for direct control over what the test queue displays.
James: to clarify, let's imagine a test like Combobox is finalized and active. Let's say it's version 1. Then we decide in 18 months to add support for touch-based screen readers. This is going to have implications for the tests, so we create version 2. Can this be a draft, tested with a different group of testers. They're not testing primarily for results, but as a review process, a QA process. Occurring within the app. Could it be
possible to have combobox v1 and combobox v2 handled, assigned or shared separately.
Seth: yes. That is a use case we want to support but don't currently, which is why we're making this change.
James: What would happen now?
Seth: since the only parameter you can change is the global version number. But when you change the version number the old runs will be, not deleted, but inaccessible. Currently. The context for that limitation being that we expected the test results to be published separately - this was the test cycle concept - but the issue there is that it assumes a model where we test, review and publish to completion and then go back to a blank
Seth: ...slate, and it's clear that this requirement to wait is something we'd like to revisit
James: The 60-day review process will require us from distinguishing changes made to tests during that window
Seth: Yes, exactly
Seth: in the past you had to immediately replace version 1 with 2, but now you can keep both around
Seth: it won't automatically group the results across 1 and 2
James: what about adding a period
Seth: so far we haven't proposed a shortcut for small changes, but it's something we could consider
James: a changelog or semantic versioning might work
Seth: right, that's a good point
Seth: What do you think of the term test cycle, James?
James: I think it's ended up less useful. I recall Matt being quite clear this is no longer an important consideration. We will have to ask him about that/
James: we need to be fluid in how we assign things
Seth: the upcoming sprint is mostly about infrastructure and testing. We will spin up a new dev environment hosted by Bocoup servers, in addition to the current staging and production environments hosted by the W3C. The purpose is to API tests our routes based on our desired behavior, to set the stage for the implementation work
Matt: exciting to have GraphQL!
Test writing meeting agenda starts here
group discusses the impact of sleep on memory and working...
joke/ should we update working mode to require minimum sleep?
Wording of assertions that use terminology for screen reader functions
Matt_King: for JAWS and VO command differences... I thought of these commands as reading the element itself... but including the words "information about", (esp. if we later add assertions for reading descriptions) feels like it's unnecessarily verbose
Matt_King: "read Combobox" vs. "read info about Combobox"
James: some elements sounded clunky e.g. "read unchecked radio button"
James: you're not really reading the unchecked radio button, but everything about that button
Matt_King: I think "name", "role", and "state" is basically implied by the word "read"
James: but if the name is "read a grid cell" it's really ambiguous since it's not reading the content... its reading info like coordinates
Sina: screen readers make some of these distinctions ambiguous, but we should be more pendantic about the different intentions to properly disambiguate between the different ways of reading an element and its metadata
Sina: analogy, for a letter... what do you say when you want to read stuff about how the letter was to, how it was delivered, and what the package is?
Matt_King: tricky part is that grid cell is both letter and envelope, while radio button is just letter
Sina: i think the executive summary is separate from the raw data... are some of your concerns Matt that the executive summary needs to be different than the raw data?
<Alyssa_G> Could you use the word "present" or "presents" instead of "read" or "reads?"
Matt_King: I want to be clear that the terminology is framed in terms of the user's intent...
Sina: why don't we just have a table of definitions that we write alongside the test instructions and defer these conversations to a discussion about what the definitions should say
Matt_King: yeah, that's what I'd like to do... and i'd like the verb teminology to be concise
Matt_King: the other one i want to bring up is "read sequentially" which feels unnecessarily verbose
James: when someone is expected to press "down arrow" to get to result... should they be expected to know how many times they should press the down arrow? or should we encode the tests so that they press the arrow key *until* they find the right element and then we would record how many times down they had to press (and we could track analytics about that data point)
Sina: in my mind if tabbing to something vs arrowing to it creates a different result, we should really be fixing that, rather than testing it and catering to it
Matt_King: i care about being able to test with granularity
Matt_King: coming back to issue https://
James: to be clear, we do have different tests for arrowing to an element vs tabbing to it…
Matt_King: so what's your take on how we distinguish multiple ways of "navigation"? just grouping them both under the same verb? we don't need different verbs, since the commands have the distinction. I'll reread and think about this again...
Sina: I think we might get pushback, but i think it's a conversation we should have with vendors... make them justify why different ways of navigating to an element should have extra or less info
Resolution: will consider changing "read information about" to "read" after we have a glossary of test terms. Then, we would script the change if we make it.