Meeting minutes
Review agenda and next meeting dates
https://
Matt_King: Requests for changes to agenda?
Matt_King: Hearing none, we'll stick with the agenda as planned
Matt_King: Next Community Group Meeting: Thursday May 1
Matt_King: Next AT Driver Subgroup meeting: Monday May 12
jugglinmike: Valentyn of Vispero ought to be able to join us for that one
Current status
Matt_King: james and IsaDC and I have been working on a schedule for upcoming test plans
Matt_King: There's a pretty solid plan through the end of June. Of course, circumstances may change, but I think we can have 25 test plans in "candidate" or "recommended" by the end of June. That's my target for now
Matt_King: That's based on the fact that we currently have four in "draft review"
Matt_King: We have seven more where the plan itself will be ready by the middle of June. That's probably the biggest stretch. I don't know how much testing we'll be able to get done before the end of June
Matt_King: And we have 14 currently in "candidate review"
Matt_King: That all totals to 25
Testing of Rating Radio Group
Matt_King: There were no conflicts when I checked yesterday
Matt_King: For JAWS, it looks like we're all done. Thank you!
Matt_King: We can mark it as "done" because there are no conflicts
IsaDC: Will do
Matt_King: In that radio group, were you getting the set size spoken consistently?
IsaDC: Yes. I wonder if it has something to do with the version, because I used the April version
IsaDC: I was going to ask if I should re-test
Matt_King: Right, I think we'll get to that in a minute
Matt_King: NVDA is set up for Joe_Humbert and dean. Joe_Humbert is all done
dean: And I plan to finish this evening
<Joe_Humbert> I'm just trying to get it done when I have time
Matt_King: No one has started VoiceOver yet. That's assigned to Joe_Humbert and mmoss
Joe_Humbert: I had 15.4.1 update which took a while last night, so I didn't get to the third screen reader last night
Matt_King: Do we have 15.4.1 in the system, isaDC?
IsaDC: Yes, we added it last week
Matt_King: This test plan is going very smoothly. This is great!
mmoss: I should be able to complete this today.
Matt_King: We may be landing this one pretty quickly, then. That's awesome
Re-run of JAWS for color viewer slider
Matt_King: This is assigned to Joe_Humbert and Hadi, and it looks like Joe_Humbert has done this one, too. Wow! But we don't have anything from Hadi, yet
Matt_King: I think he said to give him a couple weeks, so I think we're good there
Matt_King: So I guess there's no update there, other than that Joe_Humbert is done. Thank you, Joe_Humbert!
Issue 1221 - Conflicting JAWS results in radio group tests
github: https://
Matt_King: I commented on this yesterday, and I tested it with a beta version of JAWS which is directly after the April release
Matt_King: I put my information in the issue
Matt_King: I'm getting the same results that Joe_Humbert got in Windows 10
Matt_King: And then you got the same results in the radio group
IsaDC: I'm happy to change my results to match Joe_Humbert's
Matt_King: That would complete this one. Wow, we could end up being done with all the "radio groups" very soon
Matt_King: If that resolves everything, then you can go ahead and push this forward into "candidate review"
IsaDC: Sure
Issue 1212 - Sandbox testing of test plan changes
github: w3c/
james: This came about just because we've had a few instances where we updated a test plan and wanted to see how it manifested in the app. Or where results have unexpectedly not been carried over
james: The individual things which prompted it are worth discussing on their own, but essentially, we think it would be helpful to preview test plans in a version of the app that is authentic, without requiring merging a pull request
james: We currently don't have many roll-back opportunities after merging
james: for example, we have the "staging" environment and the "sandbox" environment, but they don't really reflect "production" to the degree that we can use them to reason about things. They lack the wealth of data in the "production" environment
Matt_King: If we had the ability to get a most-recent test plan data into the staging environment, but to operate off of a branch...
Matt_King: I guess the main thing here is: once a test plan is merged into the "master" branch, the only way to correct problems is to merge a new version into the "master" branch
Matt_King: The problem is that we don't have an environment that has all the prior results for the test plan, so we can predict what merging will really do.
james: yes
Matt_King: So this is really a data thing
ChrisCuellar: What was the original intention behind the "staging" environment versus the "staging" environment
Matt_King: The "sandbox" environment was for developers to push whenever they like. Like a "nightly" build
Matt_King: The "staging" environment was meant to give external stakeholders an opportunity to view something new while still using "sandbox" for ongoing work
Matt_King: We should be able to experiment with the data in the sandbox to any degree
Matt_King: I kind of wonder if maybe the data in staging should be closest to production
Matt_King: But james is saying that we don't have an environment that matches production and which is safe to mess around with
james: We also recognize that "staging" may have features which is not in production
james: I think the ideal would be the ability to have a copy of "production" on-demand. And to have that copy be able to read from GitHub
Matt_King: I want to be careful to not build a massive new thing in order to solve an occasional problem
Matt_King: I think there may be a way to use "staging" to solve this problem
Matt_King: It could be (especially given the way we've been working lately) that we could do something so that the staging environment is essentially equivalent to production, except for the possibility that we push something new to it. But we have the ability to go to the staging environment and say "at this point in time, run this script to set the data in the staging environment to be equivalent to production"
Matt_King: ...and if we also had the ability to pull in test plans from a branch other than "master", and for you to choose which branch that is
Matt_King: In staging, we could have a feature that says, "from which branch do you want to pull?" So if you have a pull request branch, you could pull from that branch and then work with it in staging, and it would be just like working with it in production
howard-e: It won't be exactly the same, though. There will be times where staging will have updates that go beyond production
Matt_King: yes, but I think 90% (or even 99%) of the time, those changes are not related to the kind of functionality which affects how the test plans are going to be processed in the test queue (and things like that)
Matt_King: We don't often touch the code that affects things like the code which controls how test results are copied
Matt_King: It feels like the velocity of change in staging is quite manageable
howard-e: Good point
howard-e: To re-share what I stated in the issue, I agree with all of this
Matt_King: I don't want to let the perfect be the enemy of the good .Even if we could get a "90%" solution...
james: I'd love to understand how often we need to share something on "staging" with an external entity. It doesn't seem like it happens very often at all
james: If this could be put in place, I would encourage us to always put changes through this flow in order to be diligent about testing our changes
Matt_King: We currently have the "preview" capability, and our process involves using the "preview" capability before we merge
james: Right, and I don't want to suggest that we should start looking at every single test in the app. I think the preview is good for reviewing the underlying test plan itself
james: Most of the time, I think this kind of review should only take a few minutes at most
Matt_King: We could have staging serve two purposes. We can have it serve the purpose you're describing, james. We can also have it serve the purpose of staging new changes to the app, but only when the process is such that the development team needs it for that purpose
Matt_King: In other words, we could go directly from sandbox to staging to production on a super-fast path almost all the time
Matt_King: Right now, we have the feature related to automated updating. When IsaDC and I were giving feedback, we did it in sandbox. When it's time to deploy that feature, do we always go through "staging", or do we go direct from "sandbox" to "production"?
howard-e: We always go through "staging"
howard-e: It generally takes a week to move from staging to production. It's a manual process that we run internally
Matt_King: I wonder if that adds much risk to the kind of previews that james is describing. If you make a change to the test plan, and it looks good in staging, so you merge it, and it goes to production. If something isn't quite right, it could take a week to resolve
james: I think it adds quite a bit of cognitive overhead to need to have a sense of "what state is staging in?"
james: Also ambiguity. It somewhat undermines what I was trying to achieve when I raised this issue
Matt_King: I don't know how to do this without a whole new environment
james: Couldn't we make that environment much more ephemeral? Couldn't it happen in GitHub Actions? We only need it for a short time, and then we can throw it away
Matt_King: I don't know
james: Or, how easy is it to get the app up and running locally? If everything is Dockerized, and all someone has to do us run "docker-compose up", then these concerns go away
howard-e: It is not Docker-ized. It could be. While the operating instructions are minimal, it may be preferable to Dockerize
ChrisCuellar: I wonder if the Bocoup team can take this internally for discussion. It sounds like there are a lot of options, and I wonder if it would be helpful for us to consider it as a team and come back to you all with some recommendations
Matt_King: Yeah, why not?
Matt_King: I was actually a little surprised that building locally which PAC might prefer
Matt_King: Maybe making it possible for anybody to do that more readily might be good for the project overall in ways that I don't foresee
james: Dockerizing is something we do with other clients quite regularly
james: We'd have to figure out a way to share the latest SQL dump
Matt_King: I'm not familiar with the dockerizing process, but I kind of wonder if others (e.g. Vispero) might benefit from that capability
Matt_King: Let's assign this issue to someone at Bocoup and remove the "agenda" label. When you are ready to discuss again, please add the "agenda" label back on
App issue 1382 - Candidate review improvement
github: w3c/
Matt_King: When someone goes to the "candidate review" page (e.g. James Craig from Apple), you have to tell them exactly where to go
Matt_King: It would be really nice if the tables on that page were sorted by the order of priorities that we cared about, and if there is not something for them to do, then there isn't a row there
Matt_King: So I'm proposing that we sort the rows in the top three tables
Matt_King: Those tables have rows for every plan. If the vendor has approved, the "status" column says "approved", but they are still present (even though there is nothing for vendors to do)
Matt_King: The summary table should always show that it is approved. But we treat the vendor tables as the "to do" and remove them as they are addressed
Matt_King: The second part is to prioritize this "to do" list according to target dates
Matt_King: I'd like to use that feature better--to set realistic targets for AT vendors
Matt_King: I was thinking of super-light-weight changes that we could do to speed things up, and this is what I came up with
howard-e: When it comes to omitting "approved", won't the reports be in a state of ambiguity?
Matt_King: I'm not talking about changing the table
howard-e: Right, but that isn't represented anywhere else
Matt_King: It would be represented in two other places: one is the "summary" table at the bottom
howard-e: I think what I want to discuss may be a separate issue. I'll raise a new issue to discuss that
Matt_King: As I'm talking out loud about this (I only just raised the issue today), I'm thinking about how to make those first three tables on the "candidate review" page to function more like a "to do" list for those representatives
Carmen: I wonder if we're trying to get the same page to work for distinct use cases, and that's making friction
Matt_King: I think it's all one purpose: it's candidate review, and it's intended for AT developers
Matt_King: Is there another thing that someone is thinking that the "candidate review" page does?
Carmen: I thought people were also using it to understand the overall status of the test plans
Matt_King: The assumption is that the "candidate review" page primarily serves the AT vendors. Though the "summary" table is somewhat for us. But it's also for them--to allow them to see where they stand relative to their peers
Carmen: I understand
Matt_King: Right now, we expose the information about each AT vendor's competitors
Matt_King: I think having three on this one page is okay, but I didn't want to go far and suggest that we customize for each AT developer. But that feels like a much bigger lift to me
Carmen: If they're saying it's complex for them, then maybe we remove the summary at the bottom
Matt_King: I actually love your thinking there, and I think it's kind of cool!
Matt_King: I don't want to make too much work, though. Maybe we can start talking as a task after this
Carmen: I can create an issue for us to revisit later
Matt_King: great!
Issue 1214 - AT version recorded for reports where bots collected some of the responses
github: w3c/
james: This came from a discussion with jugglinmike
james: It revolves around a bot run happening, and the bot run recording its browser and AT version, and then a human can take those responses and provide their assertion verdicts
james: however, when this happens, the human's test plan run retains the original browser version and AT version, which may be inaccurate
Matt_King: When we add someone else's results, we have a pop-up dialog to confirm your AT version
Matt_King: Could we utilize that functionality? Or would we make it manual for the tester?
james: I was thinking that it would work the same as when they started their own test run
james: right now, we're losing data in that we're not tracking what versions the human used
Matt_King: If a human is working with results following a bot run, do we care the versions used for the bot?
james: I guess we care if the human needs to change the results that the bot had. We might care about discussing why that was. Or, more likely, we might want to know that it happened so that we can have visibility and so that the tester can raise red flags
Matt_King: Maybe when a report is worked on by both a bot and a human, we record both sets of browser and AT versions separately. That could even be part of the run history.
Matt_King: We're out of time, but we can continue during next week's meeting
Matt_King: Thanks to everyone for the excellent support! Talk to you next week!