Meeting minutes
Matt_King: We have four topics on the agenda today, all all related to the app
Matt_King: the most meaty topic is the final one
Matt_King: Are there any additional topics anyone would like to bring?
Matt_King: Hearing none, we'll stick with these fours
Matt_King: the next meeting is Thursday October 5
Issue 795: Discuss changing title of reports page
github: w3c/
Matt_King: We talked about chaging the title of the "Candidate Review" page, but I think that the title of the "Test Reports" page could also use improvement
Matt_King: If you saw that title in Google search results, it would tell you nothing
Matt_King: Because so much of what we talk about is about AT interoperability
Matt_King: So my proposal for that page is "Assistive Technology Interoperability Reports". In contexts where we need an abbreviated version, we could use "AT Interop Reports"
Matt_King: Any thoughts?
Joe_Humbert: I like your title, and I agree it will make the page's meaning more clear when it appears in search results
James_Scholes: I support the new title
IsaDC: Me too
Matt_King: Great!
Issue 738: Changes to unexpected behavior data collection and reporting
github: w3c/
Matt_King: I'd like to prepare a better mock up so there's no ambiguity in our future discussion
Matt_King: But one of the decisions we made last time is that when we record an unexpected behavior, we would assign one of two severities (rather than one of three)
Matt_King: I was working on this because I didn't want to change from "high", "medium", and "low" to just "high" and "medium". "medium" and "low" also felt wrong
Matt_King: For now, I've settled on "high impact" and "moderate impact"
James_Scholes: As you say, "moderate" is almost meaningless because it could apply to any level impact that is not "high"
Matt_King: We want to set a fairly high bar for something that is "high impact."
Matt_King: The assertion will read somthing like "The AT must not exhibit unexpected behaviors with high impact"
Matt_King: And "The AT should not exhibit unexpected behaviors with moderate impact"
Matt_King: I'm going to move forward with those terms for now and bring it back to this meeting next week
Issue 689: Data integrity strategy
github: w3c/
Matt_King: One of the things that we have to have a lot of confidence in is that all the tallies and counts and information we present in reports is accurate--and that we don't break it
Matt_King: When you run a report, the system is going to count up the number of passes and fails, its going to calculate percentages, and it's going to capture dates and times
Matt_King: There are an awful lot of ways for things to go wrong in that process
Matt_King: And as we transfer data to APG in the form of support tables, I wanted to ask: how are we going to approach making sure that the system doesn't produce errors?
Lola_Odelola: Through some of the work that we've already done, some of these questions have already been answered
Lola_Odelola: An outstanding question is: do we have any ideas for the types of queries we'd like to preform to make sure there are no anomalies in the data?
Lola_Odelola: What kind of anomalies would we want to check before and after a deployment?
howard-e: For the most part, I'd want to be able to trust that the tests that are being added--that the system catches problems with those
Matt_King: I added quite a long list in the V2 format--a list of checks for the format
Matt_King: While I was doing that, though, I wasn't thinking about how mistakes in the plan could introduce inconsistencies in the data
Matt_King: There are some checks like, "every AT must have at least one command mapped to every assertion" or something like that
Matt_King: And I have a separate issue related to being able to specify that an AT has no command
Matt_King: But now, I'm thinking more about the data that's on the "reports" site
Matt_King: For instance, the number of assertions which have verdicts--that number shouldn't change after a data migration
Matt_King: I think it would also be important to check that for subcategories of the data (e.g. the total number of reports generated from recommended test plans, the total number of recomended test plans)
James_Scholes: Are we talking about validating user input? What are we validating against?
Matt_King: Against the data before the deployment. This is about ensuring that we maintain data integrity during deployment operations
Matt_King: Maybe we need to enumerate the scenerios where we believe data integrity could be compromised
Matt_King: I'm assuming that when you save and close in the test runner, that some checks are performed
James_Scholes: To what extent are checks like that already present?
James_Scholes: for example, during a test run, when a Tester provides results for a single Test, I assume that when they save those results, checks are made to verify that the data is correctly saved
Lola_Odelola: I think that part of the issue here (as Matt mentioned earlier), this is an oldish issue, and in the time since it was created, there have been a lot of improvement to the code and the processes
Lola_Odelola: Now, what we want to identify is: are there scenarios that could cause inconsistent data? We're asking because we have seen inconsistent data in situations we didn't expect
Lola_Odelola: I'm happy for this to be put on the back burner until something happens in the future where we need to readjust
Matt_King: Okay, though I'm still interested in building/documenting a fairly rigorous set of checks to ensure data integrity before and after deployment
Matt_King: That is, any time we make changes to the data model
Matt_King: For instance, when we do the refactor, we want to make sure that the total number of assertions doesn't change, et. cetera
James_Scholes: I'm not necessarily advocating for tabling this discussion, but I do believe that we need to have the existing checks documented before we can have a constructive conversation on the topic
Lola_Odelola: That makes sense. We can put something together for the group
Issue 791: Generating reports for specific AT/Browser versions
github: w3c/
Matt_King: Right now, everything we've been dealing with is reviewing drafts and getting them to candidate
Matt_King: We haven't cared too much about browser and AT versions; we've just recorded what Testers have been using
Matt_King: That's fine until we reach the "Recommended" phase of a report
Matt_King: At that phase, we're reporting to the world how this software works
Matt_King: The expectation we've had from the beginning is that once we have a "Recommended" test plan, we're going to keep that data current indefinitely
Matt_King: At that point, it becomes important for us to re-run the reports when each AT releases a new version and (perhaps to a slightly less critical extent) when each browser releases a new version
Matt_King: So, this can be a little bit challenging, and there are a lot of decisions to make in this space
Matt_King: Let's talk about screen readers, first
Matt_King: Right now, when we add a plan to the queue, we don't specify a screen reader version on purpose
Matt_King: (We had this ability three years ago, but we removed it because it was causing problems for us)
Matt_King: We still want to say "any recent version is fine" for draft and candidate
Matt_King: But when we reach "recommended", we need to enforce specific versions
Matt_King: One problem is that we don't have a way in the test queue that a specific version is required (that seems pretty easy to solve)
Matt_King: It also means requiring Testers to confirm that they're using the appropriate version
Matt_King: Thirdly, the Test Admin needs to choose a specific version make new Test Plans
James_Scholes: that all seems right to me
Matt_King: Okay, moving on to browsers.
Matt_King: All of this gets so much harder because (1) there are a lot more browsers, and (2) because we will need to maintain a list of browser versions, and (3) a lot of people don't have control over the version of the browser which is installed on their system
James_Scholes: for regular users who are not using corporate machines, it is quite difficult to install a specific version of a browser (first because they aren't always published--as in the case of Chrome). Secondly, even if you manage to do that, it's probably going to update itself, anyway
James_Scholes: I think that's separate from the "enterprise problem" where Testers are working in a corporate setting where there is strict external control over the applications installed on their machine
James_Scholes: I think it's really difficult to ask people to use a specific browser build. Even requiring just a major version is tricky
Matt_King: But we could ask machines to do this
Matt_King: I'm thinking about a decision that we could make. For the time being--until we have full automation across the board--we only ever require specific AT versions...
Matt_King: ...that we TRACK the browser versions (just like we do now), but we don't require it
James_Scholes: If we say that, "oh, we don't want someone testing with a very outdated version of Chrome", I think that may be a good middle ground
James_Scholes: Because if we go super-specific on browser version requirements, we'd probably also have to requirements on operating system versions
jugglinmike: We might want to track version release dates as well, because "four major versions old" means something very different for Chrome and for Safari
jugglinmike: The kind of warning we're discussing (telling Testers that they're browser is suspiciously old) will be hard to express for a single "version delta" across all browsers. Expressing it in terms of release date will be more meaningful
Matt_King: I agree!
Matt_King: So for the foreseeable future, we will not require updating the data for Recommended Reports in response to the release of new browser versions. We will only require updating in response to new versions of ATs...
Matt_King: And in that case, we will accept "any recent" version of the browsers under test
James_Scholes: sounds good
IsaDC: sounds good
Lola_Odelola: sounds good
Matt_King: Okay, that's good. This helps a lot. I'll update my work to consistently use the terminology "any recent browser"
Matt_King: Thanks, everyone!