Accessibility Conformance Testing Teleconference

24 Jan 2019


MaryJo, Shadi, SteinErik, Anne, Trevor, Moe, Wilco

Wilco, MaryJo



Anne, what we talked about last time. outcome reporting changed from test target to test subject. Didn't realize change before the edits last week.

Anne, We report more granular data.

Wilco, don't understand, you can still report test target level

Anne, Finding section also added which maybe to some extent covers some of it, but no term currently for stating that this target passes a certain rule.

Wilco, Meets the expectations?

Anne, Many edits makes it hard to grasp

Anne, we wanted terms or language to use to agree on. All that is removed from the test target level and put into a definition of "finding", which is only used for composite rules, but not mandatory

Wilco, ACT format doesn't have requirements for how to report - implementing

Anne, we have definitions for reporting outcomes. We decided to take aggregation out of spec, and we are back to the concept of aggregation on a page level, I guess, without having all the framework for how to work on test targets level

Anne, I am asking for us to require reporting of the outcome of test targets, which I understand was the case until I read last weeks edits

Anne, there have been assumptions for how things were working or should work that affect interpretations

Trevor, fi you pass all test targets, wouldn't that mean you pass the rule?

Anne, currently you can't pass a test target. A test target can only meet expectations.

Trevor, missed by scribe

Anne, I would like to be able to say first video on this page passed this rule, rather than whole page passed this rule

Shadi, I see what you mean. We discussed this when we discussed reporting, and we agreed to move it to informative.

Shadi, in WCAG conformance is defined on a page level

Shadi, what does it mean if one video fails the rule?

Shadi, it is a design issue. How do we break it down? What is stopping you from doing the detailed reporting in your implementation?

Shadi, In the reporting we tried to find a consistent way of reporting.

Shadi, the video in your example, would you be pointing to the video element or the button or which part of that? That is why we decided this would be a reporting/implementation thing.

Shadi, If we have these test cases and implementations report the same, we don't really care.

Skotkjerra, in our definition we say a test subject can be a collection - how do we handle this? E.g. in the bypass block SC

Shadi, the most granular subject in WCAG is pages.

Shadi, What are we trying to achieve that is not on implementation level

Anne, clearer definitions.

Anne, terms are more loosely defined or not defined.

SHadi, there is an issue of clarity vs. general terms. We have tried to work on this through the document.

Shadi, the other question is the level of reporting

Wilco, I am concerned to. It was unfortunate that we lost this as a consequence of the decision we have made

Anne, Unsure which decision lead to this

Wilco, it is unfortunate that we had to rewrite expectations to meet test subjects rather than test targets. That is the result of a discussion we had at TPAC, and that this was the way to go.

Wilco, The discussion at TPAC in Lyon: Discussed how do you do the accessibility mapping. Initial proposal was to leave this fairly open so there were more options to mapping.

Wilco, Objections to that.

Wilco, we would have to redefine expectations to contain expectations of the atomic rules...

Shadi, I recall the discussions, but recalling a different discussion about reporting and pointing to the different test targets.

Shadi,m which proposals are on the table?

Shadi, several people indicate that they would wish for a change

Anne, I have made an example, and we could say that this rule applies to all images on the page and 85% of them satisfies this certain condition. We should be able to do that within the old wording with test targets

Wilco, so this wouldn't be a composite rule?

<anne_thyme> https://github.com/w3c/wcag-act/pull/322#issuecomment-457185354

Wilco, it doesn't seem like you can compose a rule like this

Wilco, we would have to reevaluate the accessibility mapping because multiple pass and fail outcomes for each rules

Anen, that was what we thought we had before. Might be because we weren't grasping consequences of using test subjects

Trevor, if you have a mixture of passed and fail targets, would that mean that the atomic rules fails?

Anne, if you have anything failing in your test subject, then your test subject is failing

Anne, if you only have passed or inapplicable test targets within your test subject, the test will be passing.

Wilco, So your proposal here, I am guessing, is revert and if you want some sort of complex aggregation you have to rewrite the rule to accomodate for it?

Anne, yes…..maybe……or we could allow for composite rules that could say 80% of test targets in this atomic rule is passed for this rule to pass

Wilco, I don't think I fully understand the consequences of the proposal

Wilco, I am ok with the current solution, although I do not think it is ideal.

Anne, I could live with this solution with improved definitions

Wilco, proposal is to strengthen the definition of findings

Anne, what I liked before I thought that there was a term for meets all expectations that was just passed. That is kind of what I am missing now

Anne, especially when using results in composite rule

Anne, I really like passed and failed, but they have other meanings, so I don't really know

Wilco, not sure what we could change

<MoeKraft> Here's the current definition of Outcome:

<MoeKraft> Outcome One of three types of conclusions that come from evaluating an ACT Rule on a test subject. An outcome can be: Inapplicable: No part of the test subject matches the applicability Passed: The test subject meets all expectations Failed: The test subject does not meet all expectations

Wilco, meets all expectations, is that enough?

Anne, a lot of word instead of one term. Gets vague and conversational.

Anne, if we put in meets all expectations, that should b enough, but could use a better word

Wilco, "meets all expectations from the following rules"

Wilco, how about I create a definition "meets all expectations"?

Anne, let's go for it

<MoeKraft> So are we throwing out the definition of passed?

Wilco, the glossary section replaces the requirement for a definition of outcomes

Wilco, can we do a CFC?

Spec readiness for a TF review to get a CFC to ask AG WG to publish

Wilco, I'll send out a new version later, and everybody wil have until Monday

Wilco, if no comments before Monday, we go to CFC

Conclussion: CFC on Monday if no comments

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes manually created (not a transcript), formatted by David Booth's scribe.perl version 1.154 (CVS log)
$Date: 2019/01/24 16:06:56 $