maryjom: We have been going round and round about how to define rules and how their logic should separated.
... Now there has been discussions about how we test test targets vs test subjects. We have three options and are going to discuss those today
... The first is to roll back the decision on the mapping to pre-TPAC where we can have more complex aggregation logic
... Or we could choose to have the rules apply on the test subject level and then be able to report back on the test target level.
... The third option would be to have individual test targets and have other rules that apply that apply to sets of test targets.
<Wilco> done
*** Few minute reading break to read Anne's email regarding this issue ***
<scribe> done
<shadi> https://lists.w3.org/Archives/Public/public-wcag-act/2019Feb/0005.html
<kathyeng> done
<shadi> done
<maryjom> done
maryjom: Paraphrasing what anne said, she proposed a fourth option where you could have an input rule that would apply to indivual elements and expectations would be written to address each test target. The composite rule would be an aggregation of the applicability of each input rule, and there would be some expectation for the percentage that passed.
wilco: That is pretty much option number 3.
kathy: Also, I think she meant input rule to be atomic rule
alistair: In siteimproves mind, what is the test subject?
wilco: It is the page level. It is usually the page level.
alistair: That may be for you, but not everyone may be testing on a page level basis.
shadi: In my mind, the entire first part of the email, where she describes the mapping, does not address the very last example she gives about needing 85% of the test targets to pass.
... This was a use case raised at TPAC that we would want, and I don't think her mapping above would do that.
... You are making a statement about a collection of test targets, so you have to look at them as a whole.
maryjom: We also had to think about epub, silver, and rga
alistair: But that would also mean we are storing the past conditions. I thought we steered away from keeping the past conditions saved.
<kathyeng> https://w3c.github.io/wcag-act/act-rules-format.html#test-subject
kathy: The definition of test subject and test target are within the write up. Link above
shadi: What about the idea about the aggregation logic being only in composite rules. What Anne describes in her example, what is the issue with this approach.
wilco: What is the problem for using rules for aggregation? That isn't how we use rules, it turns rules into eithe rrules for testing or rules for aggregating
... Having something to aggregate is fine. But having something be a different variation of a rule seems odd and unnecessary to me
... Either the three listed, or we don't allow rules of this type
alistair: We haven't ever thought of using a test that tests for 85%. We would usually do that in the reporting.
... If we do that we don't much about with the atomic and cumulative tests that I like.
wilco: That is relatively close to option 1.
alistair: If you don't have the possibility to record passes it makes reporting harder
... Our tools don't record passes either.
kathy: We don't record things that pass in our manual testing either
maryjom: So siteimprove does record things that pass?
shadi: The proposal on the table is basically what anne describes in the first part of her email that there is a specific mapping on test targets and what it means for a test target to fail/pass and how that effects the outcome of the rule
... As far as I can think, this should cover all of WCAG 2.0 requirements, but would not be able to do 85% type rules
... The conclusion that some percentage pass would be some aggregation outside of the rule. We could have something in the output part of the specification. The aggregation would be outside the scope of such rules
alistair: I want to caveat that one. If 85% of some test targets is a big deal, why not just put that in the atomic rule. And write the test in there
... So if you want to say 85% of the images in the page. It would mean find all of the images, test them to see if they do blah, blah, and then output if 85% passed or failed
wilco: You could do that, but then you will have to rewrite all of your rules to do that. You would have a lot of duplication.
... If the applicability was an html page, the expectation would be 85% of test targets in that page
shadi: I think it is a different requirement all together. Ideally we would be able to reuse rules and such, but that may be where the overcomplication is coming in
maryjom: Should we write an example of such a rule to show how to use it
kathy; I have a question about where this came from. These percentage rules sound more like best practices than an accessibility requirement.
wilco: It depends on what you mean by best practice. The way i would define a best practice is a good idea but is not included in WCAG.
... So i guess option 4 is to not have a mechanism for these percentage tests at all.
kathy: I am still stuck on if you were to write a percentage rule, this would be handled for non-accessibility requirement rules
maryjom: That is what I would expect. Since it is not a requirement stated in wcag
trevor: Could we discuss option 2?
maryjom: Does that mean the outcome would apply for the test subject as a whole?
wilco: I don't like option 3, evwerything else I am fine with.
maryjom: **Looking at emails** Charu voted for option 2, Moe voted for option 3
shadi: I would like to hear again, thoughts on the downsides of option 4?
... I am hearing the major downside is having to rewrite rules.
wilco: It is that, plus you don't have individual test targets to report on for those kinds of rules.
alistair: You would run two sets of tests wouldn't you? You would run for the test subject and test targets. Code wise, you would just stick the code from one into another
wilco: The concern is that you may not do it consistently between implementors. It is definitely hypothetical. If you use that option, it doesn't explicity tell you how to get and report those test targets. You can infer it
alsitair: Do percentage rules work? What if you do an 85% of images and they only have 2 images. What does that mean?
alistair: It doesn't scale if you aggregate the images.
alsitair: Option 4 is not to bother with this. I think that is where my vote goes.
shadi: Not completely not to bother. You can address it.
wilco: so we are down to options 1,2, and 4 (the don't bother with it option)
maryjom: We could explain in a section that aggregation is outside the scope of this document
shadi: To make sure we are phrasing correctly. Its not that we not bothering with it. We can still express it. It is not our primary type of rule, so the percentage rules are a bit wierd. They aren't really what we are tyring to do.
... If that is how rules begin to be created in the future, then we will need to revisit. Selecting on option 4, we aren't focusing on these rules, but they are still possible.
maryjom: We are not going to directly address that spec in the case?
shadi: For me it was an eye opener that it can be done.
alistair: Something needs to go in, but just in a comment that says these rules can handle other things such as percentage, but it was not the primary thought behind it.
... Some level like that just to remind people that it was thought of. Isn't there an editors note that can added.
maryjom: Any objections, or have we come to consensus?
shadi: Maybe today, but we do need to put it out for other members of the group.
wilco: Seems like we have settled on 4.
wilco; option 4 is, rules outcome is for test targets specifically, we do not have a separate aggregation in the accessibility requirements mapping, and we add a note that other things such as percentage or complex aggregation can be done, but are rare. They cannot use results from existing rules
alistair: I would add a new atomic rule
maryjom: We will need a new pull request to show these changes.
... I can help do it, but I will need to consult with you. We will need to do a lot of work redefining outcome.
... We will need the group to read it.
shadi: I am not sure we will be able to have a cr before csun, maybe if we could have finished today.
... Yesterday before the call, I was leaning before another wide review draft. If we do go for option 4 I think we can go for cr, but I doubt it is possible