<Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Rule_Publication_Tracking
wilco: charu is no longer in the task force, so we need to reassign her work
... first rule is orientation of page not restricted. assigning to shadi, have survey need an issue opened
... next one, role attribute has valid value, I will take that one.
... Two unassigned, element marked as decorative is not exposed, meta element has no refresh. element rule assigned to kathy, meta to trevor
... All have surveys done, they need to have issues opened. Would like everyone to go through this list, make sure everyone's issues are up to date
... I think there are 11 we have done a survey that we have never resolved
maryjom: There may be a few additional rules that have not been added to the table.
... aria-required context role assigned to maryjom, heading has non-empty name assigned shadi,
... third already assigned to kathy
<Wilco> https://w3c.github.io/silver/guidelines/
<shadi> https://www.w3.org/WAI/GL/task-forces/conformance-testing/work-statement
wilco: Our work statement says that we will contribute to WCAG 3 and how to integrate ACT into that, We should start to discuss this
... Come up with a few questions that we would like to discuss. Come to the silver group with suggestion of how we might integrate
... biggest change is switch from success criteria to outcomes. Outcomes are not true or false, they are graded from 0 to 4, 0 bad and 4 is near perfect
... in addition, there are critical errors, if a page has a critical error then you fail.
... allows to have nuance, can conform with small issues but not with critical errors
... Underneath the outcomes are methods, are similar to techniques. Contains section on how it works, how to test, etc.
... Looking to start getting answers for the questions posed in the agenda.
kathy: Seems like they are giving higher value to AT testing, ACT rules may not be as easily testable through AT.
wilco: I have heard talks about AT testing at higher levels, silver or gold
... We don't know anything about that, so we need to talk about bronze level
trevor: Silver and gold are supposed to be more usability tests. We may need to formalize how implementers say their methodology.
wilco: still don' t think we know enough to have a meaningful convo.
kathy: I think there can be confusion of how criticals and non-critical errors are tested and reported. Non-critical may achieve bronze.
wilco: My understanding is you can have bronze, silver, or gold even if there are minor issues.
... They want those levels. The bigger concerns with WCAG 2 is that it doesn't allow minor issues.
trevor: would we want test cases for each level?
wilco: I don't think so, I think each level would have its own rule.
maryjom: I think it depends, for example a rule might be a percentage of images with appropriate alt text. Our rule would test if appropriate alt text exists, but not sure we could aggregate it
wilco: If we want ACT to fit into WCAG 3, we will need to add something for aggregation
shadi: My fear is regardless of binary or levels, we need to be clear of results. We need to be test driven, we may need to change the rules format or the entire approach.
... How we can help is figure out the format for writing test cases to formalize this.
wilco: Agree, test cases make all the difference, can't prove implementation without it.
shadi: In WCAG 2, wanted more testing, but still a bit inspirational. WCAG 3 is trying to get more testing
wilco: Everyone thought we understanded the same things, but ACT work has exposed differences
... Should one of our recommendations be to do tests first?
<shadi> [test-driven development]
wilco: Create tests as they create the outcomes
maryjom: I think its best to write the tests as we develop the standards to better understand the impacts. can help hone the requirement
<Wilco> https://www.w3.org/WAI/GL/WCAG3/2020/outcomes/text-alternative-available
wilco: The most obvious place for ACT to fit in, is within the methods. Its a list of methods for the text alternative]
<Wilco> https://www.w3.org/WAI/GL/WCAG3/2020/methods/images-of-text/
wilco: The method has a tests section that currently links to ACT rules, does not explain how they relate.
... Then breaks it down into automatic and manual tests, with holisitc tests still in development.
... It seems most logical place would be here.
trevor: questions on efforts duplication, when would I use one or the other (ACT rules vs. what they have documented).
wilco: Especially confusing how to get specific outcome levels, how do you combine the results to get one of those scores.
shadi: I think these examples could be test cases, with good or bad examples.
... would suggest focus on the examples, but work them out into test cases.
kathy: I agree with what shadi is saying, we have had some long discussions on specific examples. There are a lot of different directions that a reader could take.
... Within this single test, it seems to have overlap with other methods. Could cause confusion.
maryjom: Seems like they want to have an "overall" test.
kathy: They are also using different terminology, unit vs. target as an example
wilco: I think its notable they make a distinction between automated and manual testing.
... There is a whole section where they are writing test procedures and over to expectations.
trevor: Should we bring up why we struggled with some of these things?
wilco: I think so, I think we had good reasons for not doing them.
... One challange I still have is how do we go from what is inside of the method to having a score.
maryjom: And they seem to overlap as well.
<Wilco> https://w3c.github.io/silver/guidelines/#text-alternative-available
wilco: From link, you could potentially find all images and come up with the percentage.
kathy: That could be long and arduous
shadi: Potentially prone to manipulation
wilco: Also seems unclear how you would deal with a single image failing one method and passing another method. Fail once vs. pass once
... Would it make sense for us to create a document of recommendations for WCAG 3. I can collect some of what was said and bring it back to the group