Accessibility Conformance Testing Teleconference -- 26 Oct 2018

<shadi> scribe: shadi

<Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Exit_Criteria_for_Rules_Format_Spec

<romain> related gh issue: https://github.com/w3c/wcag-act/issues/224

Exit Criteria

<romain> scribenick: romain

shadi: we need to show different kinds of independent implementations for all the features in the spec
... also different technologies
... to show that the purpose of the spec is met
... all the requirements MUST/SHOULD/MAY should be covered
... we have proposed exit crieteria

Wilco: it was part of the CfC

shadi: it will need to be reviewed by the WG
... according the process, the director needs to approve these exit criteria
... usually delegated to Ralph Swick
... it's the so-called transition request
... we should let him know in advance, once we have the WG agreement
... in case of a dispute, or any other technical questions, there would be CR negociations meetings
... but I don't think we'll need them

Wilco: I think the exit criteria are in pretty good shape
... we'll need to update them after the discussion from yesterday (nested composite rules)
... then we actually have to produce this stuff
... is there a common process for this, how the information is collected?

shadi: two things before that: we need to ensure that every requirement is covered
... and also make sure that the "spirit" of the spec is covered
... ensure that we can prove it is really implementable in practice
... the big advantage we have is the parallel CG which already has a good collection of rules
... we need a test plan to work that out
... for WCAG it's a bit different, tools need to be built, etc
... btw, it doesn't need to be the same two implementations that implement the features [the 2-implementation requirement is on a feature-by-feature basis]
... we need to define our internal process on getting this documented
... who's going to edit this, do we split the work

Wilco: I wanted to show something, PR #291

<Wilco> https://github.com/auto-wcag/auto-wcag/pull/291

Wilco: Jay worked on a way to start tracking implementations of rules, for the auto-wcag website

<scribe> … New / In Progress / Done status

UNKNOWN_SPEAKER: there will be JSON manifest files
... report from different rules, reports the results of the test cases
... if all the test cases pass, it means we have one implementation

Jey: [describes the JSON test report file]

s/Jay:/Jey:/

scribe: you would see implementations and coverage reports for the different tools
... and then for each of them you can see the report details

Wilco: this was designed to meet the review process
... but it could be usable for the exit criteria as well

shadi: right. it is not only tools, but can also be evaluation methodology
... we have the WCAG report tool, and it will be able to spit out such reports

Wilco: no, not for implementations
... I think if you're doing automation, for each test case it should report a result
... if you don't produce test case results we will assume a manual evaluation

shadi: collecting information from auto-wcag will be very helpful
... but we'll need another table

maryjom: right, with the details of the MUSTs and SHOULDs

anne_thyme_: right, we'll also needs implementations outside the auto-wcag community

Wilco: not sure?

shadi: I'm still hoping the PDF folks will write a few rules, maybe if they're not implemented
... they're looking into how to use these rules in the PDF checker
... I think we'll also need a rule that is a best practice
... we can have rules that cover SVG
... actually are these SVG rule supposed to work on standalone SVG?

Kasper: yes

Wilco: so the thing we need in addition to the auto-wcag process is a table with all the exit criteria
... that should be pretty straightforward

shadi: that other table doesn't have to exclusively point to auto-wcag

Wilco: ok, let's get into the exit criteria themselves

anne_thyme_: so what is "one rule", is it just a written up ACT Rule, or an ACT Rule *and* and implementation?
... this was confusing for people who reviewed this
... we need to clarify that, what we mean by "Rule" in this document

Wilco: the rules are just the document, the implementations are different

anne_thyme_: so all of these exit criteria about ACT Rules don't need implementations?

Wilco: right

anne_thyme_: OK, then I think we need to make that explicit

Wilco: are you in favor of breaking it down this way, or differently?

anne_thyme_: I'm good with either approach as long as we intereprete this the same way
... it could maybe just be in the introduction

maryjom: OK, I'll edit this

https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Exit_Criteria_for_Rules_Format_Spec#Limitations.2C_Assumptions_or_Exceptions

<anne_thyme_> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Exit_Criteria_for_Rules_Format_Spec

anne_thyme_: next bullet is "aggregation", I don't think it's in the spec anymore

[maryjom deletes it]

scribe: "Issues list": I agree there is a requirement to have this if there *are* issues with the list
... but that also means the rule is not good enough
... so do we want to explicitly have not-good-enough rules in the exit criteria?

shadi: it's also to prepare the future

anne_thyme_: so we need to prove that it can have an issue list

romain: can it be an issue due to the limitation of the current state of technologies?

shadi: it would probably be in the "Limitations" section

Wilco: [proposes an example]
... the "required owned element" rule could work
... we need to do some research

shadi: not all the rules need to be implemented, so that could be a rule without implementation

anne_thyme_: right
... another question is about "At least two manual testing methodologies where one of their rules is comparable to an ACT rule"
... if it's just comparable it doesn't need to have an implementation
... I would like it to be a bit stricter, to require an implementation

shadi: right

anne_thyme_: I don't think we can go to CR if we haven't provent it can be used in manual testing

Wilco: the potential problem is that there are at least 3 open methodologies
... and other internal testing methodologies, does that count?

shadi: yes

anne_thyme_: don't you need to make that publicly available?

shadi: I have to check that, but I don't think. If company X says they implement it it should be OK

anne_thyme_: OK, then I definitely think we should change the wording there

Wilco: the other thing that came up at lunch is that it wasn't clear how implementations for manual rules would work
... I'm thinking of adding an informal section that explains that in the spec

shadi: can you create an issue?

Wilco: I have it on my todo list
... KathyE I've heard from you that you need to add things on top of ACT Rules, I think that's a good case of an implementation
... a manual implementation of an ACT Rule is a test procedure that explains how to evaluate an ACT Rule

shadi: what's the difference between that an a code procedure?
... for instance the rule "image is not decorative"
... someone starts programming the rule / someone describes how to manually evaluate that
... at the end of the day it's the same process

romain: are you saying that we don't need exit criteria specific to the manual implementations?

shadi: no, I don't think we necessarily need a specific section in the spec to describe manual implementations

Wilco: I think this section is worthwile because it is not obvious, a QA person cannot just take the rule an evaluate it

shadi: OK, but then we could have two sections, how to implement this in code or manually

anne_thyme_: this relates to what we talked about yesterday, a section that briefly and non-normatively describes what is a rule and what is an implementation

Wilco: ok, so we should do this sooner rather than later

anne_thyme_: I also have a question abou the "more than two" expectations

Wilco: the issue was that it was redundant, we should have "one" and "more than one"

anne_thyme_: the other issue was about the formal selector syntax
... I feel that no one uses it

Wilco: let's pick that one after coffee

[coffee break]

<shadi> scribe: Shadi

CSS Selectors

Anne: seems like a leftover from when Auto-WCAG used to use CSS Selectors

Wilco: anyone want to keep it?

Shadi: is it normative?

Anne: yes, a MAY requirement
... but you can also add anytime

Romain: how about changing it to a note
... because it contextualizes the meaning

Anne: we made the observation that CSS Selectors are often different from the plain language descriptions

Wilco: none of the rules we have now can be addressed with CSS Selector

Kasper: there are some, like unique IDs

Wilco: yes, minor stuff mostly
... suggest we remove as requested

Romain: don't feel the language is wrong
... think just MAY sould be removed
... but won't object to removing

RESOLUTION: remove MAY requirement on CSS Selectors

Identifying Best Practice Rules

Wilco: object to referring to requirement in section "Accessibility Requirement" for best practice rules

Shadi: the proposal on the table is to have another section, for example "related requirements", to relate best practice rules to requirements

KathyW: how about tagging rules, like "images", "tables", or other categories
... this way we can tell where a rule belongs

Anne: like the idea very much
... but not sure if this is for the Format spec
... versus for the UI of the rules repository

KathyW: can use existing tag-sets

Anne: one advantage of listing good practice rules, is that they may become requirements in the future

KathyW: people trying to do too many things
... need to understand what is required
... if we list the sucess criteria, it would add confusion
... we use tags in ARC

Shadi: initial comment from Glenda was to make it clearer when a rule is non-required

MaryJo: could have a placeholder "non-required" in the "Accessibility Requirements" section when there is no requirement

Eric: "non-required" or "required" doesn't say much
... need to say "required for WCAG" or "Silver" or "German standard" etc.

Wilco: suggest to turn that around to say "if not required by the latest W3C standard, then needs a flag that it is not required"

Anne: what about another standard, like RGAA?

KathyE: relates to discussion on mapping rule
... if required or not is also determined by mapping
... but in addition, identify what is good practice or not

Romain: yesterday we agreed to change the requirement for composite rules
... can now be much more granular
... so rules can be required or good practice depending on context
... makes the case to have the mapping externally

Wilco: so remove "Accessibility Requirement" section?
... suggest we do not address this any further than we already have

Romain: language of that section is unclear anyway
... for example "impacts the conformance"

Anne: still think linking to requirement with a flag that it is non-required for conformance works best
... this way, people can link to say RGAA but would know it is not rquired

MaryJo: but people may misunderstand that

Shadi: don't understand how leaving it blank is clearer than saying it is non-required

Anne: agree, would be sorry to see such information go from ACT
... would want to add it anyway
... also Norwegian government wants this
... see it as a presentation issue

Eric: in the end it is the call of the implementer to say what is best practice
... so maybe easiest to keep as-is

Shadi: it is the rule writer that decides the conformance relation, but implementers may choose to override

Anne: how about reusing the wording from AGWG
... saying "Advisory" and explaining that it is not required

KathW: seems easiest to reuse the wording of AG
... together with the mapping

<yatil> Rule X

<yatil> --- WCAG 1.2.3: required

<yatil> --- ABCD 9.8.7: not required

<Jey> https://github.com/auto-wcag/auto-wcag/blob/master/_data/wcag21-en.json

Romain: how about calling it metadata, and explaining it is default metadata that can be overridden by others

Jey: can help people insert correct mapping by using compiled list

Wilco: many people don't understand that W3C terminology

Anne: not sure we are getting anywhere
... could experiment with tags
... but referring to requirement is important

<anne_thyme> Anne's suggestion: “For [my standard] this rule in itself is not a requirement” in Accessibility Requirements section for best practices

Eric: think the proposal of adding label works best and I’m concerned that if the rules format would otherwise only allowed to specify tests that affect conformance.

KathyW: there is benefit of tying it back to WCAG

Wilco: my objection is only about putting this into the section "Accessibility Requirement"

Kasper: maybe the word "requirement" is the issue

KathyW: maybe atomic rule that relates to a requirement but is not required for conformance
... rule is often an input to a requirement but not a requirement in itself

Shadi: so have more or less the same section, together with mapping, only called "related requirement" or such

KathyW: think we should go back to the metadata idea
... see what we need in there
... like which SC it relates to
... how it relates

Wilco: need to know what conformance rules are

KathyW: will have that
... also the mapping

Anne: also want to know if it is a full or partial rule

Shadi: can we have a proposal for that as a PR?

MaryJo: some standards will make things required that are not

Wilco: concerned with replacing "Accessibility Requirements" section

KathyW: not removing it
... making it clearer when it is a requirement or not

KathyE: suggest renaming to "Accessibility References"
... with the same sub-sections
... and adding a third sub-section for "good practice" or whatever we call it

Anne: think we now have at least three types of mapping
... partial vs full, conformance relation, and good practice

<anne_thyme> https://github.com/w3c/wcag-act/issues/290

<scribe> scribe: Raf

<scribe> scribenick: Raf

<maryjom> https://github.com/w3c/wcag-act/issues/290

Cathy: If you wanna be more generic?

Anne: Are were looking for a composite rule or for a rule which is also a full test?

Marryjo: I think we are looking for both cases.

Cathy: Should we try to keep it with the same Sc?

Anne: Perhaps we should use the same that we used before?

Cathy: Best practice for headings may be confusing.

Anne: We took the wroding from an understanding document.
... Let's look for page titles. An atomic would be "the page has a title".

Shadi: And what would be a non conformance?
... My understanding is that it's a conformance requriement.
... My understanding is that it's a conformance requriement.e

Kasper: We have composite rules which are not full test for requirements, but test alternativew ways of not failing the requirement.

Maryjom: The full test for the requirement would actually be composite composite.

Kasper: Yes.

Anne: The composite composite could then be video object elements that videos are also tested?
... So the type of the requirement would be that all videos have captions.

Cathy: The best practise is that all pages have unique titles.

Maryjom: What about the usecase where you have different standards? Some will require this and some will not requrie thic conformance?

For example best practice.

Anne: This is a new usecase that something that is a requirement one standard is not what we even want in our standard.

It relates to earlier discussion about different standards.

Maryjom: We want a rule not for WCAG, but also for other standards a well.

Cathy: There is another example. There should be one title per tage.

Maryjom: I see that this example goes under comforming HTML markup.

Anne: All page should have a unique title. Is it a usecase we should consider?

Shadi: This is a good practice for input? [missed something].

<shadi> if a rule would be developed according to a sufficient technique, then passing that rule would indicate passing the associated requirement and failing that rule means more testing is required

<shadi> ...this is the opposite to typical rules that are according to failures, where failing that rule would indicate failing the associated requirement and passing that rule means more testing is required

Shadi: Let's say that the best practice is "there is only the one title on the page".

Cathy: For us the results of best practice would influence our results.

Anne: Proposes listing various requirements.

Shadi: These requirements change.
... Sometimes it is meeting a criteria and some time it is even not a good practice.

<shadi> [[

<shadi> Accessibility requirement: <reference to an accessibility standard> | Related provision: <reference to any testable statement>

<shadi> Pass condition: <satisfied | further testing needed>

<shadi> Fail condition: <not-satisfied | further testing needed>

<shadi> Inapplicable condition: <satisfied | not-satisfied | further testing needed>

<shadi> Conformance rule: <yes | no>

<shadi> Partial test: <yes | no>

<shadi> ]]

Shadi: You hav either a requirement or a related provision.
... Either this or this.
... Those three lines after the accessibility requirement are the mapping.
... We need to add actually one more section. We have all the mapping. What we need is just this Related provision section.

Anne: It's important that we have a mapping for each requirmenet/reference.
... We should describe this is why this rule refers to this requirement.

Romain: Do we need to define how the organisation can change the base for the rule?

Shadi: We have the most open license here. You can take a rule and implement or reimplement it.

Anne: It woul be nice if we could see if the rule is accepted in various standards.

Shadi: It could be a reference to extension table.
... Another model could be to go with it as it is.

Kathy: Would it make sense at this point to just have a WCAG requirement and other requirement?
... So that it's easy to see what is WCAG and what is not.

Shadi: This would require extensibility.

Kathy: I would think we would reuse what already is WCAG.

Shadi: There are two ways to do that. One is to download this rule and create your won. And then we have two rules - original and changed. And Romain's proposal is the mapping at the table outside.
... The good thing of the external mapping is that people see these rules and they can be checked if they are used.
... Let's not try to predict which of the three models will prevail.

Romain: I don't like related provisions. I like the concept of reference.

Shadi and Maryjom: Accessibility reference is good.

"Accessibility reference".

Romain: Do we want to label atomic rules?

Anne: That's where we need Accessibility Reference.

Romain: there may be very small atomic rules which refer to different criteria.
... There is also a case which accessibility requirement has a presidence in case of composite rules which consist of composite rules.

Maryjom: For the highest composite rule...

Shadi: It should be made clear in the spec that you have to have mapping for every accessibility rule in the list. What do you think?

Anne: Could be an accessibility reference just be WCAG 1.1.1?
... Two atomic rules: using standard key to leave keyboard trap and using non standard key, which requires additional description. Both should refer to a composite rule.

<shadi> trackbot, end meeting

- DRAFT -

Accessibility Conformance Testing Teleconference

26 Oct 2018

Attendees

Contents

Exit Criteria

CSS Selectors

Identifying Best Practice Rules

Summary of Action Items

Summary of Resolutions

Scribe.perl diagnostic output