W3C

Accessibility Conformance Testing Teleconference

20 Sep 2018

Attendees

Present
Alistair, Romain, Anne, Jey, Trevor, Kathy, Moe, Shadi, SteinErik
Regrets

Chair
MaryJo, Wilco
Scribe
Moe

Contents


Issue 224 Exit criteria https://github.com/w3c/wcag-act/issues/224

<Wilco> https://github.com/w3c/wcag-act/issues/224#issue-338581348

Wilco: This is an updated list of everything we may or may not want to put in our exit criteria

Anne: My comments are from a long time ago

Trevor: Not sure why we need more than two expectations

Wilco: I made this very broad and thought we would cut it down in places
... Unless objections, I agree

Can we summarize Trevor's resolution?

Shadi: Also have comments on cutting down. I think General can be cut down to say that all rules have to conform to the specification. At least two rules at beginning is illusionary. No way to meet all combinations. It will be difficult to determine a specific number .Rules need to conform is sufficient
... In other sections we repeat confirmation requirement. This is are duplicate and unnecessary.
... We need to expand on technology. HTML and something else, e.g. PDF.
... This could show application of the spec on other technology. PDF working group has agree to write 3 rule categories, e.g. color contrast, just need to change the names to have it apply to PDF. Headings could also apply but may require PDF specific format.
... 3rd category of rules only applies to PDF
... Hoping other technologies, e.g. SVG, will also be interested in contributing rules

<romain> we intend to publish ACT rules for EPUB, fwiw

Shadi: Test cases do not have any requirements in the exit criteria. Need something for test cases, e.g. each rule must have at least 2 test cases

Wilco: I agree with general that we can simplify things. I will gladly trim down

Shadi: I think explicit is good but repeating conformance requirements is not necessary

Wilco: Technology: Because we don't explicitly talk about technology in the rules format, aspects under test at least two rules are included.Forces us to do HTML plus something else.
... Is this specific or should we spell out what aspects we should include

Shadi: Aspects are in a technology, e.g. DOM is HTML based, Would be nice to have something totally different, e.g. PDF or SVG, to show that the Rules Format is technology agnostic
... Need to demonstrate that

Wilco: Could add, "at least two different web technologies"

Shadi: Yes. That could work. PDF, SVG, ePub
... Anyone looking at different technologies?

Wilco: Auto WCAG is looking at SVG

Romain: We definitely will publish some rules for ePub

Shadi: How about for the CR?

Romain: When will this run?

Shadi: October? to January

Wilco: Testcases: I didn't explicitly include test cases but rules format requires test cases. Do we want to write this out?

Shadi: Yes. Just wondering why we are not writing this out

Wilco: Expectation and Applicability each require at least 1 test case. Ok, actually 2, pass/fail

Alistair: May we also do applicability too? 3 test cases. Pass, Fail, N/A

Wilco: 2 for inapplicable, 1 pass, 1 fail

Alistair: Why 2? Only need 1 each
... Atomic rule is not actually an atomic rule if it requires at least 2 expectations. When we talk about atomic rule we only talk about 1 expectation.

Wilco: Yes. Every expectation needs 1 pass and 1 fail.

Alistair: If we have more than 1 test case in a rule, it's not atomic

Wilco: Ya, I suppose so. Atomic means it cannot be broken down further.

Alistair: E.g. any video element or audio element...3 elements in a test. This makes 3 types of tests and reporting.

Wilco: If we break it down for expectation, the true false for each expectation does not tell you anything about the accessibility of the object. Does that make sense?

Alistair: No.

Wilco: Feel free to open an issue

Steinerik: Difficult to revisit issues when we are so close to publishing. It's been a long time since we introduced atomic rules
... That's be efficient in our feedback. If we think it's critical, then we should address

Wilco: Let's address in an issue

Shadi: What if it was called something other than atomic?
... If it's just terminology, I don't think it's critical to fix before the CR. If it's critical than we should fix before CR.
... Let's open a Github issue to address if it's critical or just terminology?

Alistair: I just think it's weird that atomic rules are looking at different things in this case but not a showstopper

<scribe> ACTION: Open a Github issue to address the terminology/definition of atomic rule containing more than 1 test case

<trackbot> Error finding 'Open'. You can review and register nicknames at <https://www.w3.org/WAI/GL/task-forces/conformance-testing/track/users>.

Shadi: Last point is that we repeat conformance requirements. We don't need to repeat bullets

Wilco: At least two rules that are automated, at least two rules semi-automated and at least two rules that use manual steps
... Do we need more rules from manual test sources like Trusted Tester?

Shadi: The WAI Tools project should be producing at least 5. We should have manual rules to show

Wilco: Do they map to existing technologies?

Kathy: I want to get a better understanding of what you are asking. Is this for a specific success criteria?

Wilco: At least one rule should look like a WCAG SC. It can be broken down further.
... Not proposing that any of these methodologies have adopted ACT rules but could to show that they could be adopted as ACT rules.

Kathy: I hope so. I think that is possible.

Shadi: Keyboard and Headings rules should exist in Trusted Tester

Wilco: Do you think this vague exit criteria is sufficient?

Shadi: I think so. I think it is okay to keep it open.
... Under expectations, I think the bullets are conformance requirements.

Wilco: Ok, removing them
... Benchmark. Not included. Expected to be removed
... MaryJo, How are the edits coming along?

Wilco: Is it just editorial?

MaryJo: No, we have new topics

Wilco: What happens next?
... Exit criteria, we need to do a survey
... Are any issues something we need to resolve before CR?

There are some questions around definitions

E.g. Test aspect and Test subject

Mary Jo: What's the criteria?

Shadi: We need to address all substantive issues before CR
... For example, Atomic rule. If this is substantivwe issue we need to address
... We should not expect comments that a feature is broken when we exit CR. This should be viewed as Feature Freeze

Mary Jo: I don't think there is anything

Wilco: I think there are some questions around definitions but we can work on these in CR

Shadi: Is this a MUST or SHOULD. Need to clarify this

Wilco: I will work on this before the next leaders meeting

Issue 264: Clarify definition of Aspect under Test https://github.com/w3c/wcag-act/issues/264

Romain: Each aspect must be discreet with no overlap. I have a concern with this requirement.
... We don't say what we mean by this. Would rather get rid of this requirement or make it informative

Wilco: Was there opposition to this idea?

Shadi: What are the options?

Wilco: Remove the MUST
... Each aspect Should be discreet and avoid overlap.

Romain: SHOULD does not work here since it is not normative

<shadi> "Aspects under Test are discrete with no overlap..."

Romain: What does it mean for an Aspect to be discreet?

Wilco: What if we do not make it capitalized?

Shadi: No you should not use these keywords
... Just leave it out all together

Wilco: Will this address the issue?

Alistair: What if we take this out all together? I don't think we have to say this

Romain: We have to think what we add for the reader with this statement. Practically will do the write thing. We cannot not enforce this. I think we should drop the sentence.

Steinerik: I agree we should remove the statement

<cpandhi> +1

Anne: My comments, Each Aspect must be discreet but we can just remove it

<shadi> +1

<romain> +1

<anne_thyme> +1

<agarrison> +1

<Jey> +1

<trevor> +1

<kathyeng> +1

<maryjom> +1

Wilco: Can we remove it?

+1

RESOLUTION: We will remove the statement

Anne: What about editorial suggestions?
... Lots of contradictions
... Let's switch the order. Which belong to composite rules
... Might be editorial but it also changes the requirements

Also note that I am making edits to this section based on Anne and Shadi's feedback.

Summary of Action Items

[NEW] ACTION: Open a Github issue to address the terminology/definition of atomic rule containing more than 1 test case
 

Summary of Resolutions

  1. We will remove the statement
[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.153 (CVS log)
$Date: 2018/09/24 15:08:18 $