Accessibility Conformance Testing Teleconference

15 May 2017

See also: IRC log


MaryJo, Wilco, Moe, Sujasree, Romain, Shadi, Chris, Charu
Wilco, MaryJo


Publication requirements https://github.com/w3c/wcag-act/issues/81

<MoeKraft> Latest draft: https://w3c.github.io/wcag-act/act-rules-format.html

Wilco: Publication requirements...any comments?

Shadi: Stein-Erik will follow up with me next week

<Wilco> https://github.com/w3c/wcag-act/issues/82

Wilco: seems to be strong emphasis on automation

what are people's comments on this?

MoeKraft: There is no specific call out on automation. Ruleset talks to it. Explain in scope.

<shadi> +1 to Romain

Romain D : QA testing and distinguishing between automatic testing and semi-automated , test tools in broader sense

<MoeKraft> https://w3c.github.io/wcag-act/act-rules-format.html#abstract

Shadi: I agree with Romain's thought. Testing tools in the broader sense

Wilco: May not even have to say for "tools" , rather for evaluation of accessibility

<MoeKraft> Currently the first statement reads: The Accessibility Conformance Testing (ACT) Rules Format 1.0 is a specification designed to harmonize how accessibility rules are described for automated test tools, and how test procedures are written for quality assurance testing.

Shadi: good to mention test tools, but could mention later on. Primary goal is to harmonize accessibility rules...."including" automated and semi-automated tools.
... Will take as a pull request and work on it.

MoeKraft: will open item on that for Shadi

Wilco: Do we want to include partial test results text into introduction as well? Combining different test results...abstract...

Shadi: A rule doesn't necessarily test one success criteria, could include many within one test

Wilco: Rule aggregation already covers that.

Shadi: this should be a heading of its own. Relationship between test rules and WCAG SC's. Or broaden criteria. Mapping between test rules and SCs. Needs own heading.

Wilco: Mentions we can talk to this in section 3.3.

Shadi: either in introduction or scope would be better, more general and not inside test rule
... use "Mapping" vs. "relate".

MoeKraft: will create ticket on creating section within scope on mapping between rules and success criteria
... will assign that to Wilco

Wilco: Talking towards the comment 2: Outcome "Cannot tell". Confusing on what that actually means. Might consider moving away from that outcome in regard to testing / auditing. Was part of EARL 1.0 , why did it happen that way?

Shadi: Cannot tell, in EARL 1.0 was more general. WCAG SCs are designed that they are always applicable.

If no video, there is no applicable content, so SC is met when no applicable content

heuristic tests and human input is needed...

Shadi: in 2.0 there is no such thing as Not Applicable

Wilco: when it comes down to it, human input is needed.

Shadi: several tests may be needed to accomplish the rule, and there may be cases where the person completed 1 of 2 tests and can't meet.

Wilco: ACT rules are supposed to be independent of each other.

Shadi: Is result always pass or fail ? Or are other results possible?

Wilco: yes, there are others than pass / fail.

Charu P: in IBM rules, it is a violation, clear cut pass / fail. Usually fail. Then possible violation, if more than one way to meet SC. If rule is looking for one technique....manual tests form should have submit button as an example

manually testing: video needs testing manually for captions

Wilco: needs review auditing issue type...these all somewhat relate to the cannot tell type.
... Do we want to use a different wording than "cannot tell"?

Charu: cannot tell should point to human decision required.

Wilco: more thought would need to be put into this.

<MoeKraft> I like "potential violation"

Shadi: example situations could be introduced in terms of "cannot tell".

<MoeKraft> or "requires review"

Wilco: What happens if we don't go with EARL terminology?

Shadi: subset can be put into place. But what is need? I.e. is outcome undetermined, neither pass or fail.

<MoeKraft> +1 Shadi's term "undetermined"

<Wilco> https://w3c.github.io/wcag-act/act-rules-format.html#output-outcome

General question from me: I.e. is undetermined not present and therefore not testable?

<MoeKraft> Sorry, my phone batter died. Just rejoined.

<shadi> "The result of the test is undetermined. This may be because human intervention or further testing is needed."

Wilco: Will ask Raph to clarify comments for 6.2. Will talk to variations of this.

Proposal for how to leverage existing testcases https://github.com/w3c/wcag-act/issues/76

<Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources

<Wilco> http://lists.w3.org/Archives/Public/public-wcag-act/2017Apr/0020.html

Wilco: Would like to start doing is have a look at Deque's repository
... can we update the test case repositories to work with this file format as well?

Charu: Take each test case and put this in this format?

Wilco: Put JSON file , pull into large repository

MoeKraft: to Charu: Karma test run...instead of JSON that takes rules to scan content. JSON for each test case. JSON used to describe rule format i.e. meta data

expected outcomes

<Wilco> https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/Testing_Resources#Test_cas_format_.28explicit_results.29

Wilco: Moe and Charu will do that at IBM and repositories will be talking to each other soon hopefully.

TPAC 2017

<Wilco> https://www.w3.org/2017/11/TPAC/

Wilco: to book hotel with group code
... feel free to join the AG working group call tomorrow if you want, as we have a slot during that call
... talk to you in two weeks

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2017/05/15 15:09:48 $