W3C

Accessibility Conformance Testing Teleconference

23 Oct 2017

See also: IRC log

Attendees

Present
Anne, Wilco, MaryJo, Tobias, Romain, Charu, Moe, SteinErik
Regrets
Chair
Wilco, MaryJo
Scribe
Romain

Contents


Doodle poll for new meeting time https://doodle.com/poll/668bk8xeyqyxpg4t

wilco: I think that this time isn't working for everyone. Please fill up the form
... looks like Thursday is currently the best option

Consistency: Possible outcome values - suggestion to use EARL terminology https://github.com/w3c/wcag-act/issues/117

wilco: right now, we've got this weird thing called "undetermined" in section 8.2
... it's not part of EARL, we made it up ourselves
... first question is do we even need this concept?
... we answered that last week. next quesiton is should it be "undetermined" or "canttell" as is used in EARL?

Stein Erik: in my perspective, OK to use EARL's technology

wilco: sounds right to me

Stein Erik: we said we needed a dinstinction between "inapplicable" and "undetermined"

Charu: I think we add a discussion earlier, trying to remember why we didn't kept "canttell"
... we have "untested", and "not fully tested". isn't that the case when a fully automated tool isn't able to figure out the resulsts?

wilco: there's a bunch of terms, we just have to pick one
... another way we could use is to have an explicit mapping to EARL
... "canttell" may read a bit weird, which is why I think we used "undetermined"
... but I do personally prefer sticking with EARL terminiology

Stein Erik: either way we need a definition

wilco: I think the reason the EARL group came up with "cannot tell" is that it's not technology specific

Charu: when we get the "undetermined" outcome it's that it's not fully tested. can we use "not fully tested"? we'd need to map that to "cannot tell"

wilco: are you OK sticking with "cannot tell" and put a definition in our document?

charu: EARL's definition is not very specific, so if we have our own definition it should be OK

wilco: ok, let's stick with "cannot tell", we'll work on a definition

anne: ??? [bad audio]

wilco: I think the reason we put it in there is that it's not a result from a rule. An aggregation can be "inapplicable"
... it's not described explicitly in 8.2, you can find it in appendix

anne: we can think of cases where we can have "inapplicable" for the rule itself. we have tests like that

wilco: example?

anne: one where we start out by searching a constant and test if it's included in landmark. if it's not there, the rule is inapplicable
... we do have tests where for us it would be necessary to have inapplicable as an outcome

Stein Erik: [clarifies the use case]

anne: we don't start out from a CSS selector

Stein Erik: what you need to test is a parent element instead of searching all children elements

scribe: you pass a parent, but then figure it's not relevant when testing the children

wilco: are you saying one test could impact the result of another test?

Stein Erik: yes, which may already be possible. it depends on how you split tests into steps

wilco: maybe you can explore this further and come back to the group
... hopefully the time change will allow the developer to attend our calls as well

anne: this was one of the results from our internal testing

wilco: [musings about the format not requiring CSS selectors]
... the conclusion stands, use "canttell" and add a definition for it, better than the one in EARL

Avoid hard statements about rules not being automated https://github.com/w3c/wcag-act/issues/120

wilco: we may not be too explicit about which rule may be automated and which may not, technology is evolving
... I kind of agree
... I had several people asking that explicitly, to make it easier to undertsand one or the other might fall
... it's more "we think it's this one", not a necessity

stein erik: can you give an example?

wilco: this is mentioned in 5.2

stein erik: OK. I think it's a valid point

wilco: we could either remove it altogether, that doesn't mean that the test mode can't be part of the eventual result, it's left up to the implementor
... not even sure it's required by EARL
... the other option is to have a suggested test mode

charu: I like that

+1

charu: in my thinking, if you have some tests that used to be manual but somebody found a way to automate that, the rule cam be revised

wilco: you don't much care how you implement it as long as you get to the same results

stein erik: makes sense

<Wilco> Decision: Update 5.2 to say "Suggested test mode"

stein erik: is it even necessary

<MoeKraft> I prefer recommended too

rdeltour: maybe "suggested" mean "recommended"? so "indicative" or "informative" is more appropriate?

moe: I just think "recommended" carries more weight.

wilco: what about "indicative"

moe: somewhat vague. indicated by whom?

rdeltour: [clarifies]

<MoeKraft> defined maybe

stein erik: maybe "this rule has been designed or tested using this mode" ?

scribe: something like "known test mode"

wilco: what about "default test mode"?

stein erik: I like the idea of default.

moe: I was thinking "defined"
... you would need to define what the default is
... in the rules format, we have a specified mode

charu: I was more leaning to "indicative" or "specified"

stein erik: isn't it very decided in a way? very firm?

<MoeKraft> 5.2 states: Test case mode identifying whether test case steps are automated, semi-automated or require manual testing.

anne: to me "specified" suggests that you don't follow the spec if you don't do this.

stein erik: "proposed"?

wilco: +1

charu: +1

<MoeKraft> default: a preselected option adopted by a computer program or other mechanism when no alternative is specified by the user or programmer.

+1

<maryjom> +1

<MoeKraft> proposed: put forward (an idea or plan) for consideration or discussion by others.

wilco: pretty easy one I think
... are Rules a 1 to 1 relationship to SC?

stein erik: isn't it explcit in the text that they can be 1-1 or 1-many?

charu: yes, we have rules like that

anne: yes, in section 3.3 Accessibility Requirements

<Wilco> https://github.com/w3c/wcag-act/issues/135

rdeltour: [describes his comment on #135]

moe: we were trying to replicate WCAG verbage

all: [debating an appropriate rewording]

wilco: what about "An ACT Rule MUST be …"

rdeltour: what about then "An ACT Rule is a complete or partial test…"

wilco: sounds ok to me

<MoeKraft> An ACT Rule MAY test only part of an accessibility requirement, or MAY cover more than one requirement.

moe: there are several places where we use MAY, should we replace all of them?
... ok, let's just focus on this occurrence for now

<MoeKraft> Conformance requirements are expressed with a combination of descriptive assertions and RFC 2119 terminology. The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”, “SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in the normative parts of this document are to be interpreted as described in RFC 2119. However, for readability, these words do not appear in all uppercase letters in this spe[CUT]

<Wilco> https://github.com/w3c/wcag-act/issues/139

rdeltour: [rephrases what he put in #139]

wilco: kinda like the idea, helpful to get the full picture. Moe WDYT?

rdeltour: the idea is to have a section dedicated to describe the logic of evaluating a rule
... rather than have it conflated with the Rule description
... like in the "Test cases" section, which starts with statements on thte rules description itself, but ends with describing the logic on how the rule is processed an resutls aggregated

moe: romain, you're self assigned, do yo want me to wokr with you?

romain: sure, happy to propose something, I can try before TPAC

wilco: sounds fine

<Wilco> https://github.com/w3c/wcag-act/issues/133

rdeltour: not even sure if we want each rule to be universally identified or not?

wilco: in auto-wcag we're using string IDs that we can put in URLs

rdeltour: for isntance, I'm developing a rule set, am I allowed to use "image-description" as a rule ID?

wilco: one solution is to mandate a URL

rdeltour: or just be permiissive and state that we allow anything

anne: in my experience, we'd have question on what is the fornat for this identifier

rdeltour: +1, we could have a clarification that defining the identifier is left to the implementor

wilco: we can add a note

rdeltour: +1

wilco: I think we're done!

Summary of Action Items

Summary of Resolutions

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.152 (CVS log)
$Date: 2017/10/25 14:50:02 $