WCAG ACT TF Face-to-Face

Wilco: from Deque, co-facilitator of this TF

Kathy: from Interactive Accessibility, co-facilitator of Mobile A11Y TF

Alice: from Google, working on UI

Shadi: W3C staff contact for this TF

Song: from NIA, Korea

John: from Microsoft

Romain: from Daisy

Michael: W3C staff contact for parent WCAG WG
... focus on interfacing with WCAG WG

<rdeltour> ACT Overview: https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Overview_-_What_is_ACT

Wilco: many tools out there, each slightly different, conflicting results
... that's a problem
... want to figure out, how to get to a core set of rules that we can all agree on

Michael: often get questions, can't provide support requests
... sometimes authoritative review needed

Kathy: sometimes also questions on coverage of WCAG

Wilco: impression that previously such discussions were avoided

Michael: when we get a request for interpretation, often it is one that the group agrees on
... when not, we look into why
... sometimes unclear with different history of how requirements were created
... often documentation not available anymore

Wilco: would try to interpret to take the load off the WG

Shadi: so we bring results of our discussion to the WG?

Michael: only in areas of dispute, if existing supporting resources are not sufficient

Wilco: difference between success and failure?

Michael: failure is direct result - if there is a fail, requirement is not met
... to prove success, not meeting a technique is not exhaustive
... could meet it in a different way

Wilco: we assume accessible unless a failure is found

Michael: might not be a common approach

Wilco: accessibility support is also an open issue
... in aXe we check if implementation is viable
... and if there are alternatives

Kathy: how do you keep that up-to-date?

Wilco: this is exactly the problem description

[Wilco walks through "Goals" from ACT Overview page]

Wilco: rules will not be exhaustive

Shadi: but developed tests should be authoritative

Kathy: warning about calling it conformance testing
... can test requirements partially but often not fully
... difficult to prove conformance
... already confusion out there, need to be careful about wording

Romain: need to be careful with the word authoritative as well
... not exclusive, tool vendors can develop any rules too

Shadi: agree, was looking for the issue of validation
... somehow recognizing the rules that have been validated for interpretation of WCAG

Kathy: conformance has very specific meaning in WCAG

Shadi: just wanted to make sure there is the tie-in to WCAG as a specification
... rather than accessibility testing in the wild

John: layer of conformance - browser, platform, content?

Wilco: it depends

Kathy: even absolut failures are fairly rare

Wilco: not sure I agree with that
... there are always exceptions
... like conforming alternative aspect
... but that is caught in semi-automated

Kathy: ok, as long as semi-automated

Shadi: there is this perception that it is all about automated only
... but in fact, even most auto-WCAG tests are actually semi-automated

Kathy: confusion out there about what is conformance really
... most focus is on the success criteria, rather than the actual conformance requirements
... we propagate that by calling it conformance testing or automated testing
... maybe one goal of this group is to help define what is needed for conformance testing

<Zakim> MichaelC, you wanted to note conformance testing only needed for conformance claims, which aren´t required

Wilco: one of the discussions with WCAG WG, also in light of Silver

Michael: one of the requirements for WCAG was to have conformance claims

Kathy: conformance claims becoming very important in the US, for example
... seeing more and more the need

Wilco: conformance testing important in the Netherlands

Shadi: take away is that we have an issue with the naming
... suggest to take that back for later discussion in the group

Wilco: had previous discussions on that

Kathy: agree, difficult problem but needs to be discussed

[Goal #4]

Wilco: not limited to HTML
... also CSS, etc
... even UAAG

Shadi: is it a scope creep to include browser testing beyond content testing?

Kathy: think it would be important to keep in mind

Wilco: primary focus HTML, CSS, etc

Boaz: is the framework exclusive to accessibility testing at all?
... possibly shareable beyond accessibility alone

Kathy: agree with that comment

Wilco: would not want to put a hard boundary to UAAG
... everything converging

Kathy: agreed, also in light of Silver

Shadi: convinced, just worried about scope creep

Romain: digital publishing technologies not separate from web technologies
... ePub uses web technologies
... our selectors are pretty much focused on web technologies

[Target Audience]

Kathy: how to treat false positives?
... only rules that don't create false positives?

Romain: think so

Wilco: would say TBD

Romain: could be goal to reduce false positives

Shadi: do we want to differentiate false positives and false negatives, or just talk about accuracy?

Romain: false positives can often block from publication for no reason

Kathy: some tools will suggest additional code to fix a false positive

Wilco: agree need to look at accuracy

Kathy: can't eliminate false negatives in semi-automated

Shadi: depends on atomicity of tests

Romain: initial rules exclusive to HTML, CSS, and ARIA?
... what does that specifically mean?

Wilco: we're on the Initial Rules section of the Overview https://www.w3.org/WAI/GL/task-forces/conformance-testing/wiki/ACT_Overview_-_What_is_ACT
... Romain said it sounds like the Rules development is in scope of ACT
... let's change that

Romain: what's the ACT role? Overseeing or coordination?

Wilco: we should coordinate with auto-wcag
... **changes the section accordingly**

Kathy: we wanted to talk about whether we need to further define "HTML, CSS and ARIA"

Wilco: I think we don't need to be more specific
... we don't want to promise more that what we're doing to do

Romain: do we want to mention the EPUB rules
... we will develop a rule set specific to EPUB, ultimately based on the ACT fwk
... I'm not pushing for mentioning it here, but it might help with the 2-implementation req for the CR

shadi: we don't want to boil the ocen right away
... but I'm wondering if we should rephrase the second part "These rules will focus on..." to be more inclusive

Romain: what about saying sth along the lines of "and coordinate with any other community interested in developing rules for the ACT fwk"?

Wilco: I want to be explicit about Auto-WCAG

Romain: +1

Shadi: ok for Romain's proposal, with s/any other communry/other groups/

<shadi> +1

Wilco: "Use Existing Rulesets"
... we want to coordinate with orgs that have rulesets, and help them transition to ACT

kathy: we should also look at orgs checklists, not only automated tools

Wilco: **opens the editor to improve the language**
... add "or test procedures"

<Zakim> shadi, you wanted to ask about "will work with"

shadi: "will work with" means we'll support organizations?

kathy: what about we change that to say we're going to use their rulesets

shadi: there's the aspect that we're going to look at their stuff
... then there's the question of how to assist people who want to migrate
... provide some kind of support or guidance in getting their rules in the right format

kathy: then let's say that explicitly
... "The ACT TF will collect the rulesets from organizations interested in contributing to the report, for the purpose of forming and evaluating the framework"

Romain: do we need the agreement from these organizations? we can just pick the rules

kathy: right, but some rules are not open, so those organizations need to explicitly contribute them

Wilco: we also want to say that the orgs can contribute their rules to Auto-WCAG

kathy: the second thing is to talk about the "specific acceptance requirements"
... do we need to mention the orgs that do not want to contribute?

Wilco: it was there to address people asking "why should I contribute?"

kathy: ok, we might need to rephrase that

shadi: "ACT TF will provided guidance to help organization migrate their test rules and procedures"

Wilco: we're supporting auto-wcag

shadi: auto-wcag is not a body that can help organizations, it's the other way around: organizations who want to contribute can join auto-wcag

wilco: **rewords the paragraph accordingly**

eric: we should use the present tense

sharron: you need to be persuasive

eric: we can list the benefits of contributing

shadi: I'm wondering if the promotion part should be in another document
... we use the words "ruleset" "test rules" "test procedures"
... are we clear about what these are?

sharron: do you say we need a glossary?

shadi: we might be able to use one term

eric: for me, a ruleset is scoped to the thing to test, like "ruleset for WCAG AA"

wilco: in our terminology, all of these are rules, whether automated or not

shadi: for me, rules almost means automatated, test procedure is more vague

wilco: a test procedure is part of rule

kathy: for our existing ruleset it doesn't really matter
... we're giving examples, which kind of defines what we mean

romain: it's an overview document, we may not need to be super-specific

<shadi> [[Rules is the overall term, Test Procedures is part of a Rule]]

eric: as someone from the outside, it might be good to define the terms

shadi: we should revisit the wiki pages and make sure that we're consistent

wilco: do we want to list the deliverables in this document?

shadi: yes, it fits well

eric: I like being able to learn this info, but it can be hard to keep in sync with the formal deliverables
... maybe we can just link to the deliverables doc?

shadi: the deliverables are quite stable

sharron: why are there 2 groups?

shadi: to maximise the likelihood of contributions

romain: you don't need to be a w3c member to join a CG

shadi: it's also a source of confusion, people do not always know which group they should join
... people are sometimes confused by the whole w3c structure: TF, WG, CG, etc

romain: the description of "Benchmark method" sounds confusing
... it's a description on how to test for accuracy

shadi: there's accuracy and validity, two different things

wilco: validation should be part of the ACT fwk
... when you write a rule you need to provide enough background
... how do we validate the rules that can go in the ACT rule suit is still an open question

shadi: I always thought that would be part of the benchmark method
... what else can go in the benchmark method?
... do we provided a litmus test for a specfic rule

wilco: yes

shadi: so it's only the accuracy check, not the validity check
... the validation needs to happen on a rule basis (or classes of rules)

<shadi> [[validity vs accuracy checking]]

romain: in my understanding the validity check is done manually when writing the rule.
... the benchmark tool tests only the accuracy

wilco: what if the validity changes later

shadi: yes, the AT can change, etc

wilco: rules will need to be updated as technology change
... it doesn't mean the rule was invalid
... but new technologies will need different rules
... a rule is valid given its scope
... this doesn't change over time
... whereas the accuracy may change depending on how AT or UA evolve

shadi: we get the question of validation very often
... how does it move to a contributed test to become approved
... we should describe it somewhere
... if one of the thing we need to think of

romain: we're talking about incubation
... do we need to formalize the process?

shadi: at least some description to give the rough idea
... we envision there will be a process for approving test rules
... it is kind of missing in the overal description

romain: if the benchmark is to test the accuracy, does it mean we lack a deliverable to describe the validity process?

shadi: not necessarily a deliverable, it can be another section in this list

kathy: WCAG has to review the work for the TF
... the TF has to define what the rules are, then the WCAG reviews the TF work

wilco: the ACT TF can get consensus on rules

kathy: there is a document on the WCAG wiki that's institution knowledge
... do we have WCAG people in the TF?

shadi: we didn't want to pull for the WG, but bring new people

kathy: having somebody in the TF will help with the amount of back and forth

wilco: I think it will be straightforward enough

shadi: we need to have bridges

kathy: I may be willing to help out
... it will make things faster within the TF, at least to identify the things that need to go back to the WG

wilco: **edits the deliverables overview**

romain: "results sufficiently accurate for (non) conformance testing to WCAG" sounds like a bold statement

shadi: reword into "collection of rules that have passed the validation and benchmarking requirements"

kathy: in the first bullet, I would remove "conformance". we're talking about test rules for accessibility testing.

<jcraig> https://webkit.org/blog/3302/aria-and-accessibility-inspector/

<jcraig> aboxhall: demo similar to jsbin layout

<jcraig> aboxhall: html and js visible, rending and run button with console output on the other side

<jcraig> aboxhall: describes code sample that will changes the label property of an accessibility backing node for a DOM element

<jcraig> aboxhall: runs voiceover with standard labels.... clicks run... console prints done. voiceover reads new label on the button.

<jcraig> aboxhall: describes similar layout of a slider example with an "onaccessiblesetvalue" event handling silder increment/decrement events

<jcraig> aboxhall: js fetches the node, adds event listener, receives the event (when VO increments the slider) and changes the aria-value/min/max, etc

<aboxhall> One more time with feeling: https://discourse.wicg.io/t/contributing-the-accessibility-object-model-specification/1702

<jcraig> https://github.com/a11y-api/a11y-api/issues

<jcraig> aboxhall: inspecting the lang drop down

aboxhall: ...

partial ax tree in chrome ax extension

showing both unknown role and aria-equivalent role in the inspector

<aboxhall> chrome://flags/#enable-experimental-web-platform-features

set the experimental flag in chrome settings for this tool

then you can request computed role and name (label)

<aboxhall> aboxhall@chromium.org

wilco: feedback on the requirements was pretty good
... Charu provided feedback

<shadi> https://lists.w3.org/Archives/Public/public-wcag-act/2016Sep/0003.html

wilco: if rules are pseudo code then not a rule - we have gone beyond this discussion and will not be write pseudo code
... discussion of the comments
... defining standard common standard - we will make that change
... maps to an underlying requirement - when you write a rule you should be aware of validation

Romain - sounds like requirement for the rules not the framework

Wilco: yes, it is the ACT framework

Romain: the TOC should be updated to reflect this

Shadi: agree with Romain

Romain: i proposed changes in my email

Wilco: we will get to that when we review the feedback

Romain: we may not want to spend too much time on this until we look at the proposed changes

Wilco: skip over some of the comments from Charu

Move to Romain's comments

Romain: many editorial changes
... see the email for details

<shadi> https://lists.w3.org/Archives/Public/public-wcag-act/2016Sep/0004.html

Wilco: want to be careful with scope

James: how about discovering failures

Romain: next comment

rule will not test the policy itself

Shadi: I don't think we should promote other standards and policies.

James: we would want checks for example for ARIA

Wilco: good for organizations to go beyond WCAG

James: we insist that all tables have captions

Romain: instead of standards, how about guidelines

Shadi: test rules for WCAG 2 and other standards
... says the same thing in the paragraph

Wilco: I will do some wordsmithing on this

Romain: next comment

Shadi: agrees with the direction

Romain: might be a bold statement. How about "The ACT TF aims to increase the adoption of sane accessibility testing principles."

Wilco: how about promote
... we want to make it easier for developers to come to a shared approach

James: change should to can

"similar to how the HTML standard is a standard on how to write HTML documents"

Romain: I see what you mean, but I would avoid the comparison given the very different nature of the two specs
... I would remove it

Wilco removes the part of the paragraph

Romain: /ACT Rules/The rules written in conformance to the ACT Framework specification (later referred to as ACT Rules)/

Wilco makes the change

Romain: s/, that can be used in ATTs or for QA testing of accessibility./. They can be implemented by ATT, or used as a reference when performing manual accessibility QA /

Wilco: earlier we said this was not for QA
... removed the word QA

Wilco: comments from Alistair

https://lists.w3.org/Archives/Public/public-wcag-act/2016Sep/0006.html

Alistair: not throw out all techniques wholesale
...some may be relevant
...techniques are what people use

Wilco: we should come with a proposal for the WG on relationship between techniques and rules
...think specifically failure techniques may be an issue in the WG right now
...relates to our discussion

Shadi: agree not throw away everything
...but think minority will overlap with what we need
...techniques have an indirection different from the rules
...but need to develop rules first then figure out mapping

Alistair: think use techniques as starting point
...techniques are what people try to implement
...rules should check for that implementation
...rather than to try to test everything possible

Wilco: what happens when people write rules for techniques that don't exist?

Alistair: people should only write rules for existing techniques

<Wilco> ACTION: Wilco adds techniques proposal to the agenda for next meeting [recorded in http://www.w3.org/2016/09/22-wai-act-minutes.html#action01]

shadi: I think someone should work on a proposal, then discuss it in ACT before involving WCAG

wilco: maybe setting up a couple paragraph to summarize our thoughts, I'll do that
... for the last 15 minutes, let's switch back to Romain's comments

https://lists.w3.org/Archives/Public/public-wcag-act/2016Sep/0004.html

Romain: let's look at the new titles proposal for section 2 "Requirements"
... current titles sound like reqs on the Rules, but this doc is on the framework
... I propose to reword the headings
... "1. ACT Rules should be clear" could be "Ensure Rules readability"

wilco: not sure about readability. what's readability vs clarity?

shadi: these are more for tool implementors, not for the general public

romain: proposed heading 2. "Ensure soundness"

wilco: what I want for the rules is to be valid
... I want the rules to get valid results for a requirement

romain: I thout req #2 was more about mapping a rule to a success criteria, not about correctness itself

wilco: what's the difference between validity and soundness?

alistair: I don't think you can "ensure soundness" without context
... you're trying to say sth along the lines of "non interpretational"

wilco: what I try to mean is "if you ask a11y experts, they would agree this covers the requirement"

shadi: in my mind you mean validity
... I'm wondering if we're not too high level
... the framework needs to have a description for assumptions
... it needs test procedures
... the outcomes needs to be such and such
... this is what the requirements should output

romain: I think you're too specific, we're not defining rules but the requirements for the fwk
... we need sth along the lines of "the rules should reference some accesiibility guidelines"

shadi: we're talking about headings, there is content in the sections

alistair: we're not developing sth new, other industries that do testing have similiar framework
... we should do a little research, find what's interesting in them, and start from there
... we don't want to reinvent the wheel

wilco: I agree
... with this I want to outline where do we need to put the emphasis

<shadi_> +1 to alistair - look around at existing frameworks to inform our requirements development

wilco: I also agree with shadi that we can be more specific
... do we want to dive into this even further?

alistair: we should look at other frameworks instead of losing time at word smithing

wilco: can you do that alistair?

alistair: OK

shadi: the req documents should describe what the framework should be
... here I think we're describing too much what the outcome of the framework should be

+1

romain: this is very meta, it's almost a req document on a req document

shadi: for instnace, will the framework support multiple languages?
... do I need to transform the rules into a programming language?

wilco: this is covered in req 2.1

alistair: I read this as rules being expressed as plain english

shadi: so this whole section means the "test procedures must be expressed in plain english"

alistair: yes

shadi: this document is for us, to be a reference to write the specification
... I think it needs to be a bit more specific than this, maybe less specific than this

^^ maybe less specific than what I was describing

<agarrison> ISO/IEC Directives, Part 2 - Rules for the structure and drafting of International Standards - has some useful stuff in it.

wilco: we can restart the requirement discussion in next week's meeting after alistair research

shadi: either we can have things useful by tuesday...

alistair: we may have done some research available in WCAG EM

https://www.w3.org/TR/WCAG-EM/

shadi: let's do some research by tuesday, then review wednesday in prep for the meeting

<scribe> ACTION: shadi to lookup relevant part of requirements from WCAG-EM [recorded in http://www.w3.org/2016/09/22-wai-act-minutes.html#action02]

<scribe> ACTION: alistair to lookup ISO documentation [recorded in http://www.w3.org/2016/09/22-wai-act-minutes.html#action03]