Wilco: This group was started as a community
group to discuss and develop rulesets for testing WCAG compliance.
... We want to get to a more formalized solution to the problem of needing a
common test ruleset.
JB: Let's do some introductions.
... Is the director of WAI. There have been a lot of requests for this topic.
Really want to thank Wilco for hosting this topic, and those who are here for
their interest..
JG: Deque systems a11y expert.
Word at FTB (Resarch Institute Technology and Disability) in Germany. Has been involved in this interest group.
MM: With IBM and has a great interest in the group.
CP: Charu Pandhi, works for IBM.
JF: John Foliot, works at Deque. Also involved in what the next WCAG steps are and looking for interlock between this and other activities.
KH: Katie, works for Deque. Has interest in QA and tools and thinks this is very important.
Makoto: From Japan, works on the JIS standard which is the Japanese translation of WCAG 2.0
Wilco: Works for Deque, leads this group.
<Wilco> 2. Problem outline:
Wilco: Auto-WCAG started as an EU project to
determine a way to monitor website accessibility across Europe.
... A lot of different rulesets out there, and some reported compliance
failures using checks that were really best practices, not specifically WCAG
2.0 standard
... We are trying to come up with a common ruleset that checks WCAG 2.0
compliance.
<Birkir> WEll with auto-wcag we hope to maximize the amount of a11y testing that can be done without the expert. There will still be place for experts, because software can't catch everything, but we beleive there is still room for software to be smarter and catch more issues.
JB: WAI has been hearing a strong call for
normative, authoritative test procedures. Many organizations work hard on
accessibility of a product or service, and in the purchase of that tool an
evaluation tool is used that reports failures that aren't always WCAG 2.0
... There are many different web technologies and techniques, but no overall
normative way to test. This is sometimes causing questions and conflicts.
... So Shadi and Judy looked at options to solve this issue and saw this group
as a good candidate to carry forward this work.
... There is an initial focus mostly for automated test, but as time allows
semi-automated and manual test can be addressed as well.
KH: This is really a problem for us every day, not just sometimes. So very glad we will address.
<Birkir> I have an entire CSUN presentation on whether elements should be links or buttons, and what accessibility failures can be called on that.
+1 to Katie
<cpandhi> +1 to Katie, this is very much needed
DB: There are a lot of false positives with different tools and it causes distrust among developers of the tools we use. There is a lot of interpretation and variability causing this distrust as well.
<Judy> scribe: JB
<Judy> MJM: definitely at IBM we have had similar problem as others describe
<Judy> ...for instance when we dev a product or service, our tools test it as compliant
<Judy> ...but others' tools take something that was informative, and consider it a requirement
<Judy> ...we've had problems with many testing approaches that have given failure outcomes
<Judy> scribe: maryjom
Jesse: Any kinds of errors a testing system raises that create false positives cause issues especially in continuous integration.
JB: We have been getting requests on this topic
for the past 4-5 years. We didn't start something formal earlier because the
scope was extremely daunting.
... We are getting to a point now that at least the automated portion. We want
to ensure the confidence level in automated testing is there.
JN: We want to make sure we frame this that the automated test is not the full testing required for WCAG 2.0 compliance.
We hope that when we have better automated testing that it becomes integrated into the regular QA tools and then we can get more QA people involved in accessibility.
JG: John Gunderson - University of Illinois, involved in automated testing. Question, will we only work on pass/fail rules?
WC: Wendy Chisolm, Microsoft - is working on automated test for Microsoft.
JN: James Nurthen, Oracle - develops most of internal tools. Makes sure not to have Pass/Fail rules unless it is an absolute.
<Wilco> 3. Stakeholders:
<Dylan> Awesome group of participants
Wilco: Glad there is such a broad group of
participants today. The group has been working on this and is glad there will
be increased participation from additional organizations.
... How can your organization contribute?
KH: Now with HML5, ARIA, etc being taken up the complexity of testing is increased, so the automated test is important.
JB: Interested in finding out who can contribute to the work of this group - rules, etc. Some organizations have made their rules open, and we hope the focus will be on normative rules and get confirmation of contribution of rulesets to help move this forward.
Wilco: What process should we use to get the work done through the W3C?
JB: W3C processes and practices are evolving. If
people think it would be useful to have a recommendation or standard that is
normative and authoritative, then that work would need to be done in a W3C
working group.
... Work can be incubated prior to formalization of a working group. This
community group can elect a path they prefer to use to get the formal work
done. Resources from W3C staff and tools is less for community groups, so it
may slow progress of work.
Wilco: Interested organizations should join the auto-wcag interest group to start with.
JN: Doesn't want to join the group until we know the work will actually happen.
JB: It is expected that the work will have to incubate with this community group for 2-3 months prior to moving on to a working group.
JF: This sounds like it could be a task force. It might be an easier path forward.
JB: Is happy to help us explore the possibilities. Some organizations may have a more difficult time joining a task force than a working group or community group.
Wilco: Perhaps the more formal work that would be
proposed as a standard could be done under a task force, and the informal work
remain in the community group.
... We'll have to also come up with exactly what we want to be standardized.
<Ryladog_> +1 Katie to participating
Wendy: Won't have bandwidth to participate more than monthly. Will be following the work closely though.
<annika> I have to leave now. I think the work and ideas discussed today have great potential. I'd like to join the discussion onwards. Bye.
Jeffrey: Interested in coding aspects. Difficult to join a working group. Working in an open-source project is easier.
Wilco: Potentially the community group can have an open repository to contribute.
<Birkir> I have to drop off for another meeting. I'll continue to work with Wilco and the group to move this work forward.Thanks all.
Wendy: A comprehensive suite of passes/failures is out there, but the different libraries need harmonization and a key piece is to have harmonization between them.
JB: We are interested to see what different
groups are willing to contribute as far as rulesets. There will also be a need
to cross-vet the checks, such as a unified description of what is covered in
the rulesets to come up with a common test approach.
... There may be some parallel work to be done in a task force vs. what is
done in the community group. So the facilitators will have to sort that out so
the participants aren't spread too thin and there isn't duplication of effort
between the groups.
Jesse: How the tools take data in and how they
return the data could be standardized, a standard test signature, so that if
there are different repositories of rules they can all run on the test tool.
... Deque has a good number of tests, and QUAIL also has many tests.
Wilco: We have been developing test rules and use them in testing to vet them. But we want to focus on a standard set of rules and a solid foundation to run the rules on top of.
Dylan: I agree, non-false positive, standard rule set is very important. Would like to also emphasize the standardization effort has to be independent of the implementation framework in which the rules execute.
<Judy> Judy wants to clarify something
<jessebeach_> I think I just got recruited
JB: The W3C process has a stage that is candidate
recommendation which is to have independent implementations of features that
can be across a number of tools. The candidate recommendation period is
designed to catch issues like the one described by Wilco.
... We leverage the implementation across actual tools to validate the
standard.
<jongund> +1
<jessebeach_> Totally agree with James
<jessebeach_> Let us code fast
JN: Test cases for each of the rules to validate the rule implementation need to be part of the process.
JG: The test cases are the most valuable part to help determine what actually is going to be tested and ensure it is done correctly.
<Dylan> I have to leave
CP: +1 with everyone. Another criteria for standardization is cross-browser support.
JB: One important initial decision is whether there will be a standard developed. A working group in W3C is used to create standards, so the group will have to decide upon the desired end-product to determine how to containerize the work.
KH: Supports creation of a standard.
Wilco: Thinks there will be a standard, and then rules that are standardized to sit on top of that.
KH: Thinks the rules should also be standardized.
Doesn't think we need to say we've created the "complete set", but definitely
thinks that a set of standard rules is important.
... It wouldn't have to be a complete test. The sufficient techniques need to
be able to be grown, so we'll need a model to follow with a group of absolutely
required tests, and some that are recommended.
MM: There should be layering of tests. There can be some normative parts that are always required, and some that cover options that can be used to meet criteria but aren't absolutely required always.
Wilco: Next meeting should probably be scheduled for 31 March and we could meet every 2 weeks.
JB: Thinks meeting every other week will help progress the work forward.
Wilco: Open up for final thoughts.
JB: WAI wants to support this group's efforts, and is excited to see the participation.
CP: Happy to be participating in the group.
MM: Glad to see such broad participation. This is important work to IBM and obvious it is important to others too.
Makoto: Will be working to determine how he can participate.