ACT Deliverables

Use the discussion page for comments.

This page is part of the (Proposed) Accessibility Conformance Testing (ACT) Task Force Work Statement.

Overview

(proposed) Accessibility Conformance Testing (ACT) Task Force will develop the following resources, to address the goals outlined in Accessibility Conformance Testing for W3C:

1. ACT Framework: A W3C Recommendation defining how to write rules for accessibility conformance testing. These ACT rules must give accurate indication of conformance to a standard. The rules are consistently reliable and compatible with each other, regardless of who authored the rule or which software product they are implemented in. The test rules may be common across technologies to verify WCAG 2.0 success criteria or may be specific to different web technologies, such as HTML, WAI-ARIA, EPUB, or others. The ACT Framework will set the requirements for any such test rule. It will be compatible with WCAG 2.0 and be applicable to other web accessibility standards, such as future versions of WCAG or company policies.

2. ACT Benchmark: A method, which outlines how to test the accuracy of an ACT Rule, using real world web content, and an implementation of this method as a software product. The ACT Benchmark Method might be a stand-alone specification (as a W3C Note), or part of the ACT Framework specification. It will combine automatic and manual approaches for testing ACT Rules. The software implementation will be built according to this method, to support test rules developers and the working group.

3. ACT Rule Suite: A set of approved ACT Rules that meet the requirements of the ACT Framework specification and the ACT Benchmark Method. This includes the individual files with the ACT Rules and the web-based front-end to manage these ACT Rules. ACT Rules will be added, updated, and deprecated on a rolling basis according to WCAG WG consensus decisions. The development and maintenance of the ACT Rules will continue in the open Auto-WCAG Community Group.

Note: All deliverables developed in Auto-WCAG CG will be overseen by the ACT Task Force. They are developed in Auto-WCAG CG to leverage community participation and contribution. All deliverables are overseen and approved by WCAG WG, the parent group of the ACT Task Force.

1. ACT Framework Spec

The ACT Framework will be a W3C recommendation. It is the framework on top of which conformance rules can be developed. Setting up this framework is the first stage of this process. ACT Framework will outline what ACT Rules are and how they relate to each other. The document outline will look something like this, addressing each of these topics.

• Introduction
• Manual & Automation: How user testing and automated testing come together in ACT Rules
• Rule Scope
  • Accessibility Standards: Rules for WCAG 2.0/Next, National laws, company policies, technology best practices, etc.
  • Web technologies: What technologies does the rule apply to.
  • Execution context: Where the rules are applied: Source files (linting), HTTP Responses (parsing), Browsers (Integration testing)
• Rule Structure
  • Description: What the rule tests
  • Background: Why this rule is in line with it's specs
  • Assumptions: Known limitations to the rule
  • Selector: What a rule tests.
  • Test steps: How the test is done.
• Output Format: How is a result structured and what should it contain.
• Result Aggregation: How results from rules and other accessibility tests can be aggregated to get the 'big picture' insight.
• Accessibility Support: Ensuring results of rules line up with an the intended accessibility support baseline.
• Rule validation: How to ensure rule does not cause (unexpected) false positives

Much of this work already exists in one form or another by the W3C, by Auto-WCAG, and by the Open Ajax Alliance. But further development will be needed, bringing everything together and coming to a shared approach. The following are existing resources that will serve as input for the ACT Framework

• https://www.w3.org/community/auto-wcag/wiki/Introduction_to_auto-wcag_test_design
• https://www.w3.org/community/auto-wcag/wiki/Test_design
• https://www.w3.org/community/auto-wcag/wiki/Basic_aggregation_algorithm
• http://www.openajax.org/member/wiki/Accessibility_Ruleset_Rule_and_NLS_Format_2.0
• https://www.w3.org/TR/EARL10-Schema/

2. ACT Benchmark

The only way to make sure that the assumptions that are made as part of rules, are valid assumptions, and that there aren't any assumptions a rule designer might have overlooked, is by using the rule, and comparing it's results to expert evaluations. Doing so will require the ACT Task Force to define a method on how to benchmark rules. An implementation of this benchmark can then be developed by Auto-WCAG, both to prove the effectiveness of the ACT Benchmark method, as well as for use in the ACT Suite (deliverable 3).

ACT Benchmark Method

The ACT Task Force will define a method for benchmarking ACT rules. This is intended to get hard numbers about the accuracy of rules. The following questions will need to be answered:

• What pages should a rule be tested against?
• How should the expected outcome be determined?
• How many results are required to determine the rule's accuracy?

The ACT Benchmark Method will either be published as part of the ACT Framework, or as a separate note. Further discussion is needed to determine which is the preferable approach.

ACT Benchmark Tool

The ACT Benchmark Tool will be developed by Auto-WCAG, and should serve both as proof of the method, and as tool to be used in further development of the ACT Rule Suite (deliverable 3). The exact details of this product can be determined once the benchmark method is decided upon.

This tool will be used to test that rules don't cause (unexpected) false positives. This will be done by applying an implementation of ACT Rules, and either having those results manually verified by an accessibility expert, or comparing the result to the findings from a manual audit of that same webpage. Doing so a number of times will give an indication of the rule's accuracy.

3. ACT Rule Suite

The W3C ACT Rule Suite will be a collection of rules that are proven to return results, deemed sufficiently accurate by the WCAG Working Group, that they can be used as a smoke test for (non) conformance to WCAG. In addition to this core set of WCAG rules, less accurate rule (useful as warnings), and rules for other accessibility standards and best practices will also be kept, though these are not actively benchmarked.

The Rule suite will be made up of a repository, holding a number of ACT Rulesets, including the WCAG Core Ruleset. The suite will also have a frontend that serves as a reference point. The development and maintenance of these products will be divided between Auto-WCAG CG and the ACT Taskforce.

ACT Rule Suite Repository

ACT Rules will be text documents, and keeping them together should be done in some kind of repository, possibly using GitHub as many projects at the W3C do already for ease of collaboration.

The ACT Rule Suite will have a WCAG Core Ruleset, that is determined to have be sufficiently accurate for WCAG conformance testing by the WCAG Working Group. This ruleset will be developed and maintained by the Auto-WCAG Community Group.

In addition to the WCAG Core Ruleset, the W3C Rule Suite will hold rule sets developed by other organizations, that were determined not to fit as part of the WCAG Core Rules. This can hold rules for other accessibility standards, as well as for best practices. Rule designed to give warnings of probable WCAG violations would also be accepted into this repository.

ACT Rule Suite Frontend

To give the rule suite a public face, a front end is required. This website of web app will serve as a common place where documentation can be found about exactly what these rules do. This resource will largely (if not entirely) be based on content derived from the rules themselves. The main purpose for this frontend is as a reference point and a 'hub' for harmonized accessibility conformance testing.

The information made available through the frontend will be kept up to date through a collaboration between the ACT Taskforce and groups such as Auto-WCAG, working on accessibility conformance test rules.