Testing Resources

W3C Groups on Testing

Browser Testing and Tools Working Group
- WebDriver Specification
Web Platform Testing @@
- Test Suites for Web Platform
https://github.com/w3c/web-platform-tests
Spec-ops: Accessible Technology Test Adapter (https://spec-ops.github.io/atta-api/)
...

Accessibility Testing Repositories

Open Ajax Aliance http://oaa-accessibility.org/
aXe Rules https://dequeuniversity.com/rules/worldspace/2.0/
QuailJS https://github.com/quailjs/quail/tree/master/src/assessments and the QUAIL Test description documentation: https://quail.readthedocs.io/en/latest/
eGovMon http://wiki.egovmon.no/wiki/Category:Part_A
RGAA http://disic.github.io/rgaa_referentiel_en/RGAA3.0_Criteria_English_version_v1.html
Code Sniffer tests http://squizlabs.github.io/HTML_CodeSniffer/Standards/WCAG2/
Google Chrome audit rules https://github.com/GoogleChrome/accessibility-developer-tools/wiki/Audit-Rules
Accessibility fails https://cfq.github.io/accessibility-fails/ - This has coded failures for automated test tools to find. They also have a github location: https://github.com/ukgovdatascience/accessibility_fails
Va11yS: https://ibma.github.io/Va11yS/
Dynamic Assessment Plugin (DAP) ruleset: http://aat.mybluemix.net/ Note*: Need to first create an IBM id.
BenToWeb v1 suite: https://github.com/webcc/bentoweb-wcag20-test-suite-v1
BenToWeb v2 suite: https://github.com/webcc/bentoweb-wcag20-test-suite-v2
BenToWeb v3 suite: https://github.com/webcc/bentoweb-wcag20-test-suite-v3
...

Standards on Testing

ISO/IEC 29119
Web Accessibility Conformance Evaluation Methodology (working group note): https://www.w3.org/TR/WCAG-EM/
...

Test case format (explicit results)

{
  "name": "axe-core",
  "version": "2.1.7",
  "license": "MPL-2.0",
  "environment": [{
       "os": "Windows",
       "osVersion": "10",
       "browser": "Chrome",
       "browserVersion": "x.x.x",
       "resultsReporter": "Junit | (optional) name of application wrapping engine | etc..."
   }],
  "a11y-testcases": [{
    "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
    "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html", "mytool:image-alt"],
    "failed": ["#violation1", "#violation2", "#violation3", "#violation4"],
    "passed": ["#pass1", "#pass2", "#pass3", "#pass4", "#pass5", "#pass6"],
    "na": ["#na1", "#na2", "#na3", "#na4", "#na5", "#na6"]
  }, {
    ...
  }]
}

Test case format (implicit results)

{
  "name": "axe-core",
  "version": "2.1.7",
  "license": "MPL-2.0",
  "a11y-selectors": {
    "failedSelector": "*[data-expectedoutcome='violation']",
    "passedSelector": "*[data-expectedoutcome='pass']",
  },
  "a11y-testcases": [{
    "url": "https://raw.githubusercontent.com/dequelabs/axe-core/master/test/integration/rules/image-alt/image-alt.html",
    "test": ["wcag20:text-equiv-all", "WCAG-TECHS:G90.html", "mytool:image-alt"]
  }, {
    ...
  }]
}

Test Procedures

DHS Trusted Tester (U.S. Section 508)
- Harmonized Testing Process for Section 508 Compliance: Baseline Tests for Software and Web Accessibility (ver 2.0.1 | Oct 2016) https://www.dhs.gov/sites/default/files/publications/Baseline_Tests_for_Software_and_Web_Accessibility_v2_0_1.pdf
- Dept. of Labor: Mobile Accessibility Test Script https://github.com/USDepartmentofLabor/Mobile-Accessibility-Test-Script/blob/master/QA%20Test%20Cases.md

Take-aways from WPT & ARIA Test Harness

1: WPT uses test types that can likely be reused by ACT:

IDL Harness
Test Harness (scriptable test)
Reftest (visual test)
WebDriver
Manual

2: Requirements for WPT tests are as follows:

Short
Minimal
Cross-platform
self-containing

3: All tests are written in HTML or SVG

4: Self-describing tests is an important concept to WPT:

They can be run easily on any layout engine.
They can test areas of the spec that are not precise enough to be comparable to a reference rendering. (For example, underlining cannot be compared to a reference because the position and thickness of the underline is UA-dependent.)
Failures can (should) be easily determined by a human viewing the test without needing special tools.

5: Manual tests in WPT should be:

Short, easy & quick to determine the result
Self explanatory & not require an understanding of the specification

6: There is frequent use of features like "accessible name" and "role", without indicating how those are derived.

7: Test outputs are only ever 'pass' or 'fail' for APT