ARIA 1.1 Manual Testing

How can I check where manual testing is needed?

Is the platform and user agent shown in the list of implementations for which we have a report?

• No: Manual testing is likely needed.
• Yes: Manual testing is likely NOT needed. It would only be needed if the report for that platform + user agent lacks most or all of the data we need.

The presence of the data is indicated by green "pass" cells and red "fail" cells in the test results table. Lack of data is indicated by yellowish-orange "-" table cells. Reasons for which data may be lacking include:

1. A given platform has additional mapping requirements which other platforms lack. For example, AX API's mapping for aria-modal includes the stipulation that the web content outside of the aria-modal window is not exposed when the modal window is showing. Other platforms do not have that requirement. Thus the lack of data for those other platforms means "not applicable; there is nothing more to do here."
2. We have no automated means in place to test something.
3. New tests were added but the new results have not yet been provided.

If you are unsure as to which condition is the case and/or what needs to be done to address the issue, please get in touch with the person listed as the contact for that platform and user agent. Contacts can be found in the list of implementations for which we have a report.

Using the web-platform-tests runner to conduct manual testing

1. Go to https://w3c-test.org/tools/runner/index.html
2. In "Run tests under path", type "/wai-aria"
3. Press the "Start" button (note that tests have to be loaded which can take several minutes). When the loading process has completed, a new window with the first test to be run should appear.
4. For every test to be run manually, you should find a table of expectations with at least one step and the per-platform expectations for that step. Use the appropriate accessibility inspection tool for the platform to locate the element with the specified id and verify all of the assertions.
5. Switch back to the test runner window. If all of the platform assertions for the current test passed, press the "Passed" button; otherwise press the "Failed" button.
6. Switch back to the test case window where you should find a new test has been loaded. Rinse and repeat until you have completed running all of the tests.
7. In the test runner window, press the "Download JSON results" button. Save the resulting output and email it to Joanie along with the user agent name and version number, the operating system name and version number, and the accessibility API name (version number if possible).

Tips and other information

1. Refrain from closing the test window during a run cycle. In Joanie's experience, a new window will not be created by the test runner after you click "Passed" or "Failed"; the existing test window is reused (whether it still exists or not).
2. If you click "Failed," the "Details" section of the test runner will display "NOTRUN" (as opposed to "FAILED", as Joanie would have expected). That's ok. The resulting JSON file you create will show the status as "FAIL".
3. We plan to add a means to hide accessibility APIs which are not being tested during a run so that manual testers can focus on what they are currently testing. This is not yet in place.
4. Most, if not all, user agents are exposing the element's id as an attribute of the accessibility object. As a result, you should use this means to locate the element in your platform's accessibility inspection tool.
5. As a general rule, you will want to test using the very latest versions of the user agent and accessibility API being tested. A number of the implemented ARIA 1.1 features are only available in nightly builds of the user agent, and those which have not yet been implemented and/or have bugs in the implementation will not show up in stable releases for a while. Similarly, there are a number of ARIA features which require additions and/or fixes in the platform accessibility API(s). Thus if you have the latest nightly build of your user agent, but an older version of the accessibility API, your findings may suggest that an ARIA feature has not been properly implemented in the user agent, when in reality it has been implemented.
6. If you send Joanie your JSON file as suggested above, she will do the work to get it integrated into the official implementation report. But if you would prefer to do that integration yourself, please feel free to do so. See https://w3c.github.io/test-results/wai-aria for more information.