Multi-page approach

WCAG has a number of success criteria that can only be tested when a page is compared to other pages from the same set of web pages. WCAG defines a set of web pages as "collection of Web pages that share a common purpose and that are created by the same author, group or organization".

Comparing a web page to all other pages from a set of web pages is usually not feasible. Therefore auto-wcag uses a more efficient approach to take a sample of the pages and only compare the page you are testing to this smaller sample.

Sampling

auto-wcag assumes that a sample of the web site already is available. auto-wcag does not give recommendations on selecting the sample because the sampling approach and size of the sample depend on the goal of the evaluation.

Guidance on sampling is can be found in the WCAG documentation and other sources.

Manual sampling

A good starting point when working with manual samples is the WCAG Evaluation Methodology. This approach will most likely result in a better balance between sample size and representation of web pages within the website than can be obtained from an automated sample.

Use Step 1, 2 and 3 of the WCAG Evaluation Methodology to create a manual sample:

• Define the scope of the website to be evaluated (WCAG-EM Step 1)
• Explore the website to find what kinds of information and functionality is provided (WCAG-EM Step 2)
• Create a structured and a random sample (WCAG-EM Step 3)
• Use all pages in the structured and random sample as the sample.

Special case: Testing a single page

When you are only interested in testing a single page, building an entire sample or crawling the website to create a sample is fairly inefficient. Instead, a smaller sample can be generated from the content of the page that is to be tested.

Approach:

• Take all anchor elements (<a href=...>) on the page not hidden with display:none; or visibility:hidden;.
• If the URL of the anchor element is an internal link, take the URL and add it to the sample.

Testing

There are two scenarios for multi-page tests:

• The SC requires all web pages to be similar (e.g. consistent navigation).
• The SC requires all web pages to be different (e.g. page titled).

Checking for similarity

Required by the following Success Criteria:

• 2.2.1 Timing Adjustable
• 2.4.1 Bypass Blocks
• 2.4.5 Multiple Ways
• 3.2.3 Consistent Navigation
• 3.2.4 Consistent Identification

Test procedure:

1. Assume the relevant property P and similarity measure m have been defined. E.g. For SC 3.2.3: P = navigational elements of a web page and m = same relative order
2. To check page A: compare A to all other pages in the sample.
3. If A is similar to more the 50% of the pages in the sample, return "Passed" for page A. Else return "Failed".

The test procedure is applied to each page that is tested. Each page has an individual report for each test case. The pages in the same sample might have different reports.

Note that the threshold 50% was chosen arbitrarily. The number is low to avoid false positive reports. An implementation of auto-wcag may use a different threshold or even individual thresholds per test case, as long as the threshold values are disclosed by the implementation.

Checking for difference

Required by the following Success Criterion:

• 2.4.2 Page Titled

The pages of a web site should all have different titles. However, pages belonging to the same complete process can have the same title. If it is known which pages belong to the same process (i.e. the sample is aware of complete processes), this test can be carried out automatically.

If it is not known which pages belong to the same process the following test procedure is suggested:

• Compare the title of the page to the title of the other pages in the sample.
• If two titles are the same, ask the user to determine if the the pages belong to the same process (and should have the same title) or if the titles should be different.