Result template

The Result template captures the output of a test. On the one hand the output must be a detailed as possible to be useful for presentation and further analysis. On the other hand it must be possible to map the output from different tools or evaluations to a common standard (i.e. a minimal set of properties) so that results can be aggregated and compared.

The Evaluation and Report Language (EARL) was developed by W3C to address these requirements. Other tools and projects have defined result formats as well.

EARL

The W3C Evaluation and Report Language (EARL) defines a schema to describe test results.

An EARL Assertion contains the following information:

Assertor{1}

This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.

Test Subject{1}

This can include web content (such as web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.

Test Criterion{1}

What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.

Test Result{1}

What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.

Test Mode{0,1}

This describes how a test was carried out. For example automatic or manual or semiAuto.

In the auto-wcag result template the two classes earl:TestCriterion and earl:TestResult can be used to capture the result of the test.

earl:TestCriterion

EARL10 recommends: "Rather than specifying only an earl:TestCriterion type, it is recommended that one of the following types be employed in addition:

• earl:TestRequirement - a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criterion 1.1.1, which is evaluated using several Techniques for Success Criterion 1.1.1 and combining the results.
• earl:TestCase - an atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criterion 1.1.1."

A test definded by auto-wcag would be described as earl:TestCase.

earl:TestResult

The following properties are recorded for the actual test result:

The auto-wcag approach

1. The test results reported by the auto-wcag tests are aligned with EARL. The main values can be easily mapped to EARL.
2. auto-wcag introduces a new property earl:errorMessage as a subclass of earl:info. This is the common reporting property of all auto-wcag tools.
3. Tools may use earl:info to record additional information that can facilitate analysis and further processing of the reports. Such as code extracts for easy presentation of the source of the accessibility problem or test output e.g. the value of color contrast calculation, language of the content as determined by the tool
4. Additional info in earl:errorMessage and earl:info is only reported if the test failed. This reduces the size and number of reports.
5. If necessary, further values for the outcomes could be defined:

• earl:Pass - the class of all positive outcome values
• earl:passed - an instance of passing a test requirement

The following table summarizes the auto-wcag approach to reporting in EARL:

These templates are used to specify the test results in auto-wcag:

• Template:Passed
• Template:Failed
• Template:Inapplicable
• Template:TestResult

Background

WAET

Developers' Guide to Features of Web Accessibility Evaluation Tools: Standard reporting languages

EARL10

Evaluation and Report Language (EARL)