Warning:
This wiki has been archived and is now read-only.
Result template
The Result template captures the output of a test. On the one hand the output must be a detailed as possible to be useful for presentation and further analysis. On the other hand it must be possible to map the output from different tools or evaluations to a common standard (i.e. a minimal set of properties) so that results can be aggregated and compared.
The Evaluation and Report Language (EARL) was developed by W3C to address these requirements. Other tools and projects have defined result formats as well.
EARL
The W3C Evaluation and Report Language (EARL) defines a schema to describe test results.
An EARL Assertion contains the following information:
- Assertor{1}
- This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
- Test Subject{1}
- This can include web content (such as web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
- Test Criterion{1}
- What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
- Test Result{1}
- What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.
- Test Mode{0,1}
- This describes how a test was carried out. For example
automatic
ormanual
orsemiAuto
.
In the auto-wcag result template the two classes earl:TestCriterion
and earl:TestResult
can be used to capture the result of the test.
earl:TestCriterion
EARL10 recommends: "Rather than specifying only an earl:TestCriterion type, it is recommended that one of the following types be employed in addition:
earl:TestRequirement
- a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criterion 1.1.1, which is evaluated using several Techniques for Success Criterion 1.1.1 and combining the results.-
earl:TestCase
- an atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criterion 1.1.1."
A test definded by auto-wcag would be described as earl:TestCase
.
earl:TestResult
The following properties are recorded for the actual test result:
number | Property | Description |
---|---|---|
exactly one | earl:outcome
|
one of earl:passed , earl:failed , earl:cantTell , earl:inapplicable , earl:untested ,
|
at most one | earl:info
|
additional warnings or error messages in a human-readable form |
any number of | earl:pointer
|
location within a test subject that are most relevant to a test result |
at most one | dct:title
|
|
at most one | dct:description
|
|
exactly one | dct:date
|
The auto-wcag approach
- The test results reported by the auto-wcag tests are aligned with EARL. The main values can be easily mapped to EARL.
- auto-wcag introduces a new property
earl:errorMessage
as a subclass ofearl:info
. This is the common reporting property of all auto-wcag tools. - Tools may use
earl:info
to record additional information that can facilitate analysis and further processing of the reports. Such as code extracts for easy presentation of the source of the accessibility problem or test output e.g. the value of color contrast calculation, language of the content as determined by the tool - Additional info in
earl:errorMessage
andearl:info
is only reported if the test failed. This reduces the size and number of reports. - If necessary, further values for the outcomes could be defined:
- earl:Pass - the class of all positive outcome values
- earl:passed - an instance of passing a test requirement
The following table summarizes the auto-wcag approach to reporting in EARL:
Property | Values used in auto-wcag | auto-wcag extensions |
---|---|---|
earl:outcome
|
earl:passed , earl:failed , earl:cantTell
|
new outcome value earl:notPresent (subclass of earl:Pass )
|
earl:info
|
additional warnings or messages in a human-readable form | new subclass earl:errorMessage
|
earl:errorMessage
|
error message describing the type of error, used by all auto-wcag tests | introduced by auto-wcag as new subclass of earl:info
|
earl:pointer
|
location within a test subject that are most relevant to a test result | auto-wcag still needs to agree on the format of the pointer. |
These templates are used to specify the test results in auto-wcag:
Background
- WAET
- Developers' Guide to Features of Web Accessibility Evaluation Tools: Standard reporting languages
- EARL10
- Evaluation and Report Language (EARL)