Warning:
This wiki has been archived and is now read-only.

Result template

From Automated WCAG Monitoring Community Group
Jump to: navigation, search

The Result template captures the output of a test. On the one hand the output must be a detailed as possible to be useful for presentation and further analysis. On the other hand it must be possible to map the output from different tools or evaluations to a common standard (i.e. a minimal set of properties) so that results can be aggregated and compared.

The Evaluation and Report Language (EARL) was developed by W3C to address these requirements. Other tools and projects have defined result formats as well.

EARL

The W3C Evaluation and Report Language (EARL) defines a schema to describe test results.

An EARL Assertion contains the following information:

Assertor{1}
This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject{1}
This can include web content (such as web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
Test Criterion{1}
What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result{1}
What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.
Test Mode{0,1}
This describes how a test was carried out. For example automatic or manual or semiAuto.

In the auto-wcag result template the two classes earl:TestCriterion and earl:TestResult can be used to capture the result of the test.

earl:TestCriterion

EARL10 recommends: "Rather than specifying only an earl:TestCriterion type, it is recommended that one of the following types be employed in addition:

  • earl:TestRequirement - a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criterion 1.1.1, which is evaluated using several Techniques for Success Criterion 1.1.1 and combining the results.
  • earl:TestCase - an atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criterion 1.1.1."

A test definded by auto-wcag would be described as earl:TestCase.

earl:TestResult

The following properties are recorded for the actual test result:

number Property Description
exactly one earl:outcome one of earl:passed, earl:failed, earl:cantTell, earl:inapplicable, earl:untested,
at most one earl:info additional warnings or error messages in a human-readable form
any number of earl:pointer location within a test subject that are most relevant to a test result
at most one dct:title
at most one dct:description
exactly one dct:date

The auto-wcag approach

  1. The test results reported by the auto-wcag tests are aligned with EARL. The main values can be easily mapped to EARL.
  2. auto-wcag introduces a new property earl:errorMessage as a subclass of earl:info. This is the common reporting property of all auto-wcag tools.
  3. Tools may use earl:info to record additional information that can facilitate analysis and further processing of the reports. Such as code extracts for easy presentation of the source of the accessibility problem or test output e.g. the value of color contrast calculation, language of the content as determined by the tool
  4. Additional info in earl:errorMessage and earl:info is only reported if the test failed. This reduces the size and number of reports.
  5. If necessary, further values for the outcomes could be defined:
  • earl:Pass - the class of all positive outcome values
  • earl:passed - an instance of passing a test requirement

The following table summarizes the auto-wcag approach to reporting in EARL:

Property Values used in auto-wcag auto-wcag extensions
earl:outcome earl:passed, earl:failed, earl:cantTell new outcome value earl:notPresent (subclass of earl:Pass)
earl:info additional warnings or messages in a human-readable form new subclass earl:errorMessage
earl:errorMessage error message describing the type of error, used by all auto-wcag tests introduced by auto-wcag as new subclass of earl:info
earl:pointer location within a test subject that are most relevant to a test result auto-wcag still needs to agree on the format of the pointer.

These templates are used to specify the test results in auto-wcag:

Background

WAET
Developers' Guide to Features of Web Accessibility Evaluation Tools: Standard reporting languages
EARL10
Evaluation and Report Language (EARL)