[contents]
The terms defined by this document are also provided in RDF Schema format.
Copyright © 2011 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
This document describes the formal schema of the Evaluation and Report Language (EARL) 1.0. EARL is a vocabulary, the terms of which are defined across a set of specifications and technical notes, and that is used to describe test results. The primary motivation for developing this vocabulary is to facilitate the exchange of test results between Web accessibility evaluation tools in a vendor-neutral and platform-independent format. It also provides reusable terms for generic quality assurance and validation purposes.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This 10 May 2011 Last Call Working Draft of the Evaluation and Report Language (EARL) 1.0 Schema is an update of the previous EARL 1.0 Last Call Working Draft of 29 October 2009. It meets the requirements specified in the Requirements for the Evaluation and Report Language (EARL) 1.0, and incorporates all comments received. This document is intended to be published and maintained as a W3C Recommendation after review and refinement.
The Evaluation and Repair Tools Working Group (ERT WG) believes it has addressed all issues brought forth through previous Working Draft iterations. The Working Group encourages feedback about this document, Evaluation and Report Language (EARL) 1.0 Schema, by developers and researchers who have interest in software-supported evaluation and validation of websites, and by developers and researchers who have interest in Semantic Web technologies for content description, annotation, and adaptation. In particular, the Working Group is looking for feedback on the following items which are also highlighted within the document:
foaf:Document
as a further refinement for earl:TestSubject
(see Editor Note 1)earl:Software
as a subclass of doap:Project
(see Editor Note 2)Please send comments on this Evaluation and Report Language (EARL) 1.0 Schema document by 10 June 2011 to public-earl10-comments@w3.org (publicly visible mailing list archive).
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document has been produced by the Evaluation and Repair Tools Working Group (ERT WG) as part of the Web Accessibility Initiative (WAI) Technical Activity.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any test subject tested against any set of criteria. The test subject might be a website, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.
This document provides the core schema of EARL. Other parts of the EARL suite of specifications include:
The Developer Guide for Evaluation and Report Language (EARL) 1.0 explains how to implement and use EARL, including conformance requirements for software tools. An Evaluation and Report Language (EARL) Overview is also available.
EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.
The assumed audience of this specification is developers of software tools and processes who want to express test results in a machine readable format that is semantically rich. More introductory background about EARL as well as specific guidance for quality assurance tool developers, in particular for web accessibility evaluation tool developers, is provided in the Evaluation and Report Language (EARL) 1.0 Guide.
This document assumes that the reader is familiar with the Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].
The RDF representation of the vocabulary defined by this document uses the namespace http://www.w3.org/ns/earl#
. The prefix earl
is used throughout this document to denote this namespace. Other prefixes used throughout this document include:
cnt
- Representing Content in RDF namespace http://www.w3.org/2011/content#
(defined by [Content])dct
- Dublin Core (DC) namespace http://purl.org/dc/terms/
(defined by [DC])doap
- Description of a Project (DOAP) namespace http://usefulinc.com/ns/doap#
(defined by [DOAP])foaf
- Friend of a Friend (FOAF) namespace http://xmlns.com/foaf/0.1/#
(defined by [FOAF])http
- HTTP Vocabulary in RDF namespace http://www.w3.org/2011/http#
(defined by [HTTP])ptr
- Pointer Methods in RDF namespace http://www.w3.org/2009/pointers#
(defined by [Pointers])rdf
- RDF namespace http://www.w3.org/1999/02/22-rdf-syntax-ns#
(defined by [RDF])rdfs
- RDF Schema namespace http://www.w3.org/2000/01/rdf-schema#
(defined by [RDFS])xsd
- XMLS namespace http://www.w3.org/2001/XMLSchema#
(defined by [XMLS])This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:
EARL provides flexibility to describe different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.
Example 1: A person carries out a manual evaluation of a web page to an accessibility requirement.
http://www.example.org/page.html
Example 2: A software application carries out automated validation of a web page to a technical specification.
http://validator.w3.org/
http://www.example.org/page.html
at 2004-04-14T14:00:04+1000
<li>
element on line 53, char 7 was not closed.Assertion - a statement that embodies the results of a test.
Example 3: Instance of an assertion expressed as an RDF/XML fragment.
<earl:Assertion rdf:about="#assertion">
<earl:assertedBy rdf:resource="#assertor"/>
<earl:subject rdf:resource="http://www.example.org/"/>
<earl:test rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36"/>
<earl:result rdf:resource="#result"/>
</earl:Assertion>
Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.
Rather than specifying only an earl:Assertor
type, it is recommended that one of the following types be employed in addition:
earl:Software
foaf:Agent
foaf:Person
foaf:Organization
foaf:Group
It is recommended to provide additional information about the Assertor by using the following properties from external vocabularies:
dct:title
dct:description
foaf:name
foaf:firstName
or foaf:surname
if the assertor is a person.foaf:nick
foaf:mbox
foaf:mbox_sha1sum
property.foaf:homepage
foaf:member
Example 4: An Assertor that is a person called Bob B. Bobbington.
<foaf:Person rdf:about="http://www.example.org/people/#bob">
<foaf:name>Bob B. Bobbington</foaf:name>
<foaf:mbox rdf:resource="mailto:bob@example.org"/>
<foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>
Example 5: An Assertor that is a piece of software called Cool Tool.
<earl:Software rdf:about="http://www.example.org/tools/#cooltool">
<doap:name xml:lang="en">Cool Tool</doap:name>
<doap:description xml:lang="en">My favorite tool!</doap:description>
<doap:created>2011-04-27</doap:created>
<doap:homepage rdf:resource="http://example.org/tools/cool/"/>
<doap:release>
<doap:revision>1.0.3</doap:revision>
</doap:release>
</earl:Software>
Example 6: An Assertor that is the person from example 4 using the software tool from example 5.
<foaf:Group rdf:about="#assertor">
<dct:title xml:lang="en">Bob using Cool Tool</dct:title>
<dct:description xml:lang="en">Bob doing semi-automated testing</dct:description>
<earl:mainAssertor rdf:resource="http://www.example.org/people/#bob"/>
<foaf:member rdf:resource="http://www.example.org/tool/#cooltool"/>
</foaf:Group>
Note: According to this example, "Cool Tool" is a resource of type foaf:Agent
. According to example 5, it is also a resource of type earl:Software
. These are not contradictory statements and are valid RDF representations.
Test Subject - the class of things that have been tested against some test criterion.
Rather than specifying only an earl:TestSubject
type, it is recommended that one of the following types be employed in addition:
earl:Software
cnt:Content
http:Response
foaf:Document
foaf:Document
unless compelling use-cases can be presented; feedback on this consideration is welcome.]It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:
dct:title
dct:description
dct:date
dct:hasPart
dct:isPartOf
Example 7: A group of resources that have been tested together as a single test subject.
<earl:TestSubject rdf:about="http://www.example.org/">
<dct:title xml:lang="en">example.org Web site</dct:title>
<dct:description xml:lang="en">Each page on the example.org Web site</dct:description>
<dct:hasPart rdf:resource="http://www.example.org/style.css"/>
<dct:hasPart rdf:resource="http://www.example.org/page1.html"/>
<dct:hasPart rdf:resource="http://www.example.org/page2.html"/>
<dct:hasPart rdf:resource="http://www.example.org/image1.png"/>
<dct:hasPart rdf:resource="http://www.example.org/image2.png"/>
</earl:TestSubject>
Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.
Rather than specifying only an earl:TestCriterion
type, it is recommended that one of the following types be employed in addition:
earl:TestRequirement
earl:TestCase
It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:
dct:title
dct:description
dct:hasPart
dct:isPartOf
Example 8: Instance of a test case that is described with a title and its relationship to a test suite.
<earl:TestCase rdf:about="http://www.w3.org/TR/WCAG20-TECHS/H36">
<dct:title xml:lang="en">H36</dct:title>
<dct:description xml:lang="en">Technique H36 - Using alt attributes
on images used as submit buttons </dct:description>
<dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/"/>
<dct:hasPart rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36#H36-tests"/>
</earl:TestCase>
Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).
It is recommended to provide additional information about the Test Result by using the following properties from external vocabularies:
dct:title
dct:description
dct:date
Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#failed"/>
<dct:title xml:lang="en">Invalid Markup (code #353)</dct:title>
<dct:description rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>The <code>table</code> element is not allowed to appear
inside a <code>p</code> element</p>
</div>
</dct:description>
<earl:pointer rdf:resource="#pointer"/>
<earl:info rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>It seems the <code>p</code> element has not been closed</p>
</div>
</earl:info>
</earl:TestResult>
Test Mode - describes how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries.
Where applicable it is recommended to use one of the following instances of earl:TestMode
, to categorize the mode in which the test was carried out:
earl:automatic
earl:manual
earl:semiAuto
earl:undisclosed
earl:unknownMode
It is recommended to provide additional information about the Test Mode by using the following properties from external vocabularies:
dct:title
dct:description
Example 10: The assertion from example 3 was carried out in semi-automatic mode.
<earl:Assertion rdf:about="#assertion">
<earl:mode rdf:resource="http://www.w3.org/ns/earl#semiAuto"/>
</earl:Assertion>
Outcome Value - a value or expression that describes a resulting condition from carrying out the test.
Where applicable it is recommended to use one of the following instances of earl:OutcomeValue
, to categorize the outcome of carrying out the test:
earl:passed
earl:failed
earl:cantTell
earl:inapplicable
earl:untested
In cases where it is necessary to create further instances of earl:OutcomeValue
, it is recommended that one of the following types be employed in addition:
earl:Pass
earl:Fail
earl:CannotTell
earl:NotApplicable
earl:NotTested
It is recommended to provide additional information about the Outcome Value by using the following properties from external vocabularies:
dct:title
dct:description
Example 11: A test result with an outcome of "Passed", using the corresponding instance of earl:OutcomeValue
.
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#passed"/>
</earl:TestResult>
Example 12: A test result with a non-standard outcome of "Warning", which is a type earl:Pass
.
<rdf:Description rdf:about="http://example.org/my/warning#warning">
<rdf:type rdf:resource="http://www.w3.org/ns/earl#Pass"/>
<dc:title xml:lang="en">Warning</dc:title>
<dc:description xml:lang="en">the subject passed the test but there are warnings</dc:description>
</rdf:Description>
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://example.org/my/terms#warning"/>
</earl:TestResult>
[Editor's note 2: ERT WG is looking for feedback on the use of DOAP Project to describe Software; feedback on this issue is welcome.]
A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to Authoring Tool Accessibility Guidelines [ATAG] or of a browser to User Agent Accessibility Guidelines [UAAG]).
Note: earl:Software
is a sublass of doap:Project
to denote the narrower meaning of executable "Software", that is an outcome of a "Project".
It is recommended to provide information about the Software by using the following properties from external vocabularies:
doap:name
doap:description
doap:homepage
doap:created
doap:release
Example 13: Description of a software tool.
<earl:Software rdf:about="#cooltool">
<doap:name xml:lang="en">Cool Tool</doap:name>
<doap:description xml:lang="en">My favorite tool!</doap:description>
<doap:created>2011-04-27</doap:created>
<doap:homepage rdf:resource="http://example.org/tools/cool/"/>
<doap:release>
<doap:revision>1.0.3</doap:revision>
</doap:release>
</earl:Software>
This section describes the properties defined by this document. EARL also uses properties from external vocabularies to provide additional information where necessary.
Asserted By - the assertor of an assertion.
earl:Assertion
earl:Assertor
Subject - the test subject of an assertion.
earl:Assertion
earl:TestSubject
Test - the test criterion of an assertion.
earl:Assertion
earl:TestCriterion
Result - the result of an assertion.
earl:Assertion
earl:TestResult
Mode - the mode in which the test was performed.
earl:Assertion
earl:TestMode
Main Assertor - the assertor that is primarily responsible for performing the test. It is a refinement of the term foaf:member
defined by [FOAF].
earl:Assertor
earl:Assertor
Outcome - the outcome of performing the test.
earl:TestResult
earl:OutcomeValue
Pointer - the location within a test subject that are most relevant to a test result.
earl:TestResult
ptr:Pointer
Info - additional warnings or error messages in a human-readable form.
earl:TestResult
This section summarizes the terms defined and used by this EARL 1.0 Schema specification.
Class Name | Label | Comment | Refinements | Related Properties |
---|---|---|---|---|
earl:Assertion |
Assertion | a statement that embodies the results of a test | - | |
earl:Assertor |
Assertor | an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively | ||
earl:TestSubject |
Test Subject | the class of things that have been tested against some test criterion | ||
earl:TestCriterion |
Test Criterion | a testable statement, usually one that can be passed or failed | ||
earl:TestRequirement (subclass of earl:TestCriterion ) |
Test Requirement | a higher-level requirement that is tested by executing one or more sub-tests | - | |
earl:TestCase (subclass of earl:TestCriterion ) |
Test Case | an atomic test, usually one that is a partial test for a requirement | - | |
earl:TestResult |
Test Result | the actual result of performing the test | - | |
earl:TestMode |
Test Mode | describes how a test was carried out | - | |
earl:OutcomeValue |
Outcome Value | a discrete value that describes a resulting condition from carrying out the test | ||
earl:Pass (subclass of earl:OutcomeValue ) |
Pass | the class of outcomes to denote passing a test | - | earl:outcome |
earl:Fail (subclass of earl:OutcomeValue ) |
Fail | the class of outcomes to denote failing a test | - | earl:outcome |
earl:CannotTell (subclass of earl:OutcomeValue ) |
Undetermined | the class of outcomes to denote an undetermined outcome | - | earl:outcome |
earl:NotApplicable (subclass of earl:OutcomeValue ) |
Not applicable | the class of outcomes to denote the test is not applicable | - | earl:outcome |
earl:NotTested (subclass of earl:OutcomeValue ) |
Not tested | the class of outcomes to denote the test has not been carried out | - | earl:outcome |
earl:Software |
Software | any piece of software such as an authoring tool, browser, or evaluation tool | - |
Property Name | Label | Comment | Domain | Range |
---|---|---|---|---|
earl:assertedBy |
Asserted By | assertor of an assertion | earl:Assertion |
earl:Assertor |
earl:subject |
Subject | test subject of an assertion | earl:Assertion |
earl:TestSubject |
earl:test |
Test | test criterion of an assertion | earl:Assertion |
earl:TestCriterion |
earl:result |
Result | result of an assertion | earl:Assertion |
earl:TestResult |
earl:mode |
Mode | mode in which the test was performed | earl:Assertion |
earl:TestMode |
earl:mainAssertor (subproperty of foaf:member ) |
Main Assertor | assertor that is primarily responsible for performing the test | earl:Assertor |
earl:Assertor |
earl:outcome |
Outcome | outcome of performing the test | earl:TestResult |
earl:OutcomeValue |
earl:pointer |
Pointer | location within a test subject that are most relevant to a test result | earl:TestResult |
ptr:Pointer |
earl:info |
Info | additional warnings or error messages in a human-readable form | earl:TestResult |
Literal |
Instance Name | Title | Description |
---|---|---|
earl:automatic (instance of earl:TestMode ) |
Automatic | where the test was carried out automatically by the software tool and without any human intervention |
earl:manual (instance of earl:TestMode ) |
Manual | where the test was carried out by human evaluators |
earl:semiAuto (instance of earl:TestMode ) |
Semi-Automatic | where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test |
earl:undisclosed (instance of earl:TestMode ) |
Undisclosed | where the exact testing process is undisclosed |
earl:unknownMode (instance of earl:TestMode ) |
Unknown | where the testing process is unknown or undetermined |
earl:passed (instance of earl:Pass ) |
Passed | the subject passed the test |
earl:failed (instance of earl:Fail ) |
Failed | the subject failed the test |
earl:cantTell (instance of earl:CannotTell ) |
Cannot tell | it is unclear if the subject passed or failed the test |
earl:inapplicable (instance of earl:NotApplicable ) |
Inapplicable | the test is not applicable to the subject |
earl:untested (instance of earl:NotTested ) |
Untested | the test has not been carried out |
This section provides references to related documents and specifications.
EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.
Shadi Abou-Zahra, Philip Ackermann, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Emmanuelle Gutiérrez y Restrepo, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, Rui Lopes, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, Konstantinos Votis, and Rob Yonaitis.
Besides several minor editorial changes, the most significant changes from the 29 October, 2009 Working Draft include:
earl:Software
a subclass of doap:Project
, and reused several terms from DOAPA detailed listing of the comments, resolutions, and changes made is provided in the Open Issues for EARL 1.0 Schema listing.