[contents]
Copyright © 2009 W3C® (MIT, ERCIM, Keio), All Rights Reserved. W3C liability, trademark and document use rules apply.
This document describes the formal schema of the Evaluation and Report Language (EARL) 1.0. The Evaluation and Report Language is a vocabulary to express test results. The primary motivation for developing this language is to facilitate the exchange of test results between Web accessibility evaluation tools in a vendor-neutral and platform-independent format. It also provides a reusable vocabulary for generic quality assurance and validation purposes. While this document focuses on the technical details of the specification, a companion document Evaluation and Report Language (EARL) 1.0 Guide [Guide], describes the motivations for EARL and provides an introduction to its use.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This 23 Febaruary 2009 Editor's Draft of the Evaluation and Report Language (EARL) 1.0 Schema is an update of the previous EARL 1.0 Working Draft of 23 March 2007. It meets the requirements specified in the Requirements for the Evaluation and Report Language (EARL) 1.0, and incorporates change requests received since the March 2007 Working Draft. In particular, this draft implements the decisions of the Evaluation and Repair Tools Working Group (ERT WG) at its face to face meeting in November 2008 (see history of document changes). This document is intended to be published and maintained as a W3C Recommendation after review and refinement.
The Evaluation and Repair Tools Working Group (ERT WG) believes to have addressed all issues brought forth through previous Working Draft iterations. The Working Group encourages feedback about this document, Evaluation and Report Language (EARL) 1.0 Schema, by developers and researchers who have interest in software-supported evaluation and validation of Web sites. Feedback from developers and researchers who have interest in Semantic Web technologies for content description, annotation, and adaptation is also strongly encouraged.
Please send comments on this document to the public email list of the working group public-wai-ert@w3.org. The archives of the working group mailing list are publicly available.
Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document has been produced by the Evaluation and Repair Tools Working Group (ERT WG) as part of the Web Accessibility Initiative (WAI) Technical Activity.
This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.
The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any thing tested against any set of criteria. The test subject might be a Web site, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.
EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.
A companion document Evaluation and Report Language (EARL) 1.0 Guide [Guide], to this specification provides more introductory material and explanation of the use cases for EARL. The companion document also highlights specific considerations, such as security and privacy.
The assumed audience of this specification is developers who are implementing EARL in software or processes, or those who are seeking to understand the ideas, models, or properties and classes used in the EARL vocabulary. Readers who would like more introductory material for the language with explanation of its foreseen use cases are referred to the Evaluation and Report Language (EARL) 1.0 Guide [Guide].
This document assumes that the reader is familiar with the ideas of Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].
The keywords must, required, recommended, should, may, and optional in this document are used in accordance with RFC 2119 [RFC2119].
Where an RDF term is used in its abbreviated form (e.g. Assertion
or foaf:Person
), that term is in the EARL namespace if no namespace is provided. The following prefixes are used in examples throughout this document:
[Editor's note: the following list will be revised.]
earl
http://www.w3.org/ns/earl#
which is described in this documentdc
http://purl.org/dc/elements/1.1/
whose terms are described in [DC]dct
http://purl.org/dc/terms/
described at [DCT]foaf
http://xmlns.com/foaf/0.1/
which is also where the terms are described [FOAF]http
http://www.w3.org/2006/http#
described in [HTTP]owl
http://www.w3.org/2002/07/owl#
described in [OWL]ptr
@@@@
described in [Pointers]rdf
http://www.w3.org/1999/02/22-rdf-syntax-ns#
described in [RDF]rdfs
http://www.w3.org/2000/01/rdf-schema#
described in [RDFS]xsd
http://www.w3.org/2001/XMLSchema#
described in [XMLS]This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:
EARL provides flexibility to describes different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.
Example 1: A person carries out a manual evaluation of a Web page against an accessibility requirement.
http://www.example.org/page.html
Example 2: A software application carries out automated validation of a Web page against a technical specification.
http://validator.w3.org/
http://www.example.org/page.html
at 2004-04-14T14:00:04+1000
<li>
element on line 53, char 7 was not closed.Assertion - a statement that embodies the results of a test.
Example 3: Instance of an assertion expressed as an RDF/XML fragment.
<earl:Assertion rdf:about="#assertion">
<earl:assertedBy rdf:resource="#assertor"/>
<earl:subject rdf:resource="http://www.example.org/"/>
<earl:test rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36"/>
<earl:result rdf:resource="#result"/>
</earl:Assertion>
Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.
Rather than using the generic earl:Assertor
class directly, it is recommended that one of the following refinements be employed:
earl:Software
foaf:Agent
foaf:Person
foaf:Organization
foaf:Group
It is recommended to provide additional information about the Assertor by using the following properties from external vocabularies:
dc:title
dc:description
foaf:name
foaf:firstName
, foaf:surname
, or foaf:nick
if the assertor is a person.foaf:mbox
foaf:mbox_sha1sum
property.foaf:homepage
foaf:member
Example 4: An Assertor that is a person called Bob B. Bobbington.
<foaf:Person rdf:about="http://www.example.org/people/#bob">
<foaf:name>Bob B. Bobbington</foaf:name>
<foaf:mbox rdf:resource="mailto:bob@example.org"/>
<foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>
Example 5: An Assertor that is a piece of software called Cool Tool.
<earl:Software rdf:about="http://www.example.org/tools/#cool">
<dc:title xml:lang="en">Cool Tool</dc:title>
<dc:description xml:lang="en">My favorite tool!</dc:description>
<foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
<dct:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>
Example 6: An Assertor that is the person from example 4 using the software tool from example 5.
<foaf:Agent rdf:about="#assertor">
<dc:title xml:lang="en">Bob using Cool Tool</dc:title>
<dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
<earl:mainAssertor rdf:resource="http://www.example.org/people/#bob"/>
<foad:member rdf:resource="http://www.example.org/tool/#cool"/>
</foaf:Agent>
Test Subject - the class of things that have been tested against some test criterion.
Rather than using the generic earl:TestSubject
class directly, it is recommended that one of the following refinements be employed:
earl:Software
cnt:Content
http:Response
foaf:Document
foaf:Document
is still under discussion by ERT WG, feedback on this consideration is welcome.]It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:
dc:title
dc:description
dc:date
dc:hasPart
dc:isPartOf
Example 7: A group of resources that have been tested together as a single test subject.
<earl:TestSubject rdf:about="http://www.example.org/">
<dc:title xml:lang="en">example.org Web site</dc:title>
<dc:description xml:lang="en">Each page on the example.org Web site</dc:description>
<dct:hasPart rdf:resource="http://www.example.org/style.css"/>
<dct:hasPart rdf:resource="http://www.example.org/page1.html"/>
<dct:hasPart rdf:resource="http://www.example.org/page2.html"/>
<dct:hasPart rdf:resource="http://www.example.org/image1.png"/>
<dct:hasPart rdf:resource="http://www.example.org/image2.png"/>
</earl:TestSubject>
Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.
Rather than using the generic earl:TestCriterion
class directly, it is recommended that one of the following refinements be employed:
earl:TestRequirement
earl:TestCase
It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:
dc:title
dc:description
dc:hasPart
dc:isPartOf
Example 8: Instance of a test case that is described with a title and its relationship to a test suite.
<earl:TestCase rdf:about="http://www.w3.org/TR/WCAG20-TECHS/H36">
<dc:title xml:lang="en">H36</dc:title>
<dc:description xml:lang="en">Technique H36 - Using alt attributes on images used as submit buttons </dc:description>
<dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/"/>
<dct:hasPart rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36#H36-tests"/>
</earl:TestCase>
Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).
It is recommended to provide additional information about the Test Result by using the following properties from external vocabularies:
dc:title
dc:description
dc:date
Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://www.w3.org/2007/earl#Fail"/>
<dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
<dc:description rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>The <code>table</code> element is not allowed to appear
inside a <code>p</code> element</p>
</div>
</dc:description>
<earl:pointer rdf:resource="#pointer"/>
<earl:info rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>It seems the <code>p</code> element has not been closed</p>
</div>
</earl:info>
</earl:TestResult>
The Test Mode described how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries.
Rather than using the generic earl:TestMode
class directly, it is recommended that one of the following refinements be employed:
earl:Automatic
earl:Manual
earl:SemiAuto
earl:Undisclosed
earl:Unknown
It is recommended to provide additional information about the Test Mode by using the following properties from external vocabularies:
dc:title
dc:description
Example 10: The assertion from example 3 was carried out in semi-automatic mode.
<earl:Assertion rdf:about="#assertion">
<earl:mode rdf:resource="http://www.w3.org/2007/earl#SemiAuto"/>
</earl:Assertion>
Outcome Value - a discrete value that describes a resulting condition from carrying out the test.
Rather than using the generic earl:Outcome
class directly, it is recommended that one of the following refinements be employed:
earl:Pass
earl:Fail
earl:CannotTell
earl:NotApplicable
earl:NotTested
It is recommended to provide additional information about the Outcome Value by using the following properties from external vocabularies:
dc:title
dc:description
[Editor's note: ERT WG is considering the use of DOAP terms to describe Software in place of this term. Feedback on this consideration is welcome.]
A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to the Authoring Tool Accessibility Guidelines or of a browser to the User Agent Accessibility Guidelines).
It is recommended to provide information about the Software by using the following properties from external vocabularies:
dc:title
dc:description
foaf:homepage
dc:hasVersion
dc:hasPart
dc:isPartOf
Example 11: Description of a software tool.
<earl:Software rdf:about="#tool">
<dc:title xml:lang="en">Cool Tool</dc:title>
<dc:description xml:lang="en">My favorite tool!</dc:description>
<foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
<dct:hasVersion>1.0.3</dct:hasVersion>
<dct:isPartOf rdf:resource="http://example.org/tools/cms/"/>
<dct:hasPart rdf:resource="http://example.org/tools/cool/#module-1"/>
</earl:Software>
This section describes the properties defined by this document. EARL also uses properties from external vocabularies to provide additional information where necessary.
Asserted By - the assertor of an assertion.
earl:Assertion
earl:Assertor
Subject - the test subject of an assertion.
earl:Assertion
earl:TestSubject
Test - the test criterion of an assertion.
earl:Assertion
earl:TestCriterion
Result - the result of an assertion.
earl:Assertion
earl:TestResult
Mode - the mode in which the test was performed.
earl:Assertion
earl:TestMode
[Editor's note: the concept of mainAssertor and the dropped helpAssertor needs to be revised since it does not validate with FOAF.]
Main Assertor - the assertor that is primarily responsible for performing the test. It is a refinement of the term foaf:member
defined by [FOAF].
earl:Assertor
earl:Assertor
Outcome - the outcome of performing the test.
earl:TestResult
earl:OutcomeValue
Pointer - the locations within the test subject that are most relevant to the outcome of the test.
earl:TestResult
Info - additional warnings or error messages in a human-readable form.
earl:TestResult
This section defines the conformance to the EARL 1.0 specification. Applications conform to EARL 1.0 when the following applies:
Valid EARL 1.0 reports must validate against the RDF grammar specification. In addition, the following restrictions apply to valid EARL 1.0 reports:
Class | Property | Restriction |
---|---|---|
earl:Assertion |
earl:assertedBy |
exactly one assertor per assertion |
earl:subject |
exactly one test subject per assertion | |
earl:test |
exactly one test criterion per assertion | |
earl:result |
exactly one test result per assertion | |
earl:mode |
at most one test mode per assertion | |
earl:Assertor |
foaf:name , dc:title |
exactly one title (per language) or name per assertor |
dc:description |
exactly one description (per language) per assertor | |
earl:mainAssertor |
none [Editor's note: see issue related to mainAssertor] | |
foaf:homepage |
none | |
foaf:Agent |
foaf:name , foaf:nick |
at least one name or nick name per agent |
foaf:mbox |
none | |
foaf:mbox_sha1sum |
none | |
foaf:homepage |
none | |
foaf:Person |
foaf:name , foaf:firstName , foaf:surname , foaf:nick |
at least one name, first name, surname, or nick name per person |
foaf:mbox |
none | |
foaf:mbox_sha1sum |
none | |
foaf:homepage |
none | |
foaf:Organization |
foaf:name , foaf:nick |
at least one name or nick name per organization |
foaf:mbox |
none | |
foaf:mbox_sha1sum |
none | |
foaf:homepage |
none | |
foaf:Group |
foaf:name , foaf:nick |
at least one name or nick name per organization |
foaf:member |
none | |
foaf:mbox |
none | |
foaf:mbox_sha1sum |
none | |
foaf:homepage |
none | |
@@@ | @@@ | @@@ |
[Editor's note: this section needs revision.]
http://www.dublincore.org/documents/dces/
http://www.dublincore.org/documents/dcmi-terms/
http://xmlns.com/foaf/0.1/
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Guide-20051214
http://www.w3.org/TR/HTTP-in-RDF/
http://www.w3.org/TR/owl-features/
http://www.w3.org/WAI/ER/Pointers/WD-Pointers-20070222
http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/
http://www.w3.org/TR/rdf-primer/
http://www.w3.org/TR/rdf-schema/
http://www.w3.org/DesignIssues/RDF-XML
http://www.ietf.org/rfc/rfc2119.txt
http://www.w3.org/TR/WCAG10/
http://www.w3.org/TR/xmlschema-0/
EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.
Shadi Abou-Zahra, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, and Rob Yonaitis.