The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any test subject tested against any set of criteria. The test subject might be a Web site, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.
EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.
A companion document, Evaluation and Report Language (EARL) 1.0 Guide [Guide], to this specification provides more introductory material and explanation of the use cases for EARL. The companion document also highlights specific considerations, such as security and privacy.
The assumed audience of this specification is developers who are implementing EARL in software or processes or those who are seeking to understand the ideas, models, or properties and classes used in the EARL vocabulary. Readers who would like more introductory material for the language with explanation of its foreseen use cases are referred to the Evaluation and Report Language (EARL) 1.0 Guide [Guide].
This document assumes that the reader is familiar with the Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].
The keywords must, required, recommended, should, may, and optional in this document are used in accordance with RFC 2119 [RFC 2119].
The following prefixes are used throughout this document:
earl- EARL namespace
http://www.w3.org/ns/earl# (defined by this document)
dc- Dublin Core (DC) elements namespace
http://purl.org/dc/elements/1.1/ (defined by [DC])
dct- Dublin Core (DC) terms namespace
http://purl.org/dc/terms/ (defined by [DC])
foaf- Friend of a Friend (FOAF) namespace
http://xmlns.com/foaf/spec/# (defined by [FOAF])
http- HTTP Vocabulary in RDF namespace
http://www.w3.org/2006/http# (defined by [HTTP])
cnt- Representing Content in RDF namespace
http://www.w3.org/2008/content# (defined by [Content])
ptr- Pointer Methods in RDF namespace
http://www.w3.org/2009/pointers# (defined by [Pointers])
rdf- RDF namespace
http://www.w3.org/1999/02/22-rdf-syntax-ns# (defined by [RDF])
rdfs- RDF Schema namespace
http://www.w3.org/2000/01/rdf-schema# (defined by [RDFS])
xsd- XMLS namespace
http://www.w3.org/2001/XMLSchema# (defined by [XMLS])
This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:
- Assertor
- This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
- Test Subject
- This can include Web content (such as Web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
- Test Criterion
- What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
- Test Result
- What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.
EARL provides flexibility to describe different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.
Example 1: A person carries out a manual evaluation of a Web page against an accessibility requirement.
- Assertor
- Bob B. Bobbington
- Test Subject
- A Web page located at
http://www.example.org/page.html
- Test Criterion
- Success Criteria 1.1.1 of the Web Content Accessibility Guidelines 2.0
- Test Result
- Passed
Example 2: A software application carries out automated validation of a Web page against a technical specification.
- Assertor
- The W3C Markup Validator located at
http://validator.w3.org/
- Test Subject
- The XHTML returned from a GET request to the URI
http://www.example.org/page.html at 2004-04-14T14:00:04+1000
- Test Criterion
- The validity of the XHTML code
- Test Result
- Failed, the
<li> element on line 53, char 7 was not closed.
Assertion - a statement that embodies the results of a test.
- Domain of:
- Range of: none
Example 3: Instance of an assertion expressed as an RDF/XML fragment.
<earl:Assertion rdf:about="#assertion">
<earl:assertedBy rdf:resource="#assertor"/>
<earl:subject rdf:resource="http://www.example.org/"/>
<earl:test rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36"/>
<earl:result rdf:resource="#result"/>
</earl:Assertion>
Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.
Rather than using the generic earl:Assertor class directly, it is recommended that one of the following refinements be employed:
earl:Software- Software - the assertor is a piece of software.
foaf:Agent 
- Agent - the assertor is an agent, as defined by [FOAF].
foaf:Person - Person -the assertor is a person, as defined by [FOAF].
foaf:Organization - Organization - the assertor is an organization, as defined by [FOAF].
foaf:Group - Group - the assertor is a group of agents, as defined by [FOAF].
It is recommended to provide additional information about the Assertor by using the following properties from external vocabularies:
dc:title - Human readable title for the assertor.
dc:description - Human readable description of the assertor.
foaf:name - Name of the assertor. This could be supplemented with further refinements such as
foaf:firstName or foaf:surname if the assertor is a person.
foaf:nick - Nick name of the assertor
foaf:mbox - E-mail address of the respnsible assertor, which is preferably provided in an encrypted format using the
foaf:mbox_sha1sum property.
foaf:homepage - Homepage of the assertor.
foaf:member - Member of the assertor, such as an individual in a group of testers or a tool used by an agent.
Example 4: An Assertor that is a person called Bob B. Bobbington.
<foaf:Person rdf:about="http://www.example.org/people/#bob">
<foaf:name>Bob B. Bobbington</foaf:name>
<foaf:mbox rdf:resource="mailto:bob@example.org"/>
<foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>
Example 5: An Assertor that is a piece of software called Cool Tool.
<earl:Software rdf:about="http://www.example.org/tools/#cool">
<dc:title xml:lang="en">Cool Tool</dc:title>
<dc:description xml:lang="en">My favorite tool!</dc:description>
<foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
<dct:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>
Example 6: An Assertor that is the person from example 4 using the software tool from example 5.
<foaf:Agent rdf:about="#assertor">
<dc:title xml:lang="en">Bob using Cool Tool</dc:title>
<dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
<earl:mainAssertor rdf:resource="http://www.example.org/people/#bob"/>
<foaf:member rdf:resource="http://www.example.org/tool/#cool"/>
</foaf:Agent>
Test Subject - the class of things that have been tested against some test criterion.
Rather than using the generic earl:TestSubject class directly, it is recommended that one of the following refinements be employed:
earl:Software- Software - the test subject is a piece of software being tested.
cnt:Content - Content - the test subject is a representation of the content as defined by [Content].
http:Response - HTTP Response - the test subject is the response from an HTTP server as defined by [HTTP].
foaf:Document - Document - the test subject is a document as defined by [FOAF].
- [Editor's note 1: the use of
foaf:Document is still under discussion by ERT WG, feedback on this consideration is welcome.]
- Domain of: none
- Range of:
It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:
dc:title - Human readable title for the subject.
dc:description - Human readable descriptions of the subject.
dc:date - Date on which the subject was created or identified.
dc:hasPart - Reference to another subject that are part of this subject.
dc:isPartOf - Reference to another subject of which this subject is a part of.
Example 7: A group of resources that have been tested together as a single test subject.
<earl:TestSubject rdf:about="http://www.example.org/">
<dc:title xml:lang="en">example.org Web site</dc:title>
<dc:description xml:lang="en">Each page on the example.org Web site</dc:description>
<dct:hasPart rdf:resource="http://www.example.org/style.css"/>
<dct:hasPart rdf:resource="http://www.example.org/page1.html"/>
<dct:hasPart rdf:resource="http://www.example.org/page2.html"/>
<dct:hasPart rdf:resource="http://www.example.org/image1.png"/>
<dct:hasPart rdf:resource="http://www.example.org/image2.png"/>
</earl:TestSubject>
Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.
Rather than using the generic earl:TestCriterion class directly, it is recommended that one of the following refinements be employed:
earl:TestRequirement- Test Requirement - a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criteria 1.1.1, which is evaluated using several Techniques for Success Criteria 1.1.1 and combining the results.
earl:TestCase- Test Case - an atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criteria 1.1.1.
- Domain of: none
- Range of:
It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:
dc:title - Human readable title for the test criterion.
dc:description - Human readable description of the test criterion.
dc:hasPart - Relationship to other test criteria that are part of this criterion.
dc:isPartOf - Relationship to other test criteria of which this criterion is a part of.
Example 8: Instance of a test case that is described with a title and its relationship to a test suite.
<earl:TestCase rdf:about="http://www.w3.org/TR/WCAG20-TECHS/H36">
<dc:title xml:lang="en">H36</dc:title>
<dc:description xml:lang="en">Technique H36 - Using alt attributes
on images used as submit buttons </dc:description>
<dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/"/>
<dct:hasPart rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36#H36-tests"/>
</earl:TestCase>
Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).
It is recommended to provide additional information about the Test Result by using the following properties from external vocabularies:
dc:title - Human readable title for the result.
dc:description - Human readable description of the result.
dc:date - Date on which the result was obtained (typically when the subject was tested).
Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.
<earl:TestResult rdf:about="#result">
<earl:outcome rdf:resource="http://www.w3.org/2007/earl#Fail"/>
<dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
<dc:description rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>The <code>table</code> element is not allowed to appear
inside a <code>p</code> element</p>
</div>
</dc:description>
<earl:pointer rdf:resource="#pointer"/>
<earl:info rdf:parseType="Literal" xml:lang="en">
<div xmlns="http://www.w3.org/1999/xhtml">
<p>It seems the <code>p</code> element has not been closed</p>
</div>
</earl:info>
</earl:TestResult>
[Editor's note 2: ERT WG is considering adding subclasses to parallel the provided instances, similarly to the approach taken for the OutcomeValue class. Feedback on this consideration is welcome.]
Test Mode - describes how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries.
Rather than using the generic earl:TestMode class directly, it is recommended that one of the following refinements be employed:
earl:automatic- Automatic - where the test was carried out automatically by the software tool and without any human intervention.
earl:manual- Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.
earl:semiAuto- Semi-Automatic - where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test.
earl:undisclosed- Undisclosed - where the exact testing process is undisclosed.
earl:unknownMode- Unknown - where the testing process is unknown or undetermined.
- Domain of: none
- Range of:
It is recommended to provide additional information about the Test Mode by using the following properties from external vocabularies:
dc:title - Human readable title for the test mode.
dc:description - Human readable description of the test mode.
Example 10: The assertion from example 3 was carried out in semi-automatic mode.
<earl:Assertion rdf:about="#assertion">
<earl:mode rdf:resource="http://www.w3.org/2007/earl#SemiAuto"/>
</earl:Assertion>
Outcome Value - a discrete value that describes a resulting condition from carrying out the test.
Rather than using the generic earl:OutcomeValue class directly, it is recommended that one of the following instances be employed:
earl:passed- Passed - the subject passed the test.
earl:failed- Failed - the subject failed the test.
earl:cantTell- Cannot tell - it is unclear if the subject passed or failed the test. Usually this happens when an automated test requires human judgement to make a definite decision.
earl:inapplicable- Not applicable - the test is not applicable to the subject.
earl:untested- Not tested - the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.
In cases where it is necessary to create further instances or refinements, it is recommended that one of the following subclasses be employed:
earl:Pass- Passed - the subject passed the test.
earl:Fail- Failed - the subject failed the test.
earl:CannotTell- Can not tell - it is unclear if the subject passed or failed the test. Usually this happens when an automated test requires human judgement to make a definite decision.
earl:NotApplicable- Not applicable - the test is not applicable to the subject.
earl:NotTested- Not tested - the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.
- Domain of: none
- Range of:
It is recommended to provide additional information about the Outcome Value by using the following properties from external vocabularies:
dc:title - Human readable title for the outcome value.
dc:description - Human readable description of the outcome value.
[Editor's note 3: ERT WG is considering the use of DOAP terms to describe Software in place of this term. Feedback on this consideration is welcome.]
A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to Authoring Tool Accessibility Guidelines [ATAG] or of a browser to User Agent Accessibility Guidelines [UAAG]).
It is recommended to provide information about the Software by using the following properties from external vocabularies:
dc:title - Human readable title for the software.
dc:description - Human readable description of the software.
foaf:homepage - Homepage where the software or information about the software can be obtained.
dc:hasVersion - Version number of the software application.
dc:hasPart - Reference to another software components that is part of this software.
dc:isPartOf - Reference to another software of which this software component is a part of.
Example 11: Description of a software tool.
<earl:Software rdf:about="#tool">
<dc:title xml:lang="en">Cool Tool</dc:title>
<dc:description xml:lang="en">My favorite tool!</dc:description>
<foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
<dct:hasVersion>1.0.3</dct:hasVersion>
<dct:isPartOf rdf:resource="http://example.org/tools/cms/"/>
<dct:hasPart rdf:resource="http://example.org/tools/cool/#module-1"/>
</earl:Software>
This section describes the properties defined by this document. EARL also uses properties from external vocabularies to provide additional information where necessary.
Asserted By - the assertor of an assertion.
- Domain:
earl:Assertion- Range:
earl:Assertor
Subject - the test subject of an assertion.
- Domain:
earl:Assertion- Range:
earl:TestSubject
Test - the test criterion of an assertion.
- Domain:
earl:Assertion- Range:
earl:TestCriterion
Result - the result of an assertion.
- Domain:
earl:Assertion- Range:
earl:TestResult
Mode - the mode in which the test was performed.
- Domain:
earl:Assertion- Range:
earl:TestMode
Main Assertor - the assertor that is primarily responsible for performing the test. It is a refinement of the term foaf:member defined by [FOAF].
- Domain:
earl:Assertor- Range:
earl:Assertor
Outcome - the outcome of performing the test.
- Domain:
earl:TestResult- Range:
earl:OutcomeValue
Pointer - the location within a test subject that are most relevant to a test result.
- Domain:
earl:TestResult- Range:
ptr:Pointer
Info - additional warnings or error messages in a human-readable form.
- Domain:
earl:TestResult- Range:
- Literal
[Editor's note 4: ERT WG is looking for feedback on this entire section.]
This section describes conformance with this EARL 1.0 Schema specification. It differentiates between the following entities:
- EARL Report
- An individual file or collection of files that contain EARL data
- EARL Producer
- A software tool or Web-based application that produces EARL data
- EARL Consumer
- A software tool or Web-based application that processes EARL data
Reports conforming to this EARL 1.0 Schema specification must meet the following requirements:
- A Report must be valid RDF
- A Report must contain at least one Assertion
- Every Assertion must have exactly one Assertor (referenced by
earl:assertedBy), one Test Subject (referenced by earl:subject), one Test Criterion (referenced by earl:test), one Test Result (referenced by earl:result), and at most one Test Mode (referenced by earl:mode)
- Every Assertor should be an instance of
earl:Software, foaf:Agent , foaf:Person , foaf:Organization , or foaf:Group
- Every Assertor must have at least one name (referenced by
foaf:name ) or one nick name (referenced by foaf:nick ), or exactly one title (referenced by dc:title )
- Every Test Subject should be an instance of
earl:Software, cnt:Content , http:Response , or foaf:Document
- Every Test Subject must have exactly one title (referenced by
dc:title )
- Every Test Criterion should be an instance of
earl:TestRequirement or earl:TestCase
- Every Test Criterion must have exactly one title (referenced by
dc:title )
- Every Test Result must have exactly one Outcome Value (referenced by
earl:outcome) and should have exactly one human-readable description (referenced by dc:description )
- Every Test Mode should be an instance of
earl:Automatic, earl:Manual, earl:SemiAuto, earl:Undisclosed, or earl:UnknownMode
- Every Outcome Value should be an instance of
earl:Pass, earl:Fail, earl:CannotTell, earl:NotApplicable, or earl:NotTested
- Every Software must have exactly one title (referenced by
dc:title ), and should have exactly one version number (referenced by dc:hasVersion ) and at least one homepage (referenced by foaf:homepage )
- Every instance of
cnt:Content , http:Response , or ptrs:Pointer must conform with the respective specification
- Multiple languages of the same entity (such as title, description, or homepage) are considered to be a single occurrence of the information
Producers conforming to this EARL 1.0 Schema specification must meet the following requirements:
- A Producer must generate conforming reports
- A Producer must be able to generate reports in RDF/XML serializations and should also support other RDF serializations
- A Producer must generate all of the terms summarized in Appendix A: Terms for which there is information available to it
[Editor's note 5: should conforming producers be required to fully support HTTP-in-RDF, Content-in-RDF, and Pointers-in-RDF?]
Consumers conforming to this EARL 1.0 Schema specification must meet the following requirements:
- A Consumer must process conforming reports
- A Consumer must process reports in any RDF/XML serialization and should also support other RDF serializations
- A Consumer must process all of the terms summarized in Appendix A: Terms for which there is information available to it
[Editor's note 6: should conforming consumers be required to fully support HTTP-in-RDF, Content-in-RDF, and Pointers-in-RDF?]
Reports, Producers, and Consumers that conform to this EARL 1.0 Schema are referred to as EARL Compliant or EARL Conformant. Reports, Producers, and Consumers that do not meet all conformance requirements are referred to as EARL Supporting.
Example 12: A Web accessibility evaluation tool meets all of the requirements for conforming producers. It can be referred to as an EARL Compliant evaluation tool.
Example 13: A Web content management system meets all of the requirements for conforming consumers. It can be referred to as an EARL Compliant authoring tool.
Example 14: A module in a Web content management system provides only EARL data about assertors. It can be referred to as an EARL Supporting module.
Example 15: A quality assurance tool processes only parts of EARL data to generate graphs and charts. It can be referred to as an EARL Supporting analysis tool.
This section summarizes the terms defined and used by this EARL 1.0 Schema specification.
This section provides references to related documents and specifications.
- [ATAG]
- Authoring Tool Accessibility Guidelines
- [Content]
- Representing Content in RDF
- [DC]
- Dublin Core (DC) Metadata
- [FOAF]
- Friend of a Friend (FOAF) Vocabulary
- [Guide]
- Evaluation and Report Language (EARL) 1.0 Guide
- [HTTP]
- HTTP Vocabulary in RDF
- [Pointers]
- Pointer Methods in RDF
- [RDF]
- Resource Description Framework (RDF): Concepts and Abstract Syntax
- [RDFS]
- RDF Vocabulary Description Language 1.0: RDF Schema
- [RDF-PRIMER]
- RDF Primer
- [RDF/XML]
- RDF/XML Syntax Specification
- [RDF-XML-DIFFS]
- Why RDF model is different from the XML model
- [RFC 2119]
- Key words for use in RFCs to Indicate Requirement Levels
- [UAAG]
- User Agent Accessibility Guidelines
- [WCAG]
- Web Content Accessibility Guidelines
- [XMLS]
- XML Schema Part 0: Primer
EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.
Shadi Abou-Zahra, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, and Rob Yonaitis.