Evaluation and Report Language (EARL) 1.0 Schema

2. Classes

This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:

Assertor: This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject: This can include Web content (such as Web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
Test Criterion: What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result: What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.

EARL provides flexibility to describes different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.

Examples

Example 1: A person carries out a manual evaluation of a Web page against an accessibility requirement.

Assertor: Bob B. Bobbington
Test Subject: A Web page located at http://www.example.org/page.html
Test Criterion: Success Criteria 1.1.1 of the Web Content Accessibility Guidelines 2.0
Test Result: Passed

Example 2: A software application carries out automated validation of a Web page against a technical specification.

Assertor: The W3C Markup Validator located at http://validator.w3.org/
Test Subject: The XHTML returned from a GET request to the URI http://www.example.org/page.html at 2004-04-14T14:00:04+1000
Test Criterion: The validity of the XHTML code
Test Result: Failed, the <li> element on line 53, char 7 was not closed.

2.1. Assertion Class

Assertion - a statement that embodies the results of a test.

Related Properties

Domain of:
- earl:assertedBy
- earl:subject
- earl:test
- earl:result
- earl:mode
Range of: none

Examples

Example 3: Instance of an assertion expressed as an RDF/XML fragment.

<earl:Assertion rdf:about="#assertion">
  <earl:assertedBy rdf:resource="#assertor"/>
  <earl:subject rdf:resource="http://www.example.org/"/>
  <earl:test rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36"/>
  <earl:result rdf:resource="#result"/>
</earl:Assertion>

2.2. Assertor Class

Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.

earl:Software: Software - the assertor is a piece of software.
foaf:Agent: Agent - the assertor is an agent, as defined by [FOAF].
foaf:Person: Person -the assertor is a person, as defined by [FOAF].
foaf:Organization: Organization - the assertor is an organization, as defined by [FOAF].
foaf:Group: Group - the assertor is a group of agents, as defined by [FOAF].

Related Properties

Domain of:
- earl:mainAssertor
Range of:
- earl:assertedBy
- earl:mainAssertor

It is recommended to provide additional information about the Assertor by using the following properties from external vocabularies:

dc:title: Human readable title for the assertor.
dc:description: Human readable description of the assertor.
foaf:name: Name of the assertor. This could be supplemented with further refinements such as foaf:firstName , foaf:surname , or foaf:nick if the assertor is a person.
foaf:mbox: E-mail address of the respnsible assertor, which is preferably provided in an encrypted format using the foaf:mbox_sha1sum property.
foaf:homepage: Homepage of the assertor.
foaf:member: Member of the assertor, such as an individual in a group of testers or a tool used by an agent.

Examples

Example 4: An Assertor that is a person called Bob B. Bobbington.

<foaf:Person rdf:about="http://www.example.org/people/#bob">
  <foaf:name>Bob B. Bobbington</foaf:name>
  <foaf:mbox rdf:resource="mailto:bob@example.org"/>
  <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>

Example 5: An Assertor that is a piece of software called Cool Tool.

<earl:Software rdf:about="http://www.example.org/tools/#cool">
  <dc:title xml:lang="en">Cool Tool</dc:title>
  <dc:description xml:lang="en">My favorite tool!</dc:description>
  <foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
  <dct:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>

Example 6: An Assertor that is the person from example 4 using the software tool from example 5.

<foaf:Agent rdf:about="#assertor">
  <dc:title xml:lang="en">Bob using Cool Tool</dc:title>
  <dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
  <earl:mainAssertor rdf:resource="http://www.example.org/people/#bob"/>
  <foad:member rdf:resource="http://www.example.org/tool/#cool"/>
</foaf:Agent>

2.3. TestSubject Class

Test Subject - the class of things that have been tested against some test criterion.

earl:Software: Software - the test subject is a piece of software being tested.
cnt:Content: Content - the test subject is a representation of the content as defined by [Content-in-RDF].
http:Response: HTTP Response - the test subject is the response from an HTTP server as defined by [HTTP-in-RDF].
foaf:Document: Document - the test subject is a document as defined by [FOAF].; [Editor's Note: the use of foaf:Document is still under discussion by ERT WG, feedback on this consideration is welcome.]

Related Properties

Domain of: none
Range of:
- earl:subject

It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:

dc:title: Human readable title for the subject.
dc:description: Human readable description of the subject.
dc:date: Date on which the subject was created or identified.
dc:hasPart: Reference to another subject that are part of this subject.
dc:isPartOf: Reference to another subject of which this subject is a part of.

Examples

Example 7: A group of resources that have been tested together as a single test subject.

<earl:TestSubject rdf:about="http://www.example.org/">
  <dc:title xml:lang="en">example.org Web site</dc:title> 
  <dc:description xml:lang="en">Each page on the example.org Web site</dc:description> 
  <dct:hasPart rdf:resource="http://www.example.org/style.css"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page1.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page2.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/image1.png"/> 
  <dct:hasPart rdf:resource="http://www.example.org/image2.png"/> 
</earl:TestSubject>

2.4. TestCriterion Class

Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.

earl:TestRequirement: Test Requirement - a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criteria 1.1.1, which is evaluated using several Techniques for Success Criteria 1.1.1 and combining the results.
earl:TestCase: Test Case - an atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criteria 1.1.1.

Related Properties

Domain of: none
Range of:
- earl:test

It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:

dc:title: Human readable title for the test criterion.
dc:description: Human readable description of the test criterion.
dc:hasPart: Relationship to other test criteria that are part of this criterion.
dc:isPartOf: Relationship to other test criteria of which this criterion is a part of.

Examples

Example 8: Instance of a test case that is described with a title and its relationship to a test suite.

<earl:TestCase rdf:about="http://www.w3.org/TR/WCAG20-TECHS/H36">
  <dc:title xml:lang="en">H36</dc:title>
  <dc:description xml:lang="en">Technique H36 - Using alt attributes on images used as submit buttons </dc:description>
  <dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/"/>
  <dct:hasPart rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36#H36-tests"/>
</earl:TestCase>

2.5. TestResult Class

Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).

Related Properties

Domain of:
- earl:info
- earl:mode
- earl:outcome
- earl:pointer
Range of:
- earl:result

It is recommended to provide additional information about the Test Result by using the following properties from external vocabularies:

dc:title: Human readable title for the result.
dc:description: Human readable description of the result.
dc:date: Date on which the result was obtained (typically when the subject was tested).

Examples

Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.

<earl:TestResult rdf:about="#result">
  <earl:outcome rdf:resource="http://www.w3.org/2007/earl#Fail"/>
  <dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>The <code>table</code> element is not allowed to appear
        inside a <code>p</code> element</p>
    </div>
  </dc:description>
  <earl:pointer rdf:resource="#pointer"/>
  <earl:info rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>It seems the <code>p</code> element has not been closed</p>
    </div>
  </earl:info>
</earl:TestResult>

2.6. TestMode Class

The Test Mode described how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries.

earl:Automatic: Automatic - where the test was carried out automatically by the software tool and without any human intervention.
earl:Manual: Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.
earl:SemiAuto: Semi-Automatic - where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test.
earl:Undisclosed: Undisclosed - where the exact testing process is undisclosed.
earl:Unknown: Unknown - where the testing process is unknown or undetermined.

Related Properties

Domain of: none
Range of:
- earl:mode

It is recommended to provide additional information about the Test Mode by using the following properties from external vocabularies:

dc:title: Human readable title for the test mode.
dc:description: Human readable description of the test mode.

Examples

Example 10: The assertion from example 3 was carried out in semi-automatic mode.

<earl:Assertion rdf:about="#assertion">
  <earl:mode rdf:resource="http://www.w3.org/2007/earl#SemiAuto"/> 
</earl:Assertion>

2.7. OutcomeValue Class

Outcome Value - a discrete value that describes a resulting condition from carrying out the test.

earl:Pass: Passed - the subject passed the test.
earl:Fail: Failed - the subject failed the test.
earl:CannotTell: Can not tell - it is unclear if the subject passed or failed the test. Usually this happens when an automated test requires human judgement to make a definite decision.
earl:NotApplicable: Not applicable - the test is not applicable to the subject.
earl:NotTested: Not tested - the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.

Related Properties

Domain of: none
Range of:
- earl:outcome

It is recommended to provide additional information about the Outcome Value by using the following properties from external vocabularies:

dc:title: Human readable title for the outcome value.
dc:description: Human readable description of the outcome value.

2.8. Software Class

[Editor's note: ERT WG is considering the use of DOAP terms to describe Software in place of this term. Feedback on this consideration is welcome.]

A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to the Authoring Tool Accessibility Guidelines or of a browser to the User Agent Accessibility Guidelines).

Related Properties

It is recommended to provide information about the Software by using the following properties from external vocabularies:

dc:title: Human readable title for the software.
dc:description: Human readable description of the software.
foaf:homepage: Homepage where the software or information about the software can be obtained.
dc:hasVersion: Version number of the software application.
dc:hasPart: Reference to another software components that is part of this software.
dc:isPartOf: Reference to another software of which this software component is a part of.

Examples

Example 11: Description of a software tool.

<earl:Software rdf:about="#tool">
  <dc:title xml:lang="en">Cool Tool</dc:title>
  <dc:description xml:lang="en">My favorite tool!</dc:description>
  <foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
  <dct:hasVersion>1.0.3</dct:hasVersion>
  <dct:isPartOf rdf:resource="http://example.org/tools/cms/"/>
  <dct:hasPart rdf:resource="http://example.org/tools/cool/#module-1"/>
</earl:Software>

3. Properties

This section describes the properties defined by this document. EARL also uses properties from external vocabularies to provide additional information where necessary.

3.1. assertedBy Property

Asserted By - the assertor of an assertion.

Domain:: earl:Assertion
Range:: earl:Assertor

3.2. subject Property

Subject - the test subject of an assertion.

Domain:: earl:Assertion
Range:: earl:TestSubject

3.3. test Property

Test - the test criterion of an assertion.

Domain:: earl:Assertion
Range:: earl:TestCriterion

3.4. result Property

Result - the result of an assertion.

Domain:: earl:Assertion
Range:: earl:TestResult

3.5. mode Property

Mode - the mode in which the test was performed.

Domain:: earl:Assertion
Range:: earl:TestMode

3.6. mainAssertor Property

[Editor's note: the concept of mainAssertor and the dropped helpAssertor needs to be revised since it does not validate with FOAF.]

Main Assertor - the assertor that is primarily responsible for performing the test. It is a refinement of the term foaf:member defined by [FOAF].

Domain:: earl:Assertor
Range:: earl:Assertor

3.7. outcome Property

Outcome - the outcome of performing the test.

Domain:: earl:TestResult
Range:: earl:OutcomeValue

3.8. pointer Property

Pointer - the locations within the test subject that are most relevant to the outcome of the test.

Domain:: earl:TestResult
Range:: none

3.9. info Property

Info - additional warnings or error messages in a human-readable form.

Domain:: earl:TestResult
Range:: Literal

4. Conformance

This section defines the conformance to the EARL 1.0 specification. Applications conform to EARL 1.0 when the following applies:

Producing EARL 1.0 reports: Applications must produce EARL 1.0 reports in at least RDF/XML format and should support other RDF formats where possible. Applications must produce EARL 1.0 reports that observe the Validation requirements.
Processing EARL 1.0 reports: Applications must process EARL 1.0 reports in at least RDF/XML format, and should support other RDF formats where possible. Applications must process EARL 1.0 reports that observe the Validation requirements.

Validation

Valid EARL 1.0 reports must validate against the RDF grammar specification. In addition, the following restrictions apply to valid EARL 1.0 reports:

Class	Property	Restriction
`earl:Assertion`	`earl:assertedBy`	exactly one assertor per assertion
	`earl:subject`	exactly one test subject per assertion
	`earl:test`	exactly one test criterion per assertion
	`earl:result`	exactly one test result per assertion
	`earl:mode`	at most one test mode per assertion
`earl:Assertor`	`foaf:name` , `dc:title`	exactly one title (per language) or name per assertor
	`dc:description`	exactly one description (per language) per assertor
	`earl:mainAssertor`	none [Editor's note: see issue related to mainAssertor]
	`foaf:homepage`	none
`foaf:Agent`	`foaf:name` , `foaf:nick`	at least one name or nick name per agent
	`foaf:mbox`	none
	`foaf:mbox_sha1sum`	none
	`foaf:homepage`	none
`foaf:Person`	`foaf:name` , `foaf:firstName` , `foaf:surname` , `foaf:nick`	at least one name, first name, surname, or nick name per person
	`foaf:mbox`	none
	`foaf:mbox_sha1sum`	none
	`foaf:homepage`	none
`foaf:Organization`	`foaf:name` , `foaf:nick`	at least one name or nick name per organization
	`foaf:mbox`	none
	`foaf:mbox_sha1sum`	none
	`foaf:homepage`	none
`foaf:Group`	`foaf:name` , `foaf:nick`	at least one name or nick name per organization
	`foaf:member`	none
	`foaf:mbox`	none
	`foaf:mbox_sha1sum`	none
	`foaf:homepage`	none
@@@	@@@	@@@

Appendix A: References (Non-Normative)

[Editor's note: this section needs revision.]

[DC]: The Dublin Core Metadata Element Set - DC Recommendation, 20 December 2004.
http://www.dublincore.org/documents/dces/
[DCT]: The Dublin Core Metadata Terms - DC Recommendation, 13 June 2005.
http://www.dublincore.org/documents/dcmi-terms/
[FOAF]: FOAF Vocabulary Specification - Working Draft, 3 June 2005.
http://xmlns.com/foaf/0.1/
[Guide]: Evaluation and Report Language (EARL) 1.0 Guide - Editors' Draft 14 December 2005. C. Velasco, J. Koch, S. Abou-Zahra eds.
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Guide-20051214
[HTTP]: HTTP Vocabulary in RDF - W3C Working Draft 20 December 2006. J. Koch, C. Velasco, S. Abou-Zahra eds.
http://www.w3.org/TR/HTTP-in-RDF/
[OWL]: OWL Web Ontology Language - W3C Recommendation, 10 February 2004.
http://www.w3.org/TR/owl-features/
[Pointers]: Pointer Methods in RDF - Editors' Draft 22 February 2007. J. Ley eds.
http://www.w3.org/WAI/ER/Pointers/WD-Pointers-20070222
[RDF]: RDF/XML Syntax Specification (Revised) - W3C Recommendation, 10 February 2004. D. Beckett ed.
http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/
[RDF-PRIMER]: RDF Primer - W3C Recommendation, 10 February 2004. F. Manola, E.Miller eds.
http://www.w3.org/TR/rdf-primer/
[RDFS]: RDF Vocabulary Description Language 1.0: RDF Schema - W3C Recommendation, 10 February 2004. D. Brickley, R.V. Guha eds.
http://www.w3.org/TR/rdf-schema/
[RDF-XML-DIFFS]: Why RDF model is different from the XML model - Paper, September 1998. T. Berners-Lee.
http://www.w3.org/DesignIssues/RDF-XML
[RFC2119]: Key words for use in RFCs to Indicate Requirement Levels - IETF RFC, March 1997.
http://www.ietf.org/rfc/rfc2119.txt
[WCAG10]: Web Content Accessibility Guidelines 1.0 - W3C Recommendation, 5 May 1999. W. Chisholm, I. Jacobs, G. Vanderheiden eds.
http://www.w3.org/TR/WCAG10/
[XMLS]: XML Schema Part 0: Primer Second Edition - W3C Recommendation, 28 October 2004. D. Fallside, P. Walmsley erds.
http://www.w3.org/TR/xmlschema-0/

Appendix B: Contributors (Non-Normative)

EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.

Contributors to this and/or previous Working Drafts

Shadi Abou-Zahra, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, and Rob Yonaitis.