[contents]


Abstract

This document describes the formal schema of the Evaluation and Report Language (EARL) 1.0. EARL is a vocabulary, the terms of which are defined across a set of specifications and technical notes, and that is used to describe test results. The primary motivation for developing this vocabulary is to facilitate the exchange of test results between Web accessibility evaluation tools in a vendor-neutral and platform-independent format. It also provides reusable terms for generic quality assurance and validation purposes.

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

This 8 September 2009 Editors Draft [@@ September 2009 Last Call Working Draft] of the Evaluation and Report Language (EARL) 1.0 Schema is an update of the previous EARL 1.0 Working Draft of 28 April 2009. It meets the requirements specified in the Requirements for the Evaluation and Report Language (EARL) 1.0, and incorporates all comments received. In particular, this draft implements the decisions of the Evaluation and Repair Tools Working Group (ERT WG) at its face to face meeting in November 2008, and addresses comments received on the stabilization Working Draft of 28 April 2009. This document is intended to be published and maintained as a W3C Recommendation after review and refinement.

The Evaluation and Repair Tools Working Group (ERT WG) believes to have addressed all issues brought forth through previous Working Draft iterations. The Working Group encourages feedback about this document, Evaluation and Report Language (EARL) 1.0 Schema, by developers and researchers who have interest in software-supported evaluation and validation of Web sites, and by developers and researchers who have interest in Semantic Web technologies for content description, annotation, and adaptation. In particular, the Working Group is looking for feedback on the following items which are also highlighted within the document:

Please send comments on this Evaluation and Report Language (EARL) 1.0 Schema document by [@@ October 2009] to public-earl10-comments@w3.org (publicly visible mailing list archive).

Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document has been produced by the Evaluation and Repair Tools Working Group (ERT WG) as part of the Web Accessibility Initiative (WAI) Technical Activity.

This document was produced by a group operating under the 5 February 2004 W3C Patent Policy. W3C maintains a public list of any patent disclosures made in connection with the deliverables of the group; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy.


Table of Contents

  1. Introduction
  2. Classes
  3. Properties
  4. Conformance

Appendices

  1. Terms
  2. References
  3. Contributors

1. Introduction

The Evaluation and Report Language (EARL) defines a vocabulary for expressing test results. It enables any person, software application, or organization to assert test results for any test subject tested against any set of criteria. The test subject might be a Web site, an authoring tool, a user agent, or some other entity. The set of criteria may be accessibility guidelines, formal grammars, or other types of quality assurance requirements. Thus, EARL is flexible with regard to the contexts in which it can be applied.

This document provides the core schema of EARL. Other parts of the EARL suite of specifications include:

The Evaluation and Report Language (EARL) Overview provides a brief introduction to EARL.

EARL is not a comprehensive vocabulary for describing test procedures, test criteria, or test requirements but, rather, for describing the outcomes from such testing. EARL can be supplemented by test description vocabularies or other vocabularies for different aspects of the testing cycle.

1.1. Audience of this Document

The assumed audience of this specification is developers who are implementing EARL in software or processes, or those who are seeking to understand the ideas, models, or properties and classes used in the EARL vocabulary. Readers who would like more introductory material for the language with explanation of its foreseen use cases are referred to the Evaluation and Report Language (EARL) 1.0 Guide.

This document assumes that the reader is familiar with the Resource Description Framework (RDF) and can read its XML serialization. Readers who wish to understand more about RDF should read a general introduction or the RDF Primer [RDF-PRIMER].

1.2. Document conventions

Keywords

The keywords must, required, recommended, should, may, and optional in this document are used in accordance with RFC 2119 [RFC 2119].

Namespaces

The RDF representation of the vocabulary defined by this document uses the namespace http://www.w3.org/ns/earl#. The prefix earl is used throughout this document to denote this namespace. Other prefixes used throughout this document include:

2. Classes

This section describes the classes defined by this document. Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:

Assertor
This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject
This can include Web content (such as Web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
Test Criterion
What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result
What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.

EARL provides flexibility to describe different types of assertions, such as those carried out by automated testing tools or by human evaluators, or those made about generic testing requirements or specific test cases.

Examples

Example 1: A person carries out a manual evaluation of a Web page against an accessibility requirement.

Assertor
Bob B. Bobbington
Test Subject
A Web page located at http://www.example.org/page.html
Test Criterion
Success Criteria 1.1.1 of the Web Content Accessibility Guidelines 2.0
Test Result
Passed

Example 2: A software application carries out automated validation of a Web page against a technical specification.

Assertor
The W3C Markup Validator located at http://validator.w3.org/
Test Subject
The XHTML returned from a GET request to the URI http://www.example.org/page.html at 2004-04-14T14:00:04+1000
Test Criterion
The validity of the XHTML code
Test Result
Failed, the <li> element on line 53, char 7 was not closed.

2.1. Assertion Class

Assertion - a statement that embodies the results of a test.

Related Properties

Examples

Example 3: Instance of an assertion expressed as an RDF/XML fragment.

<earl:Assertion rdf:about="#assertion">
  <earl:assertedBy rdf:resource="#assertor"/>
  <earl:subject rdf:resource="http://www.example.org/"/>
  <earl:test rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36"/>
  <earl:result rdf:resource="#result"/>
</earl:Assertion>

2.2. Assertor Class

Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.

Related Classes

Rather than specifying only an earl:Assertor type, it is recommended that one of the following types be employed in addition:

earl:Software
Software - the assertor is a piece of software.
foaf:Agent external link
Agent - the assertor is an agent, as defined by [FOAF].
foaf:Person external link
Person -the assertor is a person, as defined by [FOAF].
foaf:Organization external link
Organization - the assertor is an organization, as defined by [FOAF].
foaf:Group external link
Group - the assertor is a group of agents, as defined by [FOAF].

Related Properties

It is recommended to provide additional information about the Assertor by using the following properties from external vocabularies:

dct:title external link
Human readable title for the assertor.
dct:description external link
Human readable description of the assertor.
foaf:name external link
Name of the assertor. This could be supplemented with further refinements such as foaf:firstName external link or foaf:surname external link if the assertor is a person.
foaf:nick external link
Nick name of the assertor
foaf:mbox external link
E-mail address of the respnsible assertor, which is preferably provided in an encrypted format using the foaf:mbox_sha1sum external link property.
foaf:homepage external link
Homepage of the assertor.
foaf:member external link
Member of the assertor, such as an individual in a group of testers or a tool used by an agent.

Examples

Example 4: An Assertor that is a person called Bob B. Bobbington.

<foaf:Person rdf:about="http://www.example.org/people/#bob">
  <foaf:name>Bob B. Bobbington</foaf:name>
  <foaf:mbox rdf:resource="mailto:bob@example.org"/>
  <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>

Example 5: An Assertor that is a piece of software called Cool Tool.

<earl:Software rdf:about="http://www.example.org/tools/#cool">
  <dct:title xml:lang="en">Cool Tool</dct:title>
  <dct:description xml:lang="en">My favorite tool!</dct:description>
  <foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
  <dct:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>

Example 6: An Assertor that is the person from example 4 using the software tool from example 5.

<foaf:Agent rdf:about="#assertor">
  <dct:title xml:lang="en">Bob using Cool Tool</dct:title>
  <dct:description xml:lang="en">Bob doing semi-automated testing</dct:description>
  <earl:mainAssertor rdf:resource="http://www.example.org/people/#bob"/>
  <foaf:member rdf:resource="http://www.example.org/tool/#cool"/>
</foaf:Agent>

2.3. TestSubject Class

Test Subject - the class of things that have been tested against some test criterion.

Related Classes

Rather than specifying only an earl:TestSubject type, it is recommended that one of the following types be employed in addition:

earl:Software
Software - the test subject is a piece of software being tested.
cnt:Content external link
Content - the test subject is a representation of the content as defined by [Content].
http:Response external link
HTTP Response - the test subject is the response from an HTTP server as defined by [HTTP].
foaf:Document external link
Document - the test subject is a document, such as electronic file, as defined by [FOAF].
[Editor's note 1: the use of foaf:Document is still under discussion by ERT WG, feedback on this consideration is welcome.]

Related Properties

It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:

dct:title external link
Human readable title for the subject.
dct:description external link
Human readable descriptions of the subject.
dct:date external link
Date on which the subject was created or identified.
dct:hasPart external link
Reference to another subject that are part of this subject.
dct:isPartOf external link
Reference to another subject of which this subject is a part of.

Examples

Example 7: A group of resources that have been tested together as a single test subject.

<earl:TestSubject rdf:about="http://www.example.org/">
  <dct:title xml:lang="en">example.org Web site</dct:title> 
  <dct:description xml:lang="en">Each page on the example.org Web site</dct:description> 
  <dct:hasPart rdf:resource="http://www.example.org/style.css"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page1.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page2.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/image1.png"/> 
  <dct:hasPart rdf:resource="http://www.example.org/image2.png"/> 
</earl:TestSubject>

2.4. TestCriterion Class

Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.

Related Classes

Rather than specifying only an earl:TestCriterion type, it is recommended that one of the following types be employed in addition:

earl:TestRequirement
Test Requirement - a higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criteria 1.1.1, which is evaluated using several Techniques for Success Criteria 1.1.1 and combining the results.
earl:TestCase
Test Case - an atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criteria 1.1.1.

Related Properties

It is recommended to provide additional information about the Test Subject by using the following properties from external vocabularies:

dct:title external link
Human readable title for the test criterion.
dct:description external link
Human readable description of the test criterion.
dct:hasPart external link
Relationship to other test criteria that are part of this criterion.
dct:isPartOf external link
Relationship to other test criteria of which this criterion is a part of.

Examples

Example 8: Instance of a test case that is described with a title and its relationship to a test suite.

<earl:TestCase rdf:about="http://www.w3.org/TR/WCAG20-TECHS/H36">
  <dct:title xml:lang="en">H36</dct:title>
  <dct:description xml:lang="en">Technique H36 - Using alt attributes 
    on images used as submit buttons </dct:description>
  <dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/"/>
  <dct:hasPart rdf:resource="http://www.w3.org/TR/WCAG20-TECHS/H36#H36-tests"/>
</earl:TestCase>

2.5. TestResult Class

Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).

Related Properties

It is recommended to provide additional information about the Test Result by using the following properties from external vocabularies:

dct:title external link
Human readable title for the result.
dct:description external link
Human readable description of the result.
dct:date external link
Date on which the result was obtained (typically when the subject was tested).

Examples

Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.

<earl:TestResult rdf:about="#result">
  <earl:outcome rdf:resource="http://www.w3.org/ns/earl#failed"/>
  <dct:title xml:lang="en">Invalid Markup (code #353)</dct:title>
  <dct:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>The <code>table</code> element is not allowed to appear
        inside a <code>p</code> element</p>
    </div>
  </dct:description>
  <earl:pointer rdf:resource="#pointer"/>
  <earl:info rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>It seems the <code>p</code> element has not been closed</p>
    </div>
  </earl:info>
</earl:TestResult>

2.6. TestMode Class

Test Mode - describes how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries.

Related Instances

Where applicable it is recommended to use one of the following instances of earl:TestMode, to categorize the mode in which the test was carried out:

earl:automatic
Automatic - where the test was carried out automatically by the software tool and without any human intervention.
earl:manual
Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.
earl:semiAuto
Semi-Automatic - where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test.
earl:undisclosed
Undisclosed - where the exact testing process is undisclosed.
earl:unknownMode
Unknown - where the testing process is unknown or undetermined.

Related Properties

It is recommended to provide additional information about the Test Mode by using the following properties from external vocabularies:

dct:title external link
Human readable title for the test mode.
dct:description external link
Human readable description of the test mode.

Examples

Example 10: The assertion from example 3 was carried out in semi-automatic mode.

<earl:Assertion rdf:about="#assertion">
  <earl:mode rdf:resource="http://www.w3.org/ns/earl#semiAuto"/> 
</earl:Assertion>

2.7. OutcomeValue Class

[Editor's note 2: ERT WG is looking for feedback on its approach to provide both subclasses as well as instances for the OutcomeValue class. This approach has been selected to allow the creation of application-specific values (such as foo:nearlyPassed or bar:resultNeedsApproval), yet provide a basic level of categorization to facilitate the exchange and comparison of results from different applications.]

Outcome Value - a value or expression that describes a resulting condition from carrying out the test.

Related Instances

Where applicable it is recommended to use one of the following instances of earl:OutcomeValue, to categorize the outcome of carrying out the test:

earl:passed
Passed - the subject passed the test.
earl:failed
Failed - the subject failed the test.
earl:cantTell
Cannot tell - it is unclear if the subject passed or failed the test.
earl:inapplicable
Inapplicable - the test is not applicable to the subject.
earl:untested
Untested - the test has not been carried out.

Related Classes

In cases where it is necessary to create further instances of earl:OutcomeValue, it is recommended that one of the following types be employed in addition:

earl:Pass
Pass - the class of outcomes to denote passing a test. Subclasses may include ordinal, nominal, or continuous values or expressions.
earl:Fail
Fail - the class of outcomes to denote failing a test. Subclasses may include ordinal, nominal, or continuous values or expressions.
earl:CannotTell
Undetermined - the class of outcomes to denote an undetermined outcome. Usually this happens when an automated test requires human judgement to make a definite decision.
earl:NotApplicable
Not applicable - the class of outcomes to denote the test is not applicable. This could be due to a mismatch between the test and the subject or for any other reason.
earl:NotTested
Not tested - the class of outcomes to denote the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.

Related Properties

It is recommended to provide additional information about the Outcome Value by using the following properties from external vocabularies:

dct:title external link
Human readable title for the outcome value.
dct:description external link
Human readable description of the outcome value.

2.8. Software Class

[Editor's note 3: ERT WG is considering the use of DOAP terms to describe Software in place of this term. Feedback on this consideration is welcome.]

A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to Authoring Tool Accessibility Guidelines [ATAG] or of a browser to User Agent Accessibility Guidelines [UAAG]).

Related Properties

It is recommended to provide information about the Software by using the following properties from external vocabularies:

dct:title external link
Human readable title for the software.
dct:description external link
Human readable description of the software.
foaf:homepage external link
Homepage where the software or information about the software can be obtained.
dct:hasVersion external link
Version number of the software application.
dct:hasPart external link
Reference to another software components that is part of this software.
dct:isPartOf external link
Reference to another software of which this software component is a part of.

Examples

Example 11: Description of a software tool.

<earl:Software rdf:about="#tool">
  <dct:title xml:lang="en">Cool Tool</dct:title>
  <dct:description xml:lang="en">My favorite tool!</dct:description>
  <foaf:homepage rdf:resource="http://example.org/tools/cool/"/>
  <dct:hasVersion>1.0.3</dct:hasVersion>
  <dct:isPartOf rdf:resource="http://example.org/tools/cms/"/>
  <dct:hasPart rdf:resource="http://example.org/tools/cool/#module-1"/>
</earl:Software>

3. Properties

This section describes the properties defined by this document. EARL also uses properties from external vocabularies to provide additional information where necessary.

3.1. assertedBy Property

Asserted By - the assertor of an assertion.

Domain:
earl:Assertion
Range:
earl:Assertor

3.2. subject Property

Subject - the test subject of an assertion.

Domain:
earl:Assertion
Range:
earl:TestSubject

3.3. test Property

Test - the test criterion of an assertion.

Domain:
earl:Assertion
Range:
earl:TestCriterion

3.4. result Property

Result - the result of an assertion.

Domain:
earl:Assertion
Range:
earl:TestResult

3.5. mode Property

Mode - the mode in which the test was performed.

Domain:
earl:Assertion
Range:
earl:TestMode

3.6. mainAssertor Property

Main Assertor - the assertor that is primarily responsible for performing the test. It is a refinement of the term foaf:member external link defined by [FOAF].

Domain:
earl:Assertor
Range:
earl:Assertor

3.7. outcome Property

Outcome - the outcome of performing the test.

Domain:
earl:TestResult
Range:
earl:OutcomeValue

3.8. pointer Property

Pointer - the location within a test subject that are most relevant to a test result.

Domain:
earl:TestResult
Range:
ptr:Pointer external link

3.9. info Property

Info - additional warnings or error messages in a human-readable form.

Domain:
earl:TestResult
Range:
Literal

4. Conformance

[Editor's note 4: ERT WG is looking for feedback on this entire section.]

The following entities can conform to the Evaluation and Report Language (EARL):

EARL Report
An individual file or collection of files that contain related EARL data
EARL Producer
A software tool or Web-based application that produces EARL data
EARL Consumer
A software tool or Web-based application that processes EARL data

Each of these entities can conform to EARL at one of the following conformance levels:

EARL 1.0 Core
Conforms to this EARL 1.0 Schema specification.
EARL 1.0 HTTP
Conforms to this EARL 1.0 Schema specification, HTTP Vocabulary in RDF 1.0, and Representing Content in RDF 1.0.
EARL 1.0 Pointers
Conforms to this EARL 1.0 Schema specification and Pointer Methods in RDF 1.0.
EARL 1.0 Full
Conforms to this EARL 1.0 Schema specification, HTTP Vocabulary in RDF 1.0, Representing Content in RDF 1.0, and Pointer Methods in RDF 1.0.

4.1. Conforming Reports

EARL Reports can conform to EARL 1.0 Core, EARL 1.0 HTTP, EARL 1.0 Pointers, or EARL 1.0 Full.

EARL 1.0 Core Reports

EARL Reports conforming to EARL 1.0 Core must meet the following requirements:

  1. Reports must be valid RDF
  2. Reports must contain at least one Assertion
  3. Every Assertion must have exactly one Assertor (referenced by earl:assertedBy), one Test Subject (referenced by earl:subject), one Test Criterion (referenced by earl:test), one Test Result (referenced by earl:result), and at most one Test Mode (referenced by earl:mode)
  4. Every Assertor should be an instance of earl:Software, foaf:Agent external link, foaf:Person external link, foaf:Organization external link, or foaf:Group external link
  5. Every Assertor must have at least one name (referenced by foaf:name external link) or one nick name (referenced by foaf:nick external link), or exactly one title (referenced by dct:title external link)
  6. Every Test Subject should be an instance of earl:Software, cnt:Content external link, http:Response external link, or foaf:Document external link
  7. Every Test Subject must have exactly one title (referenced by dct:title external link)
  8. Every Test Criterion should be an instance of earl:TestRequirement or earl:TestCase
  9. Every Test Criterion must have exactly one title (referenced by dct:title external link)
  10. Every Test Result must have exactly one Outcome Value (referenced by earl:outcome) and should have exactly one human-readable description (referenced by dct:description external link)
  11. Every Test Mode should be the instance of earl:automatic, earl:manual, earl:semiAuto, earl:undisclosed, or earl:unknownMode
  12. Every Outcome Value should be an instance of earl:Pass, earl:Fail, earl:CannotTell, earl:NotApplicable, or earl:NotTested
  13. Every Software must have exactly one title (referenced by dct:title external link), and should have exactly one version number (referenced by dct:hasVersion external link) and at least one homepage (referenced by foaf:homepage external link)
  14. Every instance of cnt:Content external link, http:Response external link, or ptrs:Pointer external link must conform with the respective specification

Valid reasons for not adhering to should requirements can include situations in which the EARL Producers have no information available to generate such terms, or if generating such terms poses substantial conflicts such as a reduced efficacy of the application, or privacy or security hazards.

Note: subclasses or subproperties of terms share the same type. They are therefore considered to be equivalent entities in adhering to any of the above requirements. Also, instances in multiple languages of the same entity (such as a title, description, or homepage) are considered to be a single occurrence of the entity.

EARL 1.0 HTTP Reports

EARL Reports conforming to EARL 1.0 HTTP must meet the following requirements:

  1. Reports must conform to EARL 1.0 Core Reports
  2. Every graph of HTTP Vocabulary in RDF 1.0 terms must be a conforming HTTP-in-RDF graph
  3. Every graph of Representing Content in RDF 1.0 terms must be a conforming Content-in-RDF graph

EARL 1.0 Pointers Reports

EARL Reports conforming to EARL 1.0 Pointers must meet the following requirements:

  1. Reports must conform to EARL 1.0 Core Reports
  2. Every graph of Pointer Methods in RDF 1.0 terms must be a conforming Pointers-in-RDF graph

EARL 1.0 Full Reports

EARL Reports conforming to EARL 1.0 Full must meet the following requirements:

  1. Reports must conform to EARL 1.0 Core Reports
  2. Reports must conform to EARL 1.0 HTTP Reports
  3. Reports must conform to EARL 1.0 Pointers Reports

4.2. Conforming Producers

EARL Producers can conform to EARL 1.0 Core, EARL 1.0 HTTP, EARL 1.0 Pointers, or EARL 1.0 Full.

EARL 1.0 Core Producers

EARL Producers conforming to EARL 1.0 Core must meet the following requirements:

  1. Producers must generate conforming EARL 1.0 Core Reports
  2. Producers must generate reports in RDF/XML serializations and should also support other RDF serializations
  3. Producers must generate all of the terms defined by this EARL 1.0 Schema for which there is information available to it

EARL 1.0 HTTP Producers

EARL Producers conforming to EARL 1.0 HTTP must meet the following requirements:

  1. Producers must conform to EARL 1.0 Core Producers
  2. Producers must generate conforming EARL 1.0 HTTP Reports
  3. Producers must generate all of the terms defined by HTTP Vocabulary in RDF 1.0 for which there is information available to it
  4. Producers must generate all of the terms defined by Content in RDF 1.0 for which there is information available to it

EARL 1.0 Pointers Producers

EARL Producers conforming to EARL 1.0 Pointers must meet the following requirements:

  1. Producers must conform to EARL 1.0 Core Producers
  2. Producers must generate conforming EARL 1.0 Pointer Reports
  3. Producers must generate all of the terms defined by Pointer Methods in RDF 1.0 for which there is information available to it

EARL 1.0 Full Producers

EARL Producers conforming to EARL 1.0 Full must meet the following requirements:

  1. Producers must conform to EARL 1.0 Core Producers
  2. Producers must conform to EARL 1.0 HTTP Producers
  3. Producers must conform to EARL 1.0 Pointers Producers

4.2. Conforming Consumers

EARL Consumers can conform to EARL 1.0 Core, EARL 1.0 HTTP, EARL 1.0 Pointers, or EARL 1.0 Full.

EARL 1.0 Core Consumers

EARL Consumers conforming to EARL 1.0 Core must meet the following requirements:

  1. Consumers must process conforming EARL 1.0 Core Reports
  2. Consumers must process reports in any RDF/XML serialization and should also support other RDF serializations
  3. Consumers must process all of the terms defined by this EARL 1.0 Schema for which there is information available to it

EARL 1.0 HTTP Consumers

EARL Consumers conforming to EARL 1.0 HTTP must meet the following requirements:

  1. Consumers must conform to EARL 1.0 Core Consumers
  2. Consumers must process conforming EARL 1.0 HTTP Reports
  3. Consumers must process all of the terms defined by HTTP Vocabulary in RDF 1.0 for which there is information available to it
  4. Consumers must process all of the terms defined by Content in RDF 1.0 for which there is information available to it

EARL 1.0 Pointers Consumers

EARL Consumers conforming to EARL 1.0 Pointers must meet the following requirements:

  1. Consumers must conform to EARL 1.0 Core Consumers
  2. Consumers must process conforming EARL 1.0 Pointers Reports
  3. Consumers must process all of the terms defined by Pointer Methods in RDF 1.0 for which there is information available to it

EARL 1.0 Full Consumers

EARL Consumers conforming to EARL 1.0 Full must meet the following requirements:

  1. Consumers must conform to EARL 1.0 Core Consumers
  2. Consumers must conform to EARL 1.0 HTTP Consumers
  3. Consumers must conform to EARL 1.0 Pointers Consumers

4.4. Partial Conformance

EARL Reports, EARL Producers, or EARL Consumers are partially conformant to EARL 1.0 when they conform to complete sub-sets of the Evaluation and Report Language. This includes:

The purpose of partial conformance is to facilitate the exchange of EARL data between software or software components to support the production or the consumption of conforming EARL reports. Vocabularies and software that use EARL or parts of EARL for other purposes than the exchange of test results must not claim any level of conformance to EARL.


Appendix A: Terms

This section summarizes the terms defined and used by this EARL 1.0 Schema specification.

Classes

EARL 1.0 Classes
Class Name Label Comment Refinements Related Properties
earl:Assertion Assertion a statement that embodies the results of a test -
earl:Assertor Assertor an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively
earl:TestSubject Test Subject the class of things that have been tested against some test criterion
earl:TestCriterion Test Criterion a testable statement, usually one that can be passed or failed
earl:TestRequirement (subclass of earl:TestCriterion) Test Requirement a higher-level requirement that is tested by executing one or more sub-tests -
earl:TestCase (subclass of earl:TestCriterion) Test Case an atomic test, usually one that is a partial test for a requirement -
earl:TestResult Test Result the actual result of performing the test -
earl:TestMode Test Mode describes how a test was carried out -
earl:OutcomeValue Outcome Value a discrete value that describes a resulting condition from carrying out the test
earl:Pass (subclass of earl:OutcomeValue) Pass the class of outcomes to denote passing a test - earl:outcome
earl:Fail (subclass of earl:OutcomeValue) Fail the class of outcomes to denote failing a test - earl:outcome
earl:CannotTell (subclass of earl:OutcomeValue) Undetermined the class of outcomes to denote an undetermined outcome - earl:outcome
earl:NotApplicable (subclass of earl:OutcomeValue) Not applicable the class of outcomes to denote the test is not applicable - earl:outcome
earl:NotTested (subclass of earl:OutcomeValue) Not tested the class of outcomes to denote the test has not been carried out - earl:outcome
earl:Software Software any piece of software such as an authoring tool, browser, or evaluation tool -

Properties

EARL 1.0 Properties
Property Name Label Comment Domain Range
earl:assertedBy Asserted By assertor of an assertion earl:Assertion earl:Assertor
earl:subject Subject test subject of an assertion earl:Assertion earl:TestSubject
earl:test Test test criterion of an assertion earl:Assertion earl:TestCriterion
earl:result Result result of an assertion earl:Assertion earl:TestResult
earl:mode Mode mode in which the test was performed earl:Assertion earl:TestMode
earl:mainAssertor (subproperty of foaf:member external link) Main Assertor assertor that is primarily responsible for performing the test earl:Assertor earl:Assertor
earl:outcome Outcome outcome of performing the test earl:TestResult earl:OutcomeValue
earl:pointer Pointer location within a test subject that are most relevant to a test result earl:TestResult ptr:Pointer external link
earl:info Info additional warnings or error messages in a human-readable form earl:TestResult Literal

Instances

EARL 1.0 Instances
Instance Name Title Description
earl:automatic (instance of earl:TestMode) Automatic where the test was carried out automatically by the software tool and without any human intervention
earl:manual (instance of earl:TestMode) Manual where the test was carried out by human evaluators
earl:semiAuto (instance of earl:TestMode) Semi-Automatic where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test
earl:undisclosed (instance of earl:TestMode) Undisclosed where the exact testing process is undisclosed
earl:unknownMode (instance of earl:TestMode) Unknown where the testing process is unknown or undetermined
earl:passed (instance of earl:Pass) Passed the subject passed the test
earl:failed (instance of earl:Fail) Failed the subject failed the test
earl:cantTell (instance of earl:CannotTell) Cannot tell it is unclear if the subject passed or failed the test
earl:inapplicable (instance of earl:NotApplicable) Inapplicable the test is not applicable to the subject
earl:untested (instance of earl:NotTested) Untested the test has not been carried out

Appendix B: References

This section provides references to related documents and specifications.

[ATAG]
Authoring Tool Accessibility Guidelines
[Content]
Representing Content in RDF
[DC]
Dublin Core (DC) Metadata
[FOAF]
Friend of a Friend (FOAF) Vocabulary
[Guide]
Evaluation and Report Language (EARL) 1.0 Guide
[HTTP]
HTTP Vocabulary in RDF
[Pointers]
Pointer Methods in RDF
[RDF]
Resource Description Framework (RDF): Concepts and Abstract Syntax
[RDFS]
RDF Vocabulary Description Language 1.0: RDF Schema
[RDF-PRIMER]
RDF Primer
[RDF/XML]
RDF/XML Syntax Specification
[RDF-XML-DIFFS]
Why RDF model is different from the XML model
[RFC 2119]
Key words for use in RFCs to Indicate Requirement Levels
[UAAG]
User Agent Accessibility Guidelines
[WCAG]
Web Content Accessibility Guidelines
[XMLS]
XML Schema Part 0: Primer

Appendix C: Contributors (Non-Normative)

EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.

Contributors to this and/or previous Working Drafts

Shadi Abou-Zahra, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, and Rob Yonaitis.