Evaluation and Report Language (EARL) 1.0 Schema

2. Classes

[Editor's note: this section was moved here from the introduction (see "1.1. Structure of EARL Results" in the previous version.]

Every test result in EARL is expressed as an assertion. An EARL Assertion contains the following information:

Assertor: This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject: This can include Web content (such as Web pages, videos, applets, etc.), software (such as authoring tools, user agents, etc.), or other things being tested.
Test Criterion: What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result: What was the outcome of the test? A test result could also include contextual information such as error messages or relevant locations within the test subject.

Examples

Example 1: A person carries out a manual evaluation of a Web page against an accessibility requirement.

Assertor: Bob B. Bobbington
Test Subject: A Web page located at http://www.example.org/page.html
Test Criterion: Success Criteria 1.1.1 of the Web Content Accessibility Guidelines 2.0
Test Result: Passed

Example 2: A software carries out automated validation of a Web page against a technical specification.

Assertor: The W3C Markup Validator located at http://validator.w3.org/
Test Subject: The XHTML returned from a GET request to the URI http://www.example.org/page.html at 2004-04-14T14:00:04+1000
Test Criterion: The validity of the XHTML code
Test Result: Failed, the <li> element on line 53, char 7 was not closed.

2.1. Assertion Class

Assertion - a statement that embodies the results of a test.

Related Properties

Properties defined by this document:

Domain of:

earl:assertedBy
earl:subject
earl:test
earl:result
earl:mode

Range of:

none

Examples

Example 3: Instance of an assertion expressed as an RDF/XML fragment.

<earl:Assertion rdf:about="#assertion">
  <earl:assertedBy rdf:resource="#assertor"/>
  <earl:subject rdf:resource="#subject"/>
  <earl:test rdf:resource="#testcase"/>
  <earl:result rdf:resource="#result"/>
</earl:Assertion>

2.2. Assertor Class

Assertor - an entity such as a person, a software tool, an organization, or any other grouping that carries out a test collectively.

Related Properties

Properties defined by this document:

Domain of:

earl:mainAssertor

Range of:

earl:assertedBy
earl:mainAssertor

Properties not defined by this document:

dc:title: Human readable title for the assertor.
dc:description: Human readable description of the assertor.
foaf:name: Name of the assertor, which could be supplemented with further refinements such as foaf:firstName , foaf:surname , or foaf:nick .
foaf:mbox: E-mail address of the assertor, which is preferably provided in an encrypted format using the foaf:mbox_sha1sum property.
foaf:homepage: Homepage of the assertor, which could be supplemented with further refinements such as foaf:workplaceHomepage , foaf:workInfoHomepage , or foaf:schoolHomepage .

earl:Software: The assertor is a piece of software.
foaf:Agent: The assertor is an agent, as defined in [FOAF].
foaf:Person: The assertor is a person, as defined in [FOAF].
foaf:Organization: The assertor is an organization, as defined in [FOAF].
foaf:Group: The assertor is a group of agents, as defined in [FOAF]

Examples

Example 4: An Assertor that is a person called Bob B. Bobbington.

<foaf:Person rdf:about="#bob">
  <foaf:name>Bob B. Bobbington</foaf:name>
  <foaf:mbox rdf:resource="mailto:bob@example.org"/>
  <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>

Example 5: An Assertor that is a piece of software called Cool Tool.

<earl:Software rdf:about="#tool">
  <dc:title xml:lang="en">Cool Tool</dc:title>
  <dc:description xml:lang="en">My favorite tool!</dc:description>
  <foaf:homepage rdf:resource="http://example.org/tools/#cool"/>
  <dc:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>

Example 6: An Assertor this is the person from example 4 using the software tool from example 5.

<earl:compoundAssertor rdf:about="#assertor">
  <dc:title xml:lang="en">Bob using Cool Tool</dc:title>
  <dc:description xml:lang="en">Bob doing semi-automated testing</dc:description>
  <earl:mainAssertor rdf:resource="#bob"/>
</earl:compoundAssertor>

2.3. TestSubject Class

Test Subject - the class of things that have been tested against some test criterion. This class is intentionally generic to serve a wide variety of usages.

Related Properties

Properties defined by this document:

Domain of:

none

Range of:

earl:subject

Properties not defined by this document:

dc:title: Human readable title for the subject.
dc:description: Human readable description of the subject.
dc:date: Date on which the subject was tested (or when it was identified if more appropriate).
dc:hasPart: Relationship to other subjects that are part of this subject.
dc:isPartOf: Relationship to other subjects of which this subject is a part of.

Examples

Example 7: A group of pages that have been tested together as a single subject.

<earl:TestSubject rdf:about="http://www.example.org/">
  <dc:title xml:lang="en">example.org Web site</dc:title> 
  <dc:description xml:lang="en">Each page on the example.org Web site</dc:description> 
  <dct:hasPart rdf:resource="http://www.example.org/style.css"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page1.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page2.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page3.html"/> 
</earl:TestSubject>

Test Criterion - a testable statement, usually one that can be passed or failed. It is a super class for all types of tests including things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines [WCAG], or others.

Related Properties

Properties defined by this document:

Domain of:

none

Range of:

earl:test

Properties not defined by this document:

dc:title: Human readable title for the test criterion.
dc:description: Human readable description of the test criterion.
dc:hasPart: Relationship to other test criteria that are part of this criterion.
dc:isPartOf: Relationship to other test criteria of which this criterion is a part of.

earl:TestRequirement: A higher-level requirement that is tested by executing one or more sub-tests. For example WCAG 2.0 Success Criteria 1.1.1, which is evaluated using several Techniques for Success Criteria 1.1.1 and combining the results.
earl:TestCase: An atomic test, usually one that is a partial test for a requirement. For example, Technique H36: Using alt attributes on images used as submit buttons provides a partial test for WCAG 2.0 Success Criteria 1.1.1.

Examples

Example 8: Instance of a test case that is described with a title and its relationship to a test suite.

<earl:TestCase rdf:about="#testcase">
  <dc:title xml:lang="en">HTML Test 282</dc:title>
  <dc:description xml:lang="en">Test 282 of the HTML test suite</dc:description>
  <dct:isPartOf rdf:resource="http://example.org/tests/html/"/>
  <dct:hasPart rdf:resource="http://example.org/tests/html/#617"/>
</earl:TestCase>

2.5. TestResult Class

Test Result - the actual result of performing the test. It includes both machine-readable values as well as human-readable description of the results (typically error messages).

Related Properties

Properties defined by this document:

Domain of:

none

Range of:

earl:result

Properties not defined by this document:

dc:title: Human readable title for the subject.
dc:description: Human readable description of the subject.

[Editor's note: should we have a dc:date for the testing date (and make the date of the Test Subject only the fetch/locating/production date)?]

Examples

Example 9: A test result with a validity of fail and a description of the problem in English, and encoded in XHTML format.

<earl:TestResult rdf:about="#result">
  <earl:outcome rdf:resource="http://www.w3.org/2007/earl#fail"/>
  <dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>The <code>table</code> element is not allowed to appear
        inside a <code>p</code> element</p>
    </div>
  </dc:description>
  <earl:pointer rdf:resource="#xpointer"/>
  <earl:info rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>It seems the <code>p</code> element has not been closed</p>
    </div>
  </earl:info>
</earl:TestResult>

2.6. TestMode Class

The Test Mode described how a test was carried out. It reflects the information provided by the Assertor and is used to simplify some commonly used queries. The generic earl:TestMode class should not be used, but instead one of the following refinements:

earl:Automatic: Automatic - where the test was carried out automatically by the software tool and without any human intervention.
earl:Manual: Manual - where the test was carried out by human evaluators. This includes the case where the evaluators are aided by instructions or guidance provided by software tools, but where the evaluators carried out the actual test procedure.
earl:SemiAuto: Semi-Automatic - where the test was partially carried out by software tools, but where human input or judgment was still required to decide or help decide the outcome of the test.
earl:Undisclosed: Undisclosed - where the exact testing process is undisclosed or unknown.

[Editor's note: notice the capitalization since these are now class names -- needs confirmation by the group.]

Related Properties

Properties defined by this document:

Domain of:

none

Range of:

earl:mode

Examples

Example 10: The assertion from example 3 was carried out in semi-automatic mode.

<earl:Assertion rdf:about="#assertion">
  <earl:mode rdf:resource="http://www.w3.org/2007/earl#semiauto"/> 
</earl:Assertion>

2.7. OutcomeValue Class

[Editor's note: class name is OutcomeValue to differentiate from the outcome property (difference was only the letter case, which is hard to perceive in many situations -- needs group resolution.]

Outcome Value - a discrete value that describes a resulting condition from carrying out the test. The generic earl:OutcomeValue class should not be used, but instead one of the following refinements:

earl:Pass: Passed - the subject passed the test.
earl:Fail: Failed - the subject failed the test.
earl:CannotTell: Can not tell - it is unclear if the subject passed or failed the test. Usually this happens when an automated test requires human judgement to make a definite decision.
earl:NotApplicable: Not applicable - the test is not applicable to the subject.
earl:NotTested: Not tested - the test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring.

[Editor's note: notice the capitalization on the class refinements since these are now class names -- needs confirmation by the group.]

Related Properties

Properties defined by this document:

Domain of:

none

Range of:

earl:outcome

2.8. Software Class

[Editor's note: ERT WG is considering the use of DOAP terms to describe Software. Feedback on this consideration is welcome.]

A Software is any piece of software such as an authoring tool, browser, or evaluation tool. It can be used to describe an Assertor, such as a validation or other quality assurance tool, and it can be used to describe a Test Subject (for example to test compliance of an authoring tool to the Authoring Tool Accessibility Guidelines or of a browser to the User Agent Accessibility Guidelines).

Related Properties

Properties not defined by this document:

dc:title: Human readable title for the software.
dc:description: Human readable description of the software.
dc:hasPart: Relationship to other software components that are part of this software.
dc:isPartOf: Relationship to other software of which this software component is a part of.

[Editor's note: should we have a dc:date for the production date)?]

Examples

Example 11: The software which was an Assertor in example 5 could also be used as a Test Subject.

<earl:Assertion rdf:about="#assertion">
  <earl:subject>
    <earl:Software rdf:about="#tool">
      <--// Note: the dc:title, dc:description, foaf:homepage, and dct:hasVersion 
            properties are already defined by the RDF code in example 5 //-->
      <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date">2005-06-25</dc:date> 
      <dct:isPartOf rdf:resource="http://example.org/tools/"/>
      <dct:hasPart rdf:resource="http://example.org/tools/cool/#module-1"/>
    </earl:Software>
  </earl:subject>
</earl:Assertion>

3. Properties

3.1. assertedBy Property

Asserted By - assertor of an assertion.

Domain:: earl:Assertion
Range:: earl:Assertor

3.2. subject Property

Subject - test subject of an assertion.

Domain:: earl:Assertion
Range:: earl:TestSubject

3.3. test Property

Test - test criterion of an assertion.

Domain:: earl:Assertion
Range:: earl:TestCriterion

3.4. result Property

Result - result of an assertion.

Domain:: earl:Assertion
Range:: earl:TestResult

3.5. mode Property

Mode - mode of an assertion.

Domain:: earl:Assertion
Range:: earl:TestMode

3.6. mainAssertor Property

[Editor's note: this wording needs to be checked by he group.]

Main Assertor - assertor who is primarily responsible for performing the test.

Domain:: earl:Assertor
Range:: earl:Assertor

3.7. outcome Property

Outcome - outcome value of a test result.

Domain:: earl:TestResult
Range:: earl:OutcomeValue

3.8. pointer Property

Pointer - relevant locations within the test subject.

Domain:: earl:TestResult
Range:: none

3.9. info Property

Info - additional warnings or error messages.

Domain:: earl:TestResult
Range:: none

4. Conformance

[Editor's note: this section needs revision. Previous content is commented out within the source code. The following is a suggestion for further discussion:]

EARL 1.0 Reports

A valid EARL 1.0 report must validate against the formal RDF grammar, and must adhere to the conformance requirements defined by this document. To support XML based applications, EARL 1.0 reports should be represented in (a compact) RDF/XML format when being exported or published outside an application.

EARL 1.0 Processors

EARL 1.0 processors must process EARL 1.0 reports in RDF/XML format and should be able to parse EARL 1.0 reports in other RDF formats such as N3. EARL 1.0 processors are expected to process the following information in conforming EARL 1.0 reports:

Assertion

Equivalent Classes

none

Required Properties

earl:assertedBy
...

Optional Properties

none

Assertor

Equivalent Classes

foaf:Agent
...

Required Properties

earl:mainAssertor

Optional Properties

none

Appendix A: References (Non-Normative)

[Editor's note: this section needs revision.]

[DC]: The Dublin Core Metadata Element Set - DC Recommendation, 20 December 2004.
http://www.dublincore.org/documents/dces/
[DCT]: The Dublin Core Metadata Terms - DC Recommendation, 13 June 2005.
http://www.dublincore.org/documents/dcmi-terms/
[FOAF]: FOAF Vocabulary Specification - Working Draft, 3 June 2005.
http://xmlns.com/foaf/0.1/
[Guide]: Evaluation and Report Language (EARL) 1.0 Guide - Editors' Draft 14 December 2005. C. Velasco, J. Koch, S. Abou-Zahra eds.
http://www.w3.org/WAI/ER/EARL10/WD-EARL10-Guide-20051214
[HTTP]: HTTP Vocabulary in RDF - W3C Working Draft 20 December 2006. J. Koch, C. Velasco, S. Abou-Zahra eds.
http://www.w3.org/TR/HTTP-in-RDF/
[OWL]: OWL Web Ontology Language - W3C Recommendation, 10 February 2004.
http://www.w3.org/TR/owl-features/
[Pointers]: Pointer Methods in RDF - Editors' Draft 22 February 2007. J. Ley eds.
http://www.w3.org/WAI/ER/Pointers/WD-Pointers-20070222
[RDF]: RDF/XML Syntax Specification (Revised) - W3C Recommendation, 10 February 2004. D. Beckett ed.
http://www.w3.org/TR/2004/REC-rdf-syntax-grammar-20040210/
[RDF-PRIMER]: RDF Primer - W3C Recommendation, 10 February 2004. F. Manola, E.Miller eds.
http://www.w3.org/TR/rdf-primer/
[RDFS]: RDF Vocabulary Description Language 1.0: RDF Schema - W3C Recommendation, 10 February 2004. D. Brickley, R.V. Guha eds.
http://www.w3.org/TR/rdf-schema/
[RDF-XML-DIFFS]: Why RDF model is different from the XML model - Paper, September 1998. T. Berners-Lee.
http://www.w3.org/DesignIssues/RDF-XML
[RFC2119]: Key words for use in RFCs to Indicate Requirement Levels - IETF RFC, March 1997.
http://www.ietf.org/rfc/rfc2119.txt
[WCAG10]: Web Content Accessibility Guidelines 1.0 - W3C Recommendation, 5 May 1999. W. Chisholm, I. Jacobs, G. Vanderheiden eds.
http://www.w3.org/TR/WCAG10/
[XMLS]: XML Schema Part 0: Primer Second Edition - W3C Recommendation, 28 October 2004. D. Fallside, P. Walmsley erds.
http://www.w3.org/TR/xmlschema-0/

Appendix B: Contributors (Non-Normative)

EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, whose contributions have included editing the first versions of the EARL specifications, and Leonard Kasday who set the work in motion to develop EARL. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.

Contributors to this and/or previous Working Drafts

Shadi Abou-Zahra, Chrisoula Alexandraki, Shane Anderson, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Dan Brickley, Dan Connolly, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazaël-Massieux, Nadia Heninger, Sandor Herramhof, Ian Hickson, Björn Höhrmann, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Yehya Mohamed, Daniela Ortner, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Christophe Strobbe, Michael Squillace, Aaron Swartz, Olivier Thoreaux, Carlos Velasco, and Rob Yonaitis.

Evaluation and Report Language (EARL) 1.0 Schema

Appendices

Examples

Related Properties

Examples

Related Properties

Related Classes

Examples

Related Properties

Related Classes

Examples

Related Properties

Related Classes

Examples

Related Properties

Examples

Related Properties

Examples

Related Properties

Related Properties

Examples

EARL 1.0 Reports

EARL 1.0 Processors

Assertion

Assertor

Appendix A: References (Non-Normative)

Appendix B: Contributors (Non-Normative)

Contributors to this and/or previous Working Drafts