What is EARL?

The Evaluation and Report Language (EARL) is a standardized machine-readable language (RDF vocabulary) for expressing test results.
The primary motivation is to facilitate the exchange of test results between different checkers, such as Web accessibility evaluation tools.
Different types of EARL implementations already exist but mostly using older drafts. This brought some useful feedback but some issues as well.
EARL 1.0 is currently at Last Call Working Draft stage and we are looking for broad feedback, implementations, and advice on some RDF issues.

What's inside EARL?

The following documents are described in the EARL Overview page:

EARL 1.0 Requirements - Working Draft (WG Note)
EARL 1.0 Schema - Last Call Working Draft (Recommendation)
- HTTP Vocabulary in RDF - Working Draft (WG Note)
- Pointer Methods in RDF - Editors Draft (WG Note)
EARL 1.0 Guide - Editors Draft (WG Note or a Recommendation)

Structure of EARL

Assertor: This can include information about who or what ran the test. For example human evaluators, automated accessibility checkers, or combinations of these.
Test Subject: This can include Web content (such as Web pages, videos, applets, etc.), software (such as accessibility checkers, validators, user agents, etc.), or other things being tested.
Test Criterion: This is what we are evaluating the test subject against. This could be a specification, a set of guidelines, a test from a test suite, or some other testable statement.
Test Result: This is the outcome of the test including a human-readable description. It also describes which parts of the test subject were significant for the outcome of the test.

Example Statement #1

Assertor: Bob B. Bobbington
Test Subject: A Web page located at http://www.example.org/page.html
Test Criterion: Checkpoint 1.1 of the Web Content Accessibility Guidelines 1.0
Test Result: Passed

Example Statement #2

Assertor: The W3C Markup Validator located at http://validator.w3.org/
Test Subject: The XHTML returned from a GET request to the URI http://www.example.org/page.html at 2004-04-14T14:00:04+1000
Test Criterion: The validity of the XHTML code
Test Result: Failed, the <li> element on line 53, char 7 was not closed.

Assertor

<foaf:Person rdf:about="#bob">
  <foaf:name>Bob B. Bobbington</foaf:name>
  <foaf:mbox rdf:resource="mailto:bob@example.org"/>
  <foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>

<earl:Software rdf:about="#tool">
  <dc:title xml:lang="en">Cool Tool</dc:title>
  <dc:description xml:lang="en">My favorite tool!</dc:description>
  <foaf:homepage rdf:resource="http://example.org/tools/#cool"/>
  <dct:hasVersion>1.0.3</dct:hasVersion>
</earl:Software>

Test Subject

<earl:TestSubject rdf:about="http://www.example.org/">
  <dc:title xml:lang="en">example.org Web site</dc:title> 
  <dc:description xml:lang="en">Each page on the example.org Web site</dc:description> 
  <dct:hasPart rdf:resource="http://www.example.org/style.css"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page1.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page2.html"/> 
  <dct:hasPart rdf:resource="http://www.example.org/page3.html"/> 
</earl:TestSubject>

Test Criterion

<earl:TestCase rdf:about="http://www.w3.org/Style/CSS/Test/CSS1/20070302/sec21.htm">
  <dc:title xml:lang="en">CSS 1 Test Suite: 2.1 anchor</dc:title>
  <dc:description xml:lang="en">Test for CSS 1 Section 2.1 Anchor pseudo-classes</dc:description>
  <dct:isPartOf rdf:resource="http://www.w3.org/Style/CSS/Test/CSS1/20070302/"/>
</earl:TestCase>

<earl:TestRequirement rdf:about="http://www.w3.org/TR/REC-CSS1#anchor-pseudo-classes">
  <dc:title xml:lang="en">CSS 1 Section 2.1 Anchor pseudo-classes</dc:title>
  <dc:description xml:lang="en">Conformance to CSS 1 Section 2.1 Anchor pseudo-classes </dc:description>
  <dct:isPartOf rdf:resource="http://www.w3.org/TR/REC-CSS1"/>
</earl:TestCase>

Test Result

<earl:TestResult rdf:about="#result">
  <earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
  <dc:title xml:lang="en">Invalid Markup (code #353)</dc:title>
  <dc:description rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>The <code>table</code> element is not allowed to appear
        inside a <code>p</code> element</p>
    </div>
  </dc:description>
  <dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date">2006-08-13</dc:date>
  <earl:pointer rdf:resource="#xpointer"/>
  <earl:info rdf:parseType="Literal" xml:lang="en">
    <div xmlns="http://www.w3.org/1999/xhtml">
      <p>It seems the <code>p</code> element has not been closed</p>
    </div>
  </earl:info>
</earl:TestResult>

HTTP Vocabulary in RDF

Structure of HTTP-in-RDF

Connection: The TCP/IP connection which is not really part of HTTP.
Request: The request sent to the server, has 8 subclasses to reflect request types defined by RFC 2616.
Response: The response returned by the server, in reaction to a sent request.
Message Header: The headers of the request and response messages are broken out to simplify querries.
Body: The body of request or response messages, allows different formats

Connection

<http:Connection rdf:ID="conn">
  <http:connectionAuthority>www.example.org:80</http:connectionAuthority>
  <http:request rdf:parseType="Collection">
    <http:Request rdf:about="#req0"/>
    <http:Request rdf:about="#req1"/>
  </http:request>
</http:Connection>

Request

<http:GetRequest rdf:ID="req0">
  <http:abs_path>/foo/bar</http:abs_path>
  <http:version>1.1</http:version>
  <http:header rdf:parseType="Collection">
    <!--// Sequence of Message Headers //-->
  </http:header>
  <http:response rdf:resource="#resp0"/>
</http:GetRequest>

Response

<http:Response rdf:ID="resp0">
  <http:responseCode rdf:resource="http://www.w3.org/2006/http#200"/>
  <http:version>1.1</http:version>
  <http:header rdf:parseType="Collection">
    <!--// Sequence of Message Headers //-->
  </http:header>
</http:Response>

Message Header

<http:MessageHeader>
  <http:fieldName rdf:resource="http://www.w3.org/2006/http-header#content-type"/>
  <http:fieldValue>
    <http:HeaderElement>
      <http:elementName>application/xhtml+xml</http:elementName>
      <http:param>
        <http:Param>
          <http:paramName>charset</http:paramName>
          <http:paramValue>utf-8</http:paramValue>
        </http:Param>
      </http:param>
    </http:HeaderElement>
  </http:fieldValue>
</http:MessageHeader>

Body

<http:body rdf:parseType="Literal">
  <html xmlns="http://www.w3.org/1999/xhtml">
    <head>
      <title>Hello World!</title>
    </head>
    <body>
      <p>Hello World!
        <img src="/images/foo.png" alt="foo" width="5" height="7" /></p>
    </body>
  </html>
</http:body>

Evaluation And Report Language

What is EARL?

What's inside EARL?

EARL 1.0 Schema

Structure of EARL

Example Statement #1

Example Statement #2

Assertor

Test Subject

Test Criterion

Test Result

HTTP Vocabulary in RDF

Structure of HTTP-in-RDF

Connection

Request

Response

Message Header

Body

Relationship to MobileWeb

Thank You!