Evaluation and Report Language (EARL) 1.0 Guide

Abstract

This document provides an introductory guide to the Evaluation and Report Language (EARL) 1.0, and shall be used as an accompanying document to the normative document Evaluation and Report Language (EARL) 1.0 Schema. The Evaluation and Report Language is a framework targeted to express test results. Although the term test can be taken in its widest acception, EARL is targeted to report and exchange results of tests of Web applications and resources. EARL intends to provide a vendor neutral and platform independent format. [Editor's note: synchronize with EARL 1.0 Schema.]

Status of this document

This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.

[Editor's note: describe intent of this working draft and propose feedback questions. Synchronize with EARL 1.0 Schema.]

Please send comments to the mailing list of the ERT WG. The archives for this list are publicly available.

This is a W3C Working Draft of the Evaluation and Report Language (EARL) 1.0 Guide. This document will be published and maintained as a W3C Recommendation after review and refinement. Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.

This document was produced under the 5 February 2004 W3C Patent Policy. The Working Group maintains a public list of patent disclosures relevant to this document; that page also includes instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) with respect to this specification should disclose the information in accordance with section 6 of the W3C Patent Policy.

This document has been produced as part of the W3C Web Accessibility Initiative (WAI). The goals of the Evaluation and Repair Tools Working Group (ERT WG) are discussed in the Working Group charter. The ERT WG is part of the WAI Technical Activity.

Table of Contents

1. Introduction
   • 1.1 Pre-requisites
2. What is EARL?
   • 2.1 EARL use cases
   • 2.2 EARL audience
   • 2.3 Fitting EARL to the test process
3. Structure of an EARL report: concepts and core classes
   • 3.1 Namespace
   • 3.2 The EARL root element
   • 3.3 Our first EARL report
   • 3.4 Core EARL Classes
4. Complex examples
5. Processing and aggregating reports
6. Extending EARL
7. Conclusions and open issues

Appendices

1. References
2. Contributors

1. Introduction

This document provides an introductory guide to the Evaluation and Report Language (EARL) 1.0, and shall be used as an accompanying document to the normative document Evaluation and Report Language (EARL) 1.0 Schema.

[Editor's note: ...]

The objectives of this document are:

• To provide an introduction to the use of EARL in different scenarios.
• To clarify the key concepts of EARL and their translation into the different classes.
• To explain the restrictions of the different EARL components.
• To show how to aggregate and process EARL reports.
• To demonstrate how to extend and customise EARL.

1.1 Pre-requisites

Although this document does not assume any previous knowledge of EARL, the following knowledge is assumed:

• Basic knowledge of XML [XML] and associated technologies.
• Basic knowledge about the Semantic Web and RDF. For references, consult [RDF], [RDF-PRIMER] and [RDFS].

2. What is EARL?

The Evaluation and Report Language (EARL) is a framework targeted to express and compare test results. EARL builds on top of the Resource Description Framework [RDF], which is the basis for the Semantic Web. It is not the object of this document to introduce the reader to the intricacies of RDF, and some basic knowledge must be assumed as a pre-requisite (see, e.g., [RDF-PRIMER] for more information). As any RDF vocabulary, EARL is not more than a collection of statements about resources, each with a subject, a predicate (or a verb) and an object. These statements can be serialized in many ways (RDF/XML or Notation 3, N3). A typical EARL report could contain the following statements (oversimplifying the notation and not including namespaces):

<#someone> <#checks> <#resource> .
<#resource> <#fails> <#test> .

From these simple two statements, it can be inferred already the main components of an EARL Report (wrapped up in an assertion):

• Who (or which tool) runs a test.
• The resource tested.
• The tested criterion(-a).
• The result(s) of the test.

This structure shows the universal applicability of EARL and its ability to refer to any type of test: bug reports, software unit tests, test suite evaluations, conformance claims or even tests outside the world of software and the World Wide Web (although for such cases, there might be too open issues for its full aplicability). It must be stressed again the semantic nature of EARL: its purpose is to facilitate the extraction and comparison of test results by humans and especially by tools (the semantic Web paradigm); it is not simply an storage of information, for which some other XML application might be more suitable.

Summarising, the objectives of EARL are to:

• Create a standardised way to produce test reports;
• Support the exchange of reports between testers (humans or testing tools);
• Facilitate the comparison of test results; and
• Ease the aggregation of test results (e.g., like a different set of tests on the same subject).

It is also remarkable that the extensibility of RDF (or EARL) allows to tool vendors or developers the addition of new functionalities to the vocabulary, without losing any of the aforementioned characteristics, as other testers might ignore those extensions that they do not understand when processing third party results.

2.1. EARL use cases

The applicability of EARL to different scenarios can be seen in the following use cases:

Evaluating a Web site using tools in different languages

A group of people speaking different languages are evaluating a Web site for conformance to different legal environments, such as, e.g., Section 508 in the USA and BITV in Germany. The use of EARL:

• allows localized messages explaining where problems are met. The report can contain messages in the languages spoken by the evaluators so that each of them understands the messages.
• allows "keywords" to express the conformance level reached by the Web site that are language-independent. Thus a software tool can translate the validity levels in different languages.

Combining results from different evaluation tools

A Web site evaluator uses different tools for the task. Each tool can perform specific tests that the other tools cannot do. The evaluator's client wants a complete evaluation report. All the evaluation tools used produce a report in EARL format. Therefore, the evaluator can combine the separate reports into one bigger report, query the results, and offer to her customer statistical reports and a detailed conformance claim that specifies where the Web site does not meet the required level.

Comparing results from different tools

A Web site evaluator uses different tools for evaluation. The tools perform the same tests. All the evaluation tools used produce a report in EARL format. Therefore, the evaluator can compare the results from different tools to increase the confidence level of the test results. It will also help to make assertions about a given resource, when one of the tools is only able to give a warning on a problem, but the other performs a thorough test that removes the aforementioned uncertainty.

Benchmarking an evaluation tool against a test suite

For a benchmarking test of a given provider, different tools perform their tests on sample documents from a test suite. Some evaluation tools may produce false positives or false negatives. All of them create an EARL report with the result. Comparing the results of the tools with a theoretical output file from the test suite, evaluation tools could be rated according to accuracy against the test suite.

Monitoring a Web site over time (quality assurance)

A Web project manager wants to track the accessibility of a Web site over time by comparing current test results with previous ones. The reports contain the date/time of the tests and a way to locate the parts of the document the messages refer to. By comparing messages referring to the same locations the project manager can monitor possible improvements, and allocate resources to solve problems in the critical areas of the Web site.

Exchanging data with repair tools

A repair tool (or a similar module in an authoring tool) uses the results of an evaluation tool to identify the parts of the document that need to be fixed. For each instance of an error it provides a way for the user to notice the error and fix the document. The same scenario can be used for Content Management Systems that wish to integrate an evaluation tool into their workflow, helping to locate accessibility and validation problems to Web editors.

Exchanging data with search engines

A search engine uses a third-party service which publishes EARL reports of Web sites. The user interface lets the user choose between different levels of accessibility. The list of search results contains only documents with a chosen accessibility level. The search engine uses the test results in the calculation of the ranking/relevance, so that it affects the search results order.

[Editor's note: Maybe add some more exotic scenario outside the Web and software development.]

2.2. EARL audience

EARL is flexible enough to respond to the needs of a variety of audiences involved in a testing or quality assurance process. Typical profiles are:

• Product manager: responsible for delivering a given product;
• Product designer: designs the product and documents this in a design specification;
• Quality engineer or tester: takes the product through a series of tests to find bugs; and
• Developer: creates a product to satisfy the design specification; fixes bugs found by the quality engineer or tester.

2.3. Fitting EARL to the test process

A generic testing process has several steps. Typically, a test process consists of the following phases (see Figure 1):

1. Requirements capture
2. Test specification and design
3. Test
4. Reporting of test results
5. Collation of results
6. Query of results
7. Presentation of results

EARL is targeted to the phase #4, and supports the rest of the following ones, by providing the necessary information for the semantic interpretation of the test results.

[Editor's note: Add figure.]

3. Structure of an EARL report: concepts and core classes

EARL is not an standalone technology, and builds on top of many existing vocabularies that cover some of its needs for metadata definition. This approach avoids the re-creation of applications already established and tested like the Dublin Core elements. The referenced specifications are:

• Dublin Core Metadata Initiative: The Dublin Core is a metadata standard for describing digital resources, often expressed in XML. The first standard published is the Dublin Core Metadata Element Set. It consists of 16 optional metadata elements, any of which may be repeated or omitted. Typical elements are: Title, Creator, Subject, Description, Publisher, Contributor, Date, Type, Format, Identifier, Source, Language, etc. The Dublin Core Metadata Element Set was accepted as a NISO standard in 2001 (ANSI/NISO Z39.85-2001) and as an ISO standard in 2003 (ISO 15836:2003(E)).
• Friend of a Friend (FOAF) project: The FOAF project is about creating a Web of machine-readable resources describing people, the links between them and the things they create and do. Of particular interest for EARL are the Classes foaf:Person and foaf:Project [FOAF].

RDF can be serialized in different ways, but the XML representation [RDF/XML] is the preferred method and will be used throughout this document. However, even when selecting this approach, there are many equivalent ways to express an RDF model.

3.1. Namespaces

Table 1 presents the core namespaces used by EARL. The prefix refers to the convention used in this document to denote a given namespace, and can be freely modified.

[Editor's note: Versioning terms during the process of developing the vocabulary is an issue the group is working on. It is possible that a new namespace will be used for a final version of the vocabulary.]

3.2. The EARL root element

[Editor's note: Short introduction on the RDF/XML serialization.]

Let us start building our first EARL report with its root element. The root element of any EARL report is an RDF node, as with any RDF vocabulary. There, we declare the corresponding namespaces, as described in Table 1, plus any custom namespace used to define additional classes and or properties.

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
         xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
         xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#">

    <!-- ... -->

</rd:RDF>

3.3. Our first EARL report

Once we have defined the root element, let us build in a step-by-step the different components of an EARL report. Let us assume we want to express the results of an XHTML validation in a given document with the W3C HTML Validator in EARL. The tested document has the following HTML code:

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html lang="en" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
  <head>
  <title>Example of project pages</title>
  </head>
  <body>
  <h1>Project description</h1>
  <h2>My project name</h2>
  <p>The strategic goal of this project is to make you understand EARL.</p>

  <ul>
    <li>Here comes objective 1.
    <li>Here comes objective 2.</li>
  </ul>
  <p alt="what?">And goodbye ...</p>
  </body>
</html>

This document has three errors that will constitute the basis of our EARL report:

1. Error - Line 14 column 7: document type does not allow element "li" here; missing one of "ul", "ol" start-tag.
2. Error - Line 15 column 6: end tag for "li" omitted, but OMITTAG NO was specified.
3. Error - Line 16 column 9: there is no attribute "alt".

Let us start by defining who (or what) run the test. In the EARL jargon, that is an Assertor. There are two types of Assertors, SingleAssertor and CompoundAssertor. Assuming the test was run by a person with the W3C HTML Validator, we can use a CompoundAssertor with the following structure.

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:CompoundAssertor rdf:nodeID="assertor_01">
        <dc:title>John Doe and the W3C HTML Validator</dc:title>
        <earl:mainAssertor rdf:nodeID="johnDoe" />
        <earl:helpAssertor rdf:resource="http://validator.w3.org/about.html#" />
    </earl:CompoundAssertor>

    <earl:mainAssertor rdf:about="http://validator.w3.org/about.html#">
        <dc:title xml:lang="en">W3C HTML Validator</dc:title>
        <dc:description xml:lang="en">
            W3C Markup Validation Service, a free service that checks
            Web documents in formats like HTML and XHTML for conformance
            to W3C Recommendations and other standards.
        </dc:description>
        <!-- TODO: earl:uri ?? -->
        <dc:location rdf:resource="http://validator.w3.org/" />
        <dct:hasVersion>0.7.1</dct:hasVersion>
    </earl:mainAssertor>

    <foaf:Person rdf:nodeID="johnDoe">
        <foaf:name>John Doe</foaf:name>
        <foaf:mbox rdf:resource="mailto:john@example.org" />
    </foaf:Person>

    <!-- ... -->

</rdf:RDF>

Notice that this is only one of multiple serializations via an anonymous node. This representation is equivalent, for instance, to the following:

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:CompoundAssertor rdf:nodeID="assertor_01">
        <dc:title>John Doe and the W3C HTML Validator</dc:title>
        <earl:mainAssertor>
            <foaf:Person>
                <foaf:name>John Doe</foaf:name>
                <foaf:mbox rdf:resource="mailto:john@example.org" />
            </foaf:Person>
        </earl:mainAssertor>
        <earl:helpAssertor
            rdf:resource="http://validator.w3.org/about.html#" />
    </earl:CompoundAssertor>

    <earl:mainAssertor
        rdf:about="http://validator.w3.org/about.html#">
        <dc:title xml:lang="en">W3C HTML Validator</dc:title>
        <dc:description xml:lang="en">
            W3C Markup Validation Service, a free service that checks
            Web documents in formats like HTML and XHTML for conformance
            to W3C Recommendations and other standards.
        </dc:description>
        <!-- TODO: earl:uri ?? -->
        <dc:location rdf:resource="http://validator.w3.org/" />
        <dct:hasVersion>0.7.1</dct:hasVersion>
    </earl:mainAssertor>

    <!-- ... -->

</rdf:RDF>

The next step is to define the tested resource. For that, EARL offers the TestSubject class. This class is a generic wrapper for things to be tested like Web resources (WebContent) or software (Software). For our first example, let us use the generic TestSubject class, which must have a Dublin Core date property:

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:TestSubject rdf:about="http://example.org/resource/index.html">
        <dc:date>2006-02-14</dc:date>
        <dc:title xml:lang="en">Project Description</dc:title>
    </earl:TestSubject>

    <!-- ... -->

</rdf:RDF>

The third step is to define the used criterion for testing the given resource. In our example, we are testing validity with XHTML 1.0 Strict, which could be expressed in the following way via the TestRequirement class:

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:TestRequirement
        rdf:about="http://www.w3.org/TR/xhtml1/dtds.html#a_dtd_XHTML-1.0-Strict">
        <dc:title xml:lang="en">XHTML 1.0 Strict Document Type Definition</dc:title>
        <dc:description xml:lang="en">
            DTD for XHTML 1.0 Strict.
        </dc:description>
        <dct:isPartOf rdf:resource="http://www.w3.org/TR/xhtml1/" />
    </earl:TestRequirement>

    <!-- ... -->

</rdf:RDF>

[Editor's note: Pending consensus on definition of test case and test requirements.]

Before we build up our assertion about our hypothetical evaluation, let us construct the result of the test with the three errors highlighted by the validator. This is done via the TestResult class.

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:TestResult rdf:ID="error_1">
        <earl:validity
            rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
        <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error - Line 14 column 7: document type does not allow element
                <code>li</code> here; missing one of <code>ul</code>,
                <code>ol</code> start-tag.
                </p>
            </div>
        </dc:description>
    </earl:TestResult>
    <earl:TestResult rdf:ID="error_2">
        <earl:validity
            rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
        <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error - Line 15 column 6: end tag for <code>li</code>
                omitted, but OMITTAG NO was specified.
                </p>
            </div>
        </dc:description>
    </earl:TestResult>
    <earl:TestResult rdf:ID="error_3">
        <earl:validity
            rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
        <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error - Line 16 column 9: there is no attribute
                <code>alt</code>.
                </p>
            </div>
        </dc:description>
    </earl:TestResult>

    <!-- ... -->

</rdf:RDF>

[Editor's note: Comment on location of errors in test subject pending.]

For the sake of completeness, we have included the three validation errors. We might need only one of them to build up our assertion. The assertion is the core of EARL, and allows to put together all pieces of the puzzle. An Assertion must have the following properties:

• earl:assertedBy - The Assertion must be asserted by an assertor. The assertor is a human or software, or groups of these, that determine the result.
• earl:subject - The thing that is being tested against some given requirements or test cases.
• earl:requirement - The requirement that is used to test a subject.
• earl:result - The result of the test, i.e., whether the subject passes or fails the test case (or there is some other result).

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:Assertion rdf:ID="assertion_1">
        <earl:assertedBy rdf:nodeID="assertor_01" />
        <earl:subject rdf:resource="http://example.org/resource/index.html" />
        <earl:testcase rdf:resource="http://www.w3.org/TR/xhtml1/dtds.html#a_dtd_XHTML-1.0-Strict" />
        <earl:result rdf:resource="#error_1" />
    </earl:Assertion>

    <!-- ... -->

</rdf:RDF>

[Editor's note: Adding a list with the three assertions ??]

We can now put together all the elements of our first EARL report:

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
    xmlns:dc="http://purl.org/dc/elements/1.1/"
    xmlns:dct="http://purl.org/dc/terms/"
    xmlns:foaf="http://xmlns.com/foaf/0.1/">

    <earl:Assertion rdf:ID="assertion_1">
        <earl:assertedBy rdf:nodeID="assertor_01" />
        <earl:subject rdf:resource="http://example.org/resource/index.html" />
        <earl:testcase rdf:resource="http://www.w3.org/TR/xhtml1/dtds.html#a_dtd_XHTML-1.0-Strict" />
        <earl:result rdf:resource="#error_1" />
    </earl:Assertion>

    <earl:Assertion rdf:ID="assertion_2">
        <earl:assertedBy rdf:nodeID="assertor_01" />
        <earl:subject rdf:resource="http://example.org/resource/index.html" />
        <earl:testcase rdf:resource="http://www.w3.org/TR/xhtml1/dtds.html#a_dtd_XHTML-1.0-Strict" />
        <earl:result rdf:resource="#error_2" />
    </earl:Assertion>

    <earl:Assertion rdf:ID="assertion_3">
        <earl:assertedBy rdf:nodeID="assertor_01" />
        <earl:subject rdf:resource="http://example.org/resource/index.html" />
        <earl:testcase rdf:resource="http://www.w3.org/TR/xhtml1/dtds.html#a_dtd_XHTML-1.0-Strict" />
        <earl:result rdf:resource="#error_3" />
    </earl:Assertion>

    <earl:CompoundAssertor rdf:nodeID="assertor_01">
        <dc:title>John Doe and the W3C HTML Validator</dc:title>
        <earl:mainAssertor rdf:nodeID="johnDoe" />
        <earl:helpAssertor rdf:resource="http://validator.w3.org/about.html#" />
    </earl:CompoundAssertor>
    <earl:mainAssertor rdf:about="http://validator.w3.org/about.html#">
        <dc:title xml:lang="en">W3C HTML Validator</dc:title>
        <dc:description xml:lang="en">
            W3C Markup Validation Service, a free service that checks
            Web documents in formats like HTML and XHTML for conformance
            to W3C Recommendations and other standards.
        </dc:description>
        <!-- TODO: earl:uri ?? -->
        <dc:location rdf:resource="http://validator.w3.org/" />
        <dct:hasVersion>0.7.1</dct:hasVersion>
    </earl:mainAssertor>
    <foaf:Person rdf:nodeID="johnDoe">
        <foaf:name>John Doe</foaf:name>
        <foaf:mbox rdf:resource="mailto:john@example.org" />
    </foaf:Person>

    <earl:TestSubject rdf:about="http://example.org/resource/index.html">
        <dc:date>2006-02-14</dc:date>
        <dc:title xml:lang="en">Project Description</dc:title>
    </earl:TestSubject>

    <earl:TestRequirement
        rdf:about="http://www.w3.org/TR/xhtml1/dtds.html#a_dtd_XHTML-1.0-Strict">
        <dc:title xml:lang="en">XHTML 1.0 Strict Document Type Definition</dc:title>
        <dc:description xml:lang="en">
            DTD for XHTML 1.0 Strict.
        </dc:description>
        <dct:isPartOf rdf:resource="http://www.w3.org/TR/xhtml1/" />
    </earl:TestRequirement>

    <earl:TestResult rdf:ID="error_1">
        <earl:validity
            rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
        <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error - Line 14 column 7: document type does not allow element
                <code>li</code> here; missing one of <code>ul</code>,
                <code>ol</code> start-tag.
                </p>
            </div>
        </dc:description>
    </earl:TestResult>
    <earl:TestResult rdf:ID="error_2">
        <earl:validity
            rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
        <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error - Line 15 column 6: end tag for <code>li</code>
                omitted, but OMITTAG NO was specified.
                </p>
            </div>
        </dc:description>
    </earl:TestResult>
    <earl:TestResult rdf:ID="error_3">
        <earl:validity
            rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
        <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error - Line 16 column 9: there is no attribute
                <code>alt</code>.
                </p>
            </div>
        </dc:description>
    </earl:TestResult>

</rdf:RDF>

In this section we have reviewed all the necessary steps to build an EARL report. As mentioned already several times, Example 3.9 is only one of the multiple serializations that allow us to express the corresponding RDF graph.

[Editor's note: Insert an image with the corresponding RDF graph.]

3.4. Core EARL Classes

The previous section introduced us to the core components of an EARL report. In this section we will review the core classes and properties of EARL, together with some clarifications on its usage. The decomposition of every class is followed by the set of RDF properties the class must or may have.

3.4.1. Assertion

The earl:Assertion class is used to declare an statement about the results of a test. As shown in Section 2, an Assertion binds together the four components of EARL: the assertor, the subject, the test requirement and its result (see Example 3.8). It is therefore the fundamental unit of an EARL statement or set of statements.

An Assertion must have at least the following properties:

earl:assertedBy

Person, software, or combinations thereof that determined the result of the test. There are several ways to express these, as it will be seen later on. In the EARL jargon, these are references to Assertors.

earl:subject

The thing that is being tested against some given requirements or test cases. In the EARL jargon, these are references to TestSubjects (also Software and WebContent can be referenced here).

earl:requirement

The requirement that is used to test a subject. In the EARL jargon, these are references to TestRequirements.

earl:result

Reference to the result of the test, i.e., whether the subject passes or fails the test case (or there is some other result). In the EARL jargon, these are references to TestResults.

An Assertion may also include the following optional properties:

earl:evidence

This property may be used to describe the other results which were used to derive a result by inference.

[Editors' note: the group is waiting for a proposal on whether this is best modelled as an RDF Collection, or through some other method to describe a list of Assertions. Example pending]

earl:methodology

There are sometimes various methods that can be used to test for the same requirement. This property allows the tester to declare which methodology was used to determine the result given, or in the case of a derived result the ruleset that was used to combine other results. There is currently no particular vocabulary specified for this. [Editors' note: This is very unclear: WG discussion needed. Example pending]

earl:mode

The mode in which the test was performed: as an automated computer process, by a human making a subjective judgement, or otherwise. This can be a reference to a TestMode instance.

XXX

3.4.2. Assertor (SingleAssertor or CompoundAssertor)

The earl:Assertor class is the person, tool or combinations thereof, that performs the test and expresses the results via one or several Assertions. We have the following types:

1. earl:SingleAssertor - The Assertor is unique.

   It can be of the following types:

   [Editor's note: the use of FOAF is subject to review of the stability of these terms.]

   [Editor's note: the current requirements on properties to identify the tester are subject to revision.]

   foaf:Person

   The Assertor is a human being. This uses the FOAF vocabulary term foaf:Person to describe a person. There should be identifying information including a name, and a uniquely identifying property such as email address or an encrypted email address. The properties foaf:name, foaf:mbox and foaf:mbox_sha1sum are defined by FOAF [FOAF].

   foaf:Agent

   The Assertor is an Agent, as defined in [FOAF]. foaf:Person is a subclass of this class, which also has subclasses of foaf:Organisation and foaf:Group.

   earl:Software

   The Assertor is a piece of Software, such as a black box testing tool or an accessibility evaluation tool. See Section 3.4.3.1 for further details.

   Example 3.11. Example of a FOAF Person description.

   <rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
       xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
       xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
       xmlns:foaf="http://xmlns.com/foaf/0.1/">
   
       <foaf:Person rdf:ID="jd">
           <foaf:name>John Doe</foaf:name>
           <foaf:givenname>John</foaf:givenname>
           <foaf:family_name>Doe</foaf:family_name>
           <foaf:mbox rdf:resource="mailto:johndoe@example.org" />
           <foaf:mbox_sha1sum>d44df295919cbbe85586fc33742abbe6a9737a2b</foaf:mbox_sha1sum>
       </foaf:Person>
   
   </rdf:RDF>

2. earl:CompoundAssertor - The Assertor is a compound group of persons and/or software tools. Each group must have at least one primary Assertor and may have a secondary Assertor, identified, respectively, by the properties:

   earl:mainAssertor

   The main assertor, which shall be in turn a person or a tool (see SingleAssertor).

   earl:helpAssertor

   The assistant actor(s) to the main assertor, which also shall be in turn a person or a tool (see SingleAssertor).

   Additionally, there is an optional description identified by the Dublin Core dc:description property.

Since the range of both of these properties is the earl:Assertor class, the instances can be a Person, Agent, Software, or recursively another CompoundAssertor.

3.4.3. TestSubject

3.4.3.1. Software

3.4.3.2. WebContent

3.4.4. TestRequirement

3.4.5. TestMode

3.4.6. TestResult

3.4.7. ValidityLevel

3.4.8. ConfidenceLevel

4. Complex examples

[Editor's note:

• Defining the subject of the test
• Defining what is being tested
• Indicating cross-references
• Expressing the results

.]

5. Processing and aggregating reports

[Editor's note: An aggregation example. E.g., validation and accessibility report.]

XXX

6. Extending EARL

Although EARL is designed to cover a wide range of testing scenarios, it cannot fulfill the requirements of all possible applications. However, as with any Semantic Web application, it can be easily extended. In this section, we will present two examples showing the extensibility potential of EARL in two scenarios:

1. Extending the WebContent class to provide a full description of the tested Web resource when HTTP content negotiation occurs. For that, we will make use of [HTTPRDF]
2. Extending the TestResult to uniquely identify the error location via an XPath [XPATH] expression.

6.1 Use HTTP in RDF in earl:WebContent

As mentioned earlier, there are occasions where a URI is not sufficient to uniquely identify a Web Resource. For instance, the site may offer multi-lingual versions of its content according to the user preferences set up in her browse. To that end, we will use the W3C Note [HTTPRDF], where all HTTP headers can be expressed via RDF via modeling HTTP request/response pairs. Let us assume we are accessing the URI http://www.example.org:80/. The Vary header in the response indicates content negotiation. Thus, it is essential to record the two request headers (Accept and Accept-Language) that were used in the negotiation process.

<rdf:RDF
    xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
    xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
    xmlns:http="http://www.w3.org/1999/xx/http#">

  <earl:WebContent rdf:about="http://www.example.org/">
    <earl:httpRequest>
      <http:GetRequest>
        <http:host>www.example.org</http:host>
        <http:port>80</http:port>
        <http:absPath>/</http:absPath>
        <http:version>1.1</http:version>
        <http:accept>text/html;q=1.0, */*;q=0.01</http:accept>
        <http:acceptLanguage>de-DE;q=1.0, de;q=0.75, en-GB;q=0.5, en;q=0.25,
          *;q=0.01</http:acceptLanguage>
      </http:GetRequest>
    </earl:httpRequest>
    <earl:httpResponse>
      <http:Response>
        <http:contentType>text/html;charset=utf-8</http:contentType>
        <http:vary>accept, accept-language</http:vary>
        <http:body rdf:parseType="Resource">
          <http:bodyContent><![CDATA[aksgbq3833o3gbo4zgblakc8t9ut2]]></http:bodyContent>
          <http:bodyEncoding>Base64</http:bodyEncoding>
        </http:body>
      </http:Response>
    </earl:httpResponse>
  </earl:WebContent>

</rdf:RDF>

This simple example shows how to use [HTTPRDF] to distinguish different resources served by the same URI.

6.2 Use XPath in earl:TestResult

Location of errors within documents is a critical issue when following quality assurance methods to monitor Web sites or when benchmarking tools against a test suite to compare capabilities of tools. This example shows an approach to extend the TestResult class to support [XPath] expressions for any markup document. To do that, we first specify two RDF Schemas to express XPath and the corresponding namespace mappings.

6.2.1 RDF Schemas for the use of XPath: XPath and Namespace

First, we define an XPath class (in the exemplary namespace http://www.example.org/ns/xpath#) with two properties:

• xpath:expression for the XPath expression, and
• ns:namespace for referencing a prefix-namespace mapping (optional).
• xpath:xpath is an optional property added to TestResult to reference XPath expressions.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:owl="http://www.w3.org/2002/07/owl#"
  xmlns:xpath="http://www.example.org/ns/xpath#"
  xmlns:ns="http://www.example.org/ns/namespace#">

  <rdfs:Class rdf:about="http://www.example.org/ns/xpath#XPath">
    <rdfs:label xml:lang="en">The XPath Class</rdfs:label>
    <rdfs:subClassOf rdf:parseType="Collection">
      <owl:Restriction>
        <owl:onProperty rdf:resource="http://www.example.org/ns/xpath#expression"/>
        <owl:minCardinality
          rdf:datatype="http://www.w3.org/2001/XMLSchema#nonNegativeInteger">1</owl:minCardinality>
        <owl:maxCardinality
          rdf:datatype="http://www.w3.org/2001/XMLSchema#nonNegativeInteger">1</owl:maxCardinality>
      </owl:Restriction>
      <owl:Restriction>
        <owl:onProperty rdf:resource="http://www.example.org/ns/namespace#namespace"/>
        <owl:minCardinality
          rdf:datatype="http://www.w3.org/2001/XMLSchema#nonNegativeInteger">0</owl:minCardinality>
      </owl:Restriction>
    </rdfs:subClassOf>
  </rdfs:Class>
  <rdf:Property rdf:about="http://www.example.org/ns/xpath#expression">
    <rdfs:label xml:lang="en">XPath expression</rdfs:label>
    <rdfs:domain rdf:resource="http://www.example.org/ns/xpath#XPath"/>
  </rdf:Property>
  <rdf:Property
    rdf:about="http://www.example.org/ns/xpath#xpath">
    <rdfs:label xml:lang="en">XPath</rdfs:label>
    <rdfs:range
      rdf:resource="http://www.example.org/ns/xpath#XPath" />
  </rdf:Property>

</rdf:RDF>

The prefix-namespace mapping is done with a Namespace class (in the exemplary namespace http://www.example.org/ns/namespace#) containing two properties:

• ns:prefix for the namespace prefix, and
• ns:uri for the namespace URI.

<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:owl="http://www.w3.org/2002/07/owl#"
  xmlns:ns="http://www.example.org/ns/namespace#">

  <rdfs:Class rdf:about="http://www.example.org/ns/namespace#Namespace">
    <rdfs:label xml:lang="en">A namespace</rdfs:label>
    <rdfs:subClassOf rdf:parseType="Collection">
      <owl:Restriction>
        <owl:onProperty rdf:resource="http://www.example.org/ns/namespace#prefix"/>
        <owl:minCardinality
          rdf:datatype="http://www.w3.org/2001/XMLSchema#nonNegativeInteger">1</owl:minCardinality>
      </owl:Restriction>
      <owl:Restriction>
        <owl:onProperty rdf:resource="http://www.example.org/ns/namespace#uri"/>
        <owl:minCardinality
          rdf:datatype="http://www.w3.org/2001/XMLSchema#nonNegativeInteger">1</owl:minCardinality>
        <owl:maxCardinality
          rdf:datatype="http://www.w3.org/2001/XMLSchema#nonNegativeInteger">1</owl:maxCardinality>
      </owl:Restriction>
    </rdfs:subClassOf>
  </rdfs:Class>
  <rdf:Property rdf:about="http://www.example.org/ns/namespace#prefix">
    <rdfs:label xml:lang="en">Namespace prefix</rdfs:label>
    <rdfs:domain rdf:resource="http://www.example.org/ns/namespace#Namespace"/>
  </rdf:Property>
  <rdf:Property rdf:about="http://www.example.org/ns/namespace#uri">
    <rdfs:label xml:lang="en">Namespace URI</rdfs:label>
    <rdfs:domain rdf:resource="http://www.example.org/ns/namespace#Namespace"/>
  </rdf:Property>
  <rdf:Property rdf:about="http://www.example.org/ns/namespace#namespace">
    <rdfs:label xml:lang="en">Namespace</rdfs:label>
    <rdfs:range rdf:resource="http://www.example.org/ns/namespace#Namespace"/>
  </rdf:Property>

</rdf:RDF>

An XHTML example

Let us take our Example 3.2 and try to express the results with our new classes. Let us define:

• ns:prefix as x, and
• ns:uri as http://www.w3.org/1999/xhtml.

Then we can re-state our TestResults as follows:

<rdf:RDF xmlns:earl="http://www.w3.org/WAI/ER/EARL/nmg-strawman#"
  xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
  xmlns:rdfs="http://www.w3.org/2000/01/rdf-schema#"
  xmlns:dc="http://purl.org/dc/elements/1.1/"
  xmlns:dct="http://purl.org/dc/terms/"
  xmlns:foaf="http://xmlns.com/foaf/0.1/"
  xmlns:xpath="http://www.example.org/ns/xpath#"
  xmlns:ns="http://www.example.org/ns/namespace#">

  <earl:TestResult rdf:ID="error_1">
    <earl:validity
      rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
    <dc:description rdf:parseType="Literal">
            <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
                <p>Error: document type does not allow element
                <code>li</code> here; missing one of <code>ul</code>,
                <code>ol</code> start-tag.
                </p>
            </div>
    </dc:description>
    <xpath:xpath rdf:resource="#loc_1" />
  </earl:TestResult>
  <earl:TestResult rdf:ID="error_2">
    <earl:validity
      rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
    <dc:description rdf:parseType="Literal">
      <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <p>Error: end tag for <code>li</code>
        omitted, but OMITTAG NO was specified.
        </p>
      </div>
    </dc:description>
    <xpath:xpath rdf:resource="#loc_2" />
  </earl:TestResult>
  <earl:TestResult rdf:ID="error_3">
    <earl:validity
      rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail" />
    <dc:description rdf:parseType="Literal">
      <div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
        <p>Error: there is no attribute <code>alt</code>.</p>
      </div>
    </dc:description>
    <xpath:xpath rdf:resource="#loc_3" />
  </earl:TestResult>

  <xpath:XPath rdf:ID="loc_1">
    <xpath:expression>/x:html/x:body/x:ul/x:li[1]</xpath:expression>
    <ns:namespace rdf:resource="http://www.example.org/ns0" />
  </xpath:XPath>
  <xpath:XPath rdf:ID="loc_2">
    <xpath:expression>/x:html/x:body/x:ul/x:li[2]</xpath:expression>
    <ns:namespace rdf:resource="http://www.example.org/ns0" />
  </xpath:XPath>
  <xpath:XPath rdf:ID="loc_3">
    <xpath:expression>/x:html/x:body/x:p[2]</xpath:expression>
    <ns:namespace rdf:resource="http://www.example.org/ns0" />
  </xpath:XPath>
  <ns:Namespace rdf:about="http://www.example.org/ns0">
    <ns:prefix>x</ns:prefix>
    <ns:uri>http://www.w3.org/1999/xhtml</ns:uri>
  </ns:Namespace>

  <!-- ... -->

</rdf:RDF>

7. Conclusions and open issues

[Editor's note: Self-explanatory.]

Appendix A: References

[DC]

The Dublin Core Metadata Element Set - DC Recommendation, 20 December 2004.
http://www.dublincore.org/documents/dces/

[DCT]

The Dublin Core Metadata Terms - DC Recommendation, 13 June 2005.
http://www.dublincore.org/documents/dcmi-terms/

[FOAF]

FOAF Vocabulary Specification - Working Draft, 3 June 2005.
http://xmlns.com/foaf/0.1/

[HTTPRDF]

HTTP in RDF

[RDF]

Resource Description Framework (RDF) Model and Syntax Specification - W3C Recommendation, 22 February 1999.
http://www.w3.org/TR/1999/REC-rdf-syntax-19990222/

[RDF-PRIMER]

RDF Primer - W3C Recommendation, 10 February 2004.
http://www.w3.org/TR/rdf-primer/

[RDFS]

RDF Vocabulary Description Language 1.0: RDF Schema - W3C Recommendation, 10 February 2004.
http://www.w3.org/TR/rdf-schema/

[RDF-XML]

RDF/XML Syntax Specification (Revised) - W3C Recommendation 10 February 2004.
http://www.w3.org/TR/rdf-syntax-grammar/

[RDF-XML-DIFFS]

Why RDF model is different from the XML model - Paper by Tim Berners-Lee, September 1998.
http://www.w3.org/DesignIssues/RDF-XML

[RFC2119]

Key words for use in RFCs to Indicate Requirement Levels - IETF RFC, March 1997.
http://www.ietf.org/rfc/rfc2119.txt

[OWL]

OWL Web Ontology Language - W3C Recommendation, 10 February 2004.
http://www.w3.org/TR/owl-features/

[WCAG10]

Web Content Accessibility Guidelines 1.0 - W3C Recommendation, 5 May 1999.
http://www.w3.org/TR/WCAG10/

[XML]

[URI]

[XPath]

XML Path Language (XPath) Version 1.0 – James Clark, Steve DeRose, W3C Recommendation, 16 November 1999

To be completed.

Appendix B: Contributors

[Editor's note: To be determined.]