This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
The purpose of the Evaluation and Report Language (EARL) 1.0 Schema is to define a common vocabulary to express test results. It is intended as one part of a two-part replacement for the EARL 1.0 Working Draft of 6 December 2002. While this document focuses on the technical details of the specification, a companion document describes the motivations for EARL and provides a tutorial introduction to its use. [Editor's note: link EARL 1.0 Guide once available.] The ERT Working Group encourages feedback about this document by Web developers and researchers with interest in software supported evaluation and validation of Web sites. In particular, the group is looking for feedback about how well the schema addresses the description of accessibility test results, and how to best relate these results to resources that are available on the Web. Please refer to the EARL 1.0 Known Issues for more details and background about currently open issues. Questions are also highlighted within the appropriate sections of this document.
Please send comments to the mailing list of the Evaluation and Repair Tools Working Group (ERT WG). The archives for this list are publicly available.
This is a Working Group internal draft of the Evaluation and Report Language (EARL) 1.0 Schema. This document will be published and maintained as a W3C Recommendation after review and refinement. Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document was produced under the 5 February 2004 W3C Patent Policy. The Working Group maintains a public list of patent disclosures relevant to this document; that page also includes instructions for disclosing [and excluding] a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) with respect to this specification should disclose the information in accordance with section 6 of the W3C Patent Policy.
This document has been produced as part of the W3C Web Accessibility Initiative (WAI). The goals of the Evaluation and Repair Tools Working Group (ERT WG) are discussed in the Working Group charter. The ERT WG is part of the WAI Technical Activity.
[Editor's note: this section will be synchronized with the EARL 1.0 Guide as it is being developed.]
The Evaluation and Report Language (EARL) is a format to express test results. Test results include bug reports, test suite evaluations, and conformance claims. The test subject might be a Web site, an authoring tool, a user agent or some other entity. Thus, EARL is flexible. It enables any person, entity, or organization to state test results for any thing tested against any set of criteria.
Stating test results in EARL creates a variety of opportunities. The data can be:
- aggregated from different sources
- exchanged between tools
- used to create customized reports
- combined to compare how different test subjects fared on the same test.
A companion document to this specification is intended to provide more introductory material and explanation of the use cases for EARL.
EARL statements contain the following information:
- The context information
- This can include information about who or what ran the test, the date the test was run, and other information about the test was performed.
- The test subject
- This may include: Web pages, tools (e.g. accessibility checkers, validators), and user agents
- The result
- Did the test subject pass or fail the test? How certain can we be?
- Test criteria
- What are we evaluating the test subject against? This could be a specification, a set of guidelines, a test from a test suite, or some other test case.
Prose examples that demonstrate the above structure:
Example 1: a person carries out a manual evaluation of a Web page against a test criteria
- Context
- Mary Thompson claims on the 17th December
- Test Subject
- A Web page at
http://www.example.org/mypage
- Test Result
- Passed
- Test performed
- Checkpoint 1.1 of the Web Content Accessibility Guidelines 1.0
Example 2: an evaluation tool carries out automated evaluation of a Web page against a test criteria
- Context
- The W3C Markup Validator claims as of
2004-04-14T14:00:04+1000
- Test Subject
- The XHTML returned from a GET request to the URI
http://www.example.org/yourpage
- Test Result
- Failed
- Test Performed
- The validity of the XHTML
Editorial Comments
There are many issues noted which need to be resolved in future drafts. Comments are specifically requested on any of these issues, which are marked as follows:
[Editor's note: there are some items that the working group has discussed but not reached consensus, and others that are known issues which have not yet been discussed]
The key words MUST
, MUST NOT
, REQUIRED
, SHALL
, SHALL NOT
, SHOULD
, SHOULD NOT
, RECOMMENDED
, MAY
, and OPTIONAL
will be used in accordance with RFC 2119 [RFC2119]
Namespaces
The namespace for EARL as specified in this draft is http://www.w3.org/WAI/ER/EARL/nmg-strawman#.
[Editor's note: Versioning terms during the process of developing the vocabulary is an issue the group is working on. It is possible that a new namespace will be used for a final version of the vocabulary]
Where RDF terms are used in their abbreviated form (e.g. Assertion or foaf:Person), if no namespace is provided the term is in the EARL namespace. The following prefixes are used in examples throughout this document:
- earl
- The EARL namespace
http://www.w3.org/WAI/ER/EARL/nmg-strawman# which is described in this document
- foaf
- The FOAF namespace
http://xmlns.com/foaf/0.1/ which is also where the terms are described [FOAF]
- dc
- The Dublin Core elements namespace
http://purl.org/dc/elements/1.1/ whose terms are described in [DC]
- dct
- The Dublin Core terms namespace
http://purl.org/dc/terms/ also described at [DCT]
- rdf
- The RDF namespace
http://www.w3.org/1999/02/22-rdf-syntax-ns# described in [RDF]
- rdfs
- The RDF Schema namespace
http://www.w3.org/2000/01/rdf-schema# described in [RDFS]
- owl
- The OWL namespace
http://www.w3.org/2002/07/owl# described in [OWL]
EARL is defined as an RDF vocabulary. The Resource Description Framework (RDF) [RDF] is a general-purpose language for describing information in a way that is machine-understandable.
EARL is an RDF vocabulary used to make statements about how a resource performed against a test. In common with other RDF it assumes that other vocabularies will be used as appropriate.
For more information on RDF, please refer to the following references:
- RDF Primer [RDF-PRIMER]
- Resource Description Framework (RDF) Model and Syntax Specification [RDF]
- Why RDF model is different from the XML model [RDF-XML-DIFFS]
The EARL 1.0 defines an RDF Vocabulary that consists of classes and properties. This section describes this core vocabulary as well as its usage while later sections describe extensibility of EARL 1.0; and conformance to EARL 1.0.
[Editor's note: an image with the nodes and relationships would be helpful to visualize the schema.
An assertion is a statement about the results of performing a test. This is the fundamental part of an EARL statement.
An assertion MUST
have at least the following properties:
assertedBy- The assertion
MUST
be asserted by an assertor. The property assertedBy is defined by this specification.
subject- The thing that is being tested for conformance to some requirement. The property
subject is defined by this specification.
testcase- The requirement that is being tested. The property
testCase is defined by this specification.
result- The result of the test - whether the subject passes or fails the test case (or there is some other result). The property
result is defined by this specification.
mode- How the test was performed - as an automated computer process, by a human making a subjective judgement, or otherwise. The property
mode is defined by this specification.
Example 3: instance of an assertion expressed as an RDF/XML fragment
<earl:Assertion rdf:about="http://example.org/#assertion-1">
<earl:assertedBy rdf:resource="http://example.org/#assertor123" />
<earl:subject rdf:resource="http://example.org/#someID02495"/>
<earl:testcase rdf:resource="http://example.org/#testcase-1"/>
<earl:result rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#pass"/>
<earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#manual"/>
</earl:Assertion>
An assertor states the results of a test (i.e. an assertor asserts and assertion). An assertor must be either a person or a tool of class Assertor which is defined by this specification. An assertor MUST
belong to one of the following types:
foaf:Person- The Assertor is a human being. This uses the FOAF vocabulary term
Person for a person. There should be identifying information including a name, and a uniquely identifying property such as email address or an encrypted email address. Again, the properties name, mbox and mbox_sha1sum are defined by FOAF [FOAF].
- [Editor's note: the use of FOAF is subject to review of the stability of these terms.]
- [Editor's note: the current requirements on properties to identify the tester are subject to revision.]
PersonUsingTool- A person may have made their evaluation with the help of a tool. In this case EARL provides
PersonUsingTool class, and the usingTool and user properties to identify the people and tools concerned.
Software- The Assertor is software, such as a black box testing tool of some sort or an evaluation and repair tool. Software must have a title (using the Dublin Core property title) and may have version information (using Dublin Core terms) [Editor's note: the current requirements on properties to identify a tool are subject to revision, please see section software for more information]
The assertor in the following example is a person using software. Their personal email address is used to uniquely identify them, in both unencrypted and encrypted forms (it is not necessary to use both of these methods, but it is permitted). The test was done using a tool called "My Favorite Tool".
Example 4: an assertor person called Bob B. Bobbington using a software to carry out an assertion
<earl:PersonUsingTool rdf:about="http://example.org/#assertor123">
<earl:user>
<foaf:Person>
<foaf:name>Bob B. Bobbington</earl:name>
<foaf:mbox rdf:resource="mailto:bob@example.org"/>
<foaf:mbox_sha1sum>1a9daad476f0158b81bc66b7b27b438b4b4c19c0</foaf:mbox_sha1sum>
</foaf:Person>
</earl:user>
<earl:usingTool>
<earl:Software rdf:about="http://example.com/toolshop/MFT#20050101b2">
<dc:title xml:lang="en">My Favorite Tool</dc:title>
</earl:Software>
</earl:usingTool>
</earl:Assertor>
A Test Case is a test - usually one that can either be passed or failed. This may include many things such as validation requirements, code test cases, checkpoints from guidelines such as Web Content Accessibility Guidelines 1.0 [WCAG10], etc. These SHOULD
be identified with a URI.
A Test Case may be a single test, or may be a part of a larger compound test suite. These relations can be described using Dublin Core's hasPart or isPartOf properties.
[Editor's note: This class is included for convenience, since it allows various useful properties to be described in simple standard RDF. The working group will deprecate this class if they find an appropriate replacement from a language designed for describing test cases, something which is beyond the scope of the current specification.]
The following example uses the Test Case class to identify a test case for an Assertion:
Example 5: instance of a test case that is additionally described with a title and its relationship to a test suite
<earl:testcase>
<earl:TestCase rdf:about="http://example.org/#testcase-1">
<dc:title xml:lang="en">A sample test</dc:title>
<dc:isPartOf rdf:resource="http://example.org/#testsuite-517"</dc:isPartOf>
</earl:TestCase>
</earl:testcase>
The value of the testcase property is by definition a Test Case class so the above example can be shortened to:
Example 6: instance of a test case property with an empty Test Case class
<earl:testcase rdf:resource="http://example.org/#testcase-1" />
The value of the mode property must be exactly one test mode (or subclass of a test mode - see extensibility for more explanation). The modes defined are
manual- Where the test was performed based on a person's judgement. This includes the case where that judgement was aided through the use of a tool, although this should be noted with the
usingTool property.
automatic- Where a computer program or similar has done an automated test and generated a result without human assistance.
heuristic- [Editor's note: this property is under discussion and may be changed]
- This property was designed to cover assertions which are made by inference, for example based on several existing test results.
Example 7: instance of an assertion that has been carried out in manual mode
<earl:Assertion rdf:about="http://example.org/#assertion-1">
<earl:mode rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#manual"/>
</earl:Assertion>
The result of the test. This MUST
have exactly one validity property.
It MAY
also include a description, using the Dublin Core description property. This is meant to be human-readable text or markup (for example HTML included as an XML Literal). This can be inserted by a tool or by a human evaluator.
There is a confidence property which allows an evaluator to assert their confidence in the result. This MAY
be used where a tool wants to assert that it has difference levels of confidence about two possible results (for example low confidence that a test has been passed, but also high confidence that it is not applicable. [Editor's note: the confidence property is still under discussion, please refer to the confidence property for more information]
The following example shows a result with a validity of fail and a description of the problem in XHTML:
Example 8: A test result with a validity and a description property
<earl:result rdf:parseType="Resource">
<earl:validity rdf:resource="http://www.w3.org/WAI/ER/EARL/nmg-strawman#fail"/>
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml">
<p>The <code>table</code> element is not allowed to appear
inside a <code>p</code> element</p>
</div>
</dc:description>
</earl:result>
[Editor's note: there are currently value specified as confidence values, of high, medium, and low, but these are under discussion and may be removed or significantly revised since it is not yet clear how to ensure that they can be used interoperably]
Confidence in the result by an assertor. The values of the confidence level MUST
be one of the following:
high- An assertor is very confident of the result
medium- There may be some uncertainty in the result
low- This result is not reliable for further usage
Outcome of a test. The values of the validity level MUST
be one of the following:
cannotTell- An assertor can not tell for sure what the outcome of the test is. Usually this happens when an automated test requires human judgement to decide between
pass or fail
fail- The
subject did not meet the criteria defined by the testcase
notApplicable- A test is not applicable to the given
subject
notTested- Test has not been carried out. This is useful for reporting as well as for other uses of progress monitoring
pass- An assertor is claims a test passed successfully.
Validity level can also be customized subclasses of these values, please refer to the section on extensibility for more information about this.
The class of things that have been evaluated. The Test Subject call MUST
have exactly one (Dublin Core) Date property. It MAY
also have any of the following properties:
dc:title- Human readable title for the subject (e.g. "Home Page")
dc:description- Human readable description of the subject (e.g. "Node page of the Web site"
dc:hasPart- Relationship to other subjects that are part of this subject
dc:isPartOf- Relationship to other subjects of which this subject is a part of
This class is intentionally generic to serve a wide variety of usages. The two classes Software and Web Content can be used for more specific use cases.
Example 9: a test subject that is a file in C programming language and that is part of an application
<earl:TestSubject rdf:about="http://example.org/#code.c"/>
<dc:title xml:lang="en">IO functions</dc:title>
<dc:description xml:lang="en">functions used to input and output data such as writing to disk</dc:description>
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#gDate">2005-06-25</dc:date>
<dc:isPartOf rdf:resource="http://example.org/#application"</dc:isPartOf>
</earl:TestSubject>
A piece of software such as an authoring tool, browser, or evaluation tool. Software must have a title (using the Dublin Core property title) and may have version information (using Dublin Core terms). Additionally, if it is used as a test subject then it inherits additional properties such as title, description, and date.
[Editor's note: Tool used to be a sub class of Assertor. The term Tool has been deprecated in favour of Software, which may also be a Test Subject]
Example 10: instance of a software which may used as an assertor or a test subject depending on how it is used in an assertion
<earl:Software rdf:about="http://example.org/#someID02495"/>
<dc:title xml:lang="en">TestThisTool version 4</dc:title>
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#gDate">2005-06-25</dc:date>
</earl:Software>
Information on the World Wide Web. This should include enough information to identify the content actually tested, taking account of content type and language negotiation and similar factors that can change the content actually served. This class is intended to be used as a test subject but can be reused for other purposes (outside EARL) as well. The Web content class has the following properties:
dc:format- Format of the Web resource. For example "text/HTML", "application/PNG" or other mime types that re returned from the Web server.
dc:location- The URL or other types of pointers that identify an address for the resource.
httpRequest- HTTP request sent by the client to the server in order to initiate a response
httpResponse- HTTP response (except the actual payload of the date) sent by the server to the client
A Web content class MAY
include several request/response sequences to document the content and language negotiation that took place before the content was finally sent.
[Editor's note: the request/response properties of this class are still being refined]
This section records current changes and future plans for changes as the working draft matures to Recommendation level.
- Document split into a specification (this document) and a companion document. Most content stripped out of specification document, and new issues under discussion noted.
- Schema presented without the use of XML entities
- Typographical errors in schema fixed, so it validates
- Re-ordering of the Schema and document. Instead of being organised as a list of classes, then properties, then instance classes, the document and the schema are both organised around the key classes as the framework for defining their use.
- Redundant (rdf:Resource) and incorrect (rdfs:Resource) sub-class declarations removed
- Schema labels given xml:lang attributes as appropriate
- Labels converted to grammatical english (from camelCaseAlmostWords)
- Schema encoded in default UTF-8
New terms have been introduced to the vocabulary, sometimes as replacements for deprecated terms that existed in EARLier drafts, sometime to introduce new funtionality
This draft
Software- This term replaces the deprecated
Tool and is not a subclass of Assertor, so can be used to describe any type of tool.
PersonUsingTool- This term provides a class of Assertor which describes an evaluator whoes decision is assisted by a particular tool or tools.
usingTool, user- These are properties of a Person using a tool, designed to identify the people and tools concerned.
httpRequest, httpResponse- These are placeholder properties for describing
WebContent
A number of terms have been deprecated, either in favour of terms from existing widely-used vocabularies, or because their use is somehow problematic. While this deprecation is provisional, no RDF has been published formally noting them as deprecated (for example through OWL). A quick guide to terms that have been provisionally deprecated:
This draft
reprOf- This term was deprecated in favour of a more comprehensive approach to describing Web content.
UserAgent, Tool- These terms were deprecated in favour of the more generic
Software class.
platform- This term was deprecated as it did not seem to be widely accepted and implemented.
message, format- These terms were deprecated in favour of the Dublin Core terms
dc:description and dc:format.
Person, name, email, contactInfo- These terms were deprecated in favour of equivalent terms from the FOAF vocabulary.
The working group is considering several open issues which would lead to extending the current vocabulary before it reaches recommendation.
As well as issues noted in the draft, the following issues are under active consideration:
Evidence
A mechanism to provide further evidence used to make a claim. This could be a copy of the web content analysed for a particular test, or a rule describing how may tests are required to pass some compund test, and pointers to the individual results that together satisfy the rule.
Location
Within Web content, for the use case of testing accessibility, it is very valuable to point to the location(s) that cause the test to pass or fail. This is also true of software interfaces and various other kinds of subject. THe working group has agreed in principle to include a mechanism for recording this information in an EARL report, but has not yet developed or found the necessary terms.
HTTP transaction information
There are currently placeholder properties httpRequest and httpResponse for HTTP transaction information. The group expects to develop these further, either themselves or with an existing vocabulary.
Confidence
This is under review by the working group. It is not clear how the currently defined model can be used interoperably, and whether there is an alternative approch that would allow for a more interoperable statement of confidence in some results.
EARL is the result of the work of many people over the past. The editors would particularly like to thank Wendy Chisholm, Sean B Palmer, and Daniel Dardailler, who edited the first versions of the EARL specifications. The editors apologise for any names left out of this list, and will endeavour to rectify any errors noted in comments.
Contributors to this Working Draft
Shadi Abou-Zahra, Chrisoula Alexandraki, Myriam Arrue, Gabriele Bartolini, Giorgio Brajnik, Wendy Chisholm, Karl Dubost, Dominique Hazael-Massieux, Sandor Herramhof, Carlos Iglesias, Nick Kew, Johannes Koch, Jim Ley, Charles McCathieNevile, and Chris Ridpath.
Contributors to previous Work Drafts
Giorgio Brajnik, Dan Brickley, Karl Dubost, Nick Gibbins, Al Gilman, Dominique Hazael-Massieux, Nadia Heninger, Ian Hickson, Leonard Kasday, Nick Kew, Jim Ley, William Loughborough, John Lutts, Charles McCathieNevile, Libby Miller, Tom Martin, Dave Pawson, Eric Prud'hommeaux, Pierre Queinnec, Chris Ridpath, Romain Roure, Aaron Swartz, Olivier Thoreaux, and Rob Yonaitis.