[contents]
This document is an introductory guide to the
Evaluation and Report Language (EARL) 1.0, and to accompany the
normative document Evaluation and Report
Language (EARL) 1.0
Schema. The Evaluation and Report Language is a framework for expressing test
results. Although the term test can be taken in its most widely accepted definition,
EARL is primarily intended for reporting and exchanging results of
tests of Web applications and resources. EARL is intended as a
vendor-neutral and platform-independent format.
EARL is expressed in the form of an
RDF vocabulary. The Resource Description Framework
(RDF) is a language for representing semantically information about
resources in the World Wide Web. However, EARL is not conceptually restricted to
these resources and could be applied in other scenarios.
This
section describes the status of this document at the time of its publication. Other
documents may supersede this document. A list of current W3C publications and the latest revision of
this technical report can be found in the W3C technical reports index at
http://www.w3.org/TR/.
[Editor's note: describe intent of this
working draft and propose feedback questions. Synchronize with EARL 1.0 Schema.]
Please
send comments to the mailing list of the
ERT WG. The archives for this list
are publicly available.
This is a W3C Working Draft of the Evaluation and
Report Language (EARL) 1.0
Guide. This document will be published and maintained as a W3C Recommendation after review and
refinement. Publication as a Working Draft does not imply endorsement by the W3C Membership. This is a draft document and
may be updated, replaced or obsoleted by other documents at any time. It is
inappropriate to cite this document as other than work in progress.
This document
was produced under the 5
February 2004 W3C Patent Policy. The Working Group maintains a public list of patent
disclosures relevant to this document; that page also includes instructions for
disclosing a patent. An individual who has actual knowledge of a patent which the
individual believes contains Essential Claim(s) with respect to this specification
should disclose the information in accordance with section 6 of
the W3C Patent Policy.
This document has been produced as part of the W3C Web
Accessibility Initiative (WAI). The goals of the Evaluation and Repair Tools Working Group (ERT WG) are discussed in the Working Group charter. The ERT WG is part of the WAI Technical Activity.
This document provides an introductory
guide to the Evaluation and Report Language (EARL) 1.0, and is
intended to accompany the normative document Evaluation and Report Language (EARL) 1.0 Schema.
The
objectives of this document are:
- To provide an introduction to the use of EARL in different scenarios.
- To clarify the key concepts of EARL and their translation into the
different classes.
- To explain the restrictions of the different EARL components.
- To show how to aggregate and process EARL reports.
- To demonstrate how to extend and customise EARL.
Although this document does not assume any previous knowledge
of EARL, the following
background knowledge is required:
- Basic knowledge of XML [XML] and its
associated technologies.
- Basic knowledge about the Semantic Web and RDF. For
references, consult [RDF], [RDF-PRIMER] and [RDFS].
Although the concepts of the Semantic Web are simple, their abstraction with
RDF is know to bring difficulties to beginners. It is recommended to
read carefully the aforementioned references and other tutorials found on the Web. It
must be also borne in mind that RDF is primarily targeted to be
machine processable, and therefore, some its expressions are not very intuitive for
developers used to work with XML only.
The Evaluation and
Report Language (EARL) is a
framework targeted to express and compare test results. EARL builds on top of the Resource
Description Framework [RDF], which is the basis for the Semantic
Web. It is not the object of this document to introduce the reader to the intricacies of
RDF, and some basic knowledge
must be assumed as a pre-requisite (see, e.g., [RDF-PRIMER]
for more information). Like any RDF vocabulary, EARL is no more than a collection of
statements about resources, each with a subject, a predicate (or a verb) and an object.
These statements can be serialized in many ways (e.g., RDF/XML, or Notation 3 also known as N3). A typical EARL report could contain the following
statements (oversimplifying the notation and not including namespaces):
<#someone> <#checks> <#resource> .
<#resource> <#fails> <#test> .
From these simple two statements,
it can be inferred already the main components of an EARL Report (wrapped up in an
assertion):
- Who (or which tool) runs a test: this is known in the EARL
terminology as the Assertor.
- The resource tested: known as the Test
Subject.
- The tested criterion(-a): known as the Test
Criterion.
- The result(s) of the test: known as the Test
Result.
This structure shows the universal applicability of EARL and its ability to refer to any
type of test: bug reports, software unit tests, test suite evaluations, conformance
claims or even tests outside the world of software and the World Wide Web (although in
such cases, there might also be open issues for its full aplicability). It must be
stressed again the semantic nature of EARL: its purpose is to facilitate the
extraction and comparison of test results by humans and especially by tools (the
semantic Web paradigm); it is not simply an storage of information, for which some other
XML application might be more
suitable.
Initially, EARL was created as a way to create,
merge and compare Web accessibility reports from different sources (tools, experts,
etc.). However, this original aim has been expanded to cover wider testing scenarios.
Summarising, the objectives of EARL are to:
- Create a standardised way to produce test reports;
- Support the exchange of reports between testers (humans or testing tools);
- Facilitate the comparison of test results; and
- Ease the aggregation of test results (e.g., like a different set of tests on the
same subject).
It is also important that the extensibility of RDF (or EARL) allows to tool vendors or
developers the addition of new functionalities to the vocabulary, without losing any of
the aforementioned characteristics, as other testers might ignore those extensions that
they do not understand when processing third party results.
The applicability of EARL to different scenarios can be seen
in the following use cases:
- Evaluating a Web site using tools in different languages
- A group of people speaking different languages are evaluating a Web site for
conformance to different legal environments, such as, e.g., Section 508 in the USA
and BITV in Germany. The use of EARL:
- allows localized messages explaining where problems are met. The report can
contain messages in the languages spoken by the evaluators so that each of them
understands the messages.
- allows "keywords" to express the conformance level reached by the Web site
that are language-independent. Thus a software tool can translate the validity
levels in different languages.
- Combining results from different evaluation tools
- A Web site evaluator uses different tools for the task. Each tool can perform
specific tests that the other tools cannot do. The evaluator's client wants a
complete evaluation report. All the evaluation tools used produce a report in
EARL format. Therefore,
the evaluator can combine the separate reports into one bigger report, query the
results, and offer to her customer statistical reports and a detailed conformance
claim that specifies where the Web site does not meet the required level.
- Comparing results from different tools
- A Web site evaluator uses different tools for evaluation. The tools perform the
same tests. All the evaluation tools used produce a report in EARL format. Therefore, the
evaluator can compare the results from different tools to increase the confidence
level of the test results. It will also help to make assertions about a given
resource, when one of the tools is only able to give a warning on a problem, but the
other performs a thorough test that removes the aforementioned uncertainty.
- Benchmarking an evaluation tool against a test suite
- For a benchmarking test of a given provider, different tools perform their tests
on sample documents from a test suite. Some evaluation tools may produce false
positives or false negatives. All of them create an EARL report with the result.
Comparing the results of the tools with a theoretical output file from the test
suite, evaluation tools could be rated according to accuracy against the test
suite.
- Monitoring a Web site over time (quality assurance)
- A Web project manager wants to track the accessibility of a Web site over time
by comparing current test results with previous ones. The reports contain the
date/time of the tests and a way to locate the parts of the document the messages
refer to. By comparing messages referring to the same locations the project manager
can monitor possible improvements, and allocate resources to solve problems in the
critical areas of the Web site.
- Exchanging data with repair tools
- A repair tool (or a similar module in an authoring tool) uses the results of an
evaluation tool to identify the parts of the document that need to be fixed. For
each instance of an error it provides a way for the user to notice the error and fix
the document. The same scenario can be used for Content Management Systems that wish
to integrate an evaluation tool into their workflow, helping to locate accessibility
and validation problems to Web editors.
- Exchanging data with search engines
- A search engine uses a third-party service which publishes EARL reports of Web sites. The user
interface lets the user choose between different levels of accessibility. The list
of search results contains only documents with a chosen accessibility level. The
search engine uses the test results in the calculation of the ranking/relevance, so
that it affects the search results order.
[Editor's note: Maybe add some more exotic scenario outside the
Web and software development.]
EARL is flexible enough to respond to
the needs of a variety of audiences involved in a testing or quality assurance process.
Typical profiles are:
- Web commissioner
- Some private or public Web site owner commissions her site to an external
agency, and wishes to monitor the site's compliance with a given set of legal or
internal quality requirements;
- Product manager
- Responsible for delivering a given product;
- Product designer
- Designs the product and documents this in a design specification;
- Quality engineer or tester
- Takes the product through a series of tests to find bugs; and
- Developer
- Creates a product to satisfy the design specification; fixes bugs found by the
quality engineer or tester.
The Standard [IEEE-829] defines a set of
basic software tests documents. Of course, the scope of EARL is
limited to the reporting process, but an analysis of [IEEE-829] will help us to fit EARL in any
generic test process (extrapolated from a software testing environment). For [IEEE-829], the three key documents for a testing process
are:
- Test plan, which prescribes the scope, approach, resources, and schedule of the
testing activities. This part is mainly a management planning, and lies outside the
capabilities of EARL, although the resources to be tested can be
expressed within EARL.
- Test specification, which covers three items: Test Design Specification, Test
Case Specification and Test Procedure Specification. Once again,
EARL could be used to identify Test Cases (or test criteria),
although the language is not targeted to support the different aspects of the
specifications in the standard. For this target, a Test Case Description Language
will be more appropriate.
- Test reporting. This is where EARL fits better, although it must be acknowledged
that the core language is built to support extensibility and its classes are not
directly designed to support the [IEEE-829]
standard.
Figure 1 displays graphically the aforementioned
elements:
EARL is
not an standalone vocabulary, and builds on top of many existing vocabularies that cover
some of its needs for metadata definition. This approach avoids the re-creation of
applications already established and tested like the Dublin Core elements. The
referenced specifications are:
- Dublin Core Metadata Initiative: The Dublin
Core is a metadata standard for describing digital resources, often expressed in
XML. The first standard published is the Dublin Core Metadata Element Set.
It consists of 16 optional metadata elements, any of which may be repeated or
omitted. Typical elements are: Title, Creator, Subject, Description, Publisher,
Contributor, Date, Type, Format, Identifier, Source, Language, etc. The Dublin Core
Metadata Element Set was accepted as a NISO standard in 2001 (ANSI/NISO Z39.85-2001)
and as an ISO standard in 2003 (ISO 15836:2003(E)).
- Friend of a Friend (FOAF) project:
The FOAF project is about creating a Web of machine-readable
resources describing people, the links between them and the things they create and
do. Of particular interest for EARL are the Classes
foaf:Person and foaf:Project [FOAF].
- HTTP vocabulary
in RDF: This is an RDF vocabulary used to represent
HTTP requests and responses, and it is useful to identify online
resources accessed via HTTP, which cannot be uniquely resolved
via a URI. Typical examples would be servers accessed via content negotiation, Web
applications using POST requests, etc.
- EARL Pointers.
RDF can be serialized in different ways, but the
XML representation [RDF/XML] is the
preferred method and will be used throughout this document. However, even when selecting
this approach, there are many equivalent ways to express an RDF
model.
Table 1 presents the core namespaces used by EARL. The prefix refers to the
convention used in this document to denote a given namespace, and can be freely
modified.
Table 1: EARL namespaces.
| Namespace prefix |
Namespace URI |
Description |
earl |
http://www.w3.org/ns/earl# |
The default EARL
namespace. Where RDF
terms are used in their abbreviated form (e.g., Assertion or
foaf:Person), if no namespace is provided the term is in the EARL namespace. |
rdf |
http://www.w3.org/1999/02/22-rdf-syntax-ns# |
Default RDF
namespace [RDF]. |
rdfs |
http://www.w3.org/2000/01/rdf-schema# |
Default RDF schema
namespace [RDFS]. |
owl |
http://www.w3.org/2002/07/owl# |
Default OWL namespace [OWL]. |
dc |
http://purl.org/dc/elements/1.1/ |
Dublin Core Metadata Element Set namespace. |
dct |
http://purl.org/dc/terms/ |
Dublin Core Metadata Terms namespace. |
foaf |
http://xmlns.com/foaf/0.1/ |
FOAF namespace. |
http |
http://www.w3.org/2006/http# |
Default HTTP in RDF namespace. |
http-header |
http://www.w3.org/2006/http-header# |
Default HTTP headers in RDF
namespace. |
Most of
the examples in this guide are using the RDF/XML serialization RDF/XML. Occasionally, other serializations may be
used.
Let us start building our first EARL report with its root
element. The root element of any EARL report is an
RDF node, as with any RDF vocabulary. There, we
declare the corresponding namespaces, as described in Table
1, plus any custom namespace used to define additional classes and or
properties.
Once we have
defined the root element, let us build step-by-step the different components of a simple
EARL report. Let us assume we want to express the results of an XHTML
validation in a given document with the W3C HTML
Validator in EARL. The tested document has the following HTML
code:
Example
3.2. An XHTML document to be validated.
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
"http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html lang="en" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>Example of project pages</title>
</head>
<body>
<h1>Project description</h1>
<h2>My project name</h2>
<p>The strategic goal of this project is to make you understand EARL.</p>
<ul>
<li>Here comes objective 1.
<li>Here comes objective 2.</li>
</ul>
<p alt="what?">And goodbye ...</p>
</body>
</html>This document has three errors that will constitute the basis
of our EARL report:
- Error: Line 14 column 7: document type does not allow element "
li"
here; missing one of "ul", "ol" start-tag.
- Error: Line 15 column 6: end tag for "
li" omitted, but OMITTAG NO
was specified.
- Error: Line 16 column 9: there is no attribute "
alt".
Let us start by defining who (or what) run the test. In the
EARL jargon, that is an Assertor. There are two types of
Assertors, SingleAssertor and CompoundAssertor.
Assuming the test was run by a person with the W3C HTML Validator, we can use a
CompoundAssertor with the following structure.
Example 3.3.
EARL report with a compound assertor.
<rdf:RDF
xmlns:http="http://www.w3.org/2006/12/http#"
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:CompoundAssertor rdf:ID="assertor01">
<earl:helpAssertor>
<foaf:Person>
<foaf:mbox rdf:resource="mailto:john@example.org"/>
<foaf:name>John Doe</foaf:name>
</foaf:Person>
</earl:helpAssertor>
<earl:mainAssertor rdf:resource="http://validator.w3.org/about.html#"/>
<dc:title>John Doe and the W3C HTML Validator</dc:title>
</earl:CompoundAssertor>
<earl:SingleAssertor rdf:about="http://validator.w3.org/about.html#">
<dct:hasVersion>0.7.1</dct:hasVersion>
<dc:description xml:lang="en">W3C Markup Validation Service, a free
service that checks Web documents in formats like HTML and XHTML
for conformance to W3C Recommendations and other standards.
</dc:description>
<dc:title xml:lang="en">W3C HTML Validator</dc:title>
</earl:SingleAssertor>
<!-- ... -->
</rdf:RDF>To begin to gather the richness of RDF, the previous
snippet can be expressed differently. First, the help assertor can be expressed as an
anonymous node, and the W3C HTML
Validator could be expressed as a Software class, one of the
SingleAssertor classes of EARL.
Example 3.4. EARL
report with a compound assertor. The help assertor is expressed as anonymous
node.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:CompoundAssertor rdf:ID="assertor01">
<earl:helpAssertor>
<foaf:Person>
<foaf:mbox rdf:resource="mailto:john@example.org"/>
<foaf:name>John Doe</foaf:name>
</foaf:Person>
</earl:helpAssertor>
<earl:mainAssertor>
<earl:Software rdf:about="http://validator.w3.org/about.html#"/>
</earl:mainAssertor>
<dc:title>John Doe and the W3C HTML Validator</dc:title>
</earl:CompoundAssertor>
<earl:Software rdf:about="http://validator.w3.org/about.html#">
<dct:hasVersion>0.7.1</dct:hasVersion>
<dc:description xml:lang="en">W3C Markup Validation Service,
a free service that checks Web documents in formats like
HTML and XHTML for conformance to W3C Recommendations and
other standards.
</dc:description>
<dc:title xml:lang="en">W3C HTML Validator</dc:title>
</earl:Software>
<!-- ... -->
</rdf:RDF>Notice that this is only one of multiple serializations via
an anonymous node. This representation is equivalent, for instance, to the following
non-abbreviated mode.
Example 3.5. EARL report with a
compound assertor: non-abbreviated XML serialization.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#" >
<rdf:Description rdf:about="#assertor01">
<earl:helpAssertor rdf:nodeID="johnDoe"/>
<earl:mainAssertor rdf:resource="http://validator.w3.org/about.html#"/>
<dc:title>John Doe and the W3C HTML Validator</dc:title>
<rdf:type rdf:resource="http://www.w3.org/ns/earl#CompoundAssertor"/>
</rdf:Description>
<rdf:Description rdf:nodeID="johnDoe">
<foaf:mbox rdf:resource="mailto:john@example.org"/>
<foaf:name>John Doe</foaf:name>
<rdf:type rdf:resource="http://xmlns.com/foaf/0.1/Person"/>
</rdf:Description>
<rdf:Description rdf:about="http://validator.w3.org/about.html#">
<dct:hasVersion>0.7.1</dct:hasVersion>
<dc:description xml:lang="en">W3C Markup Validation Service,
a free service that checks Web documents in formats like
HTML and XHTML for conformance to W3C Recommendations and
other standards.
</dc:description>
<dc:title xml:lang="en">W3C HTML Validator</dc:title>
<rdf:type rdf:resource="http://www.w3.org/ns/earl#Software"/>
</rdf:Description>
<!-- ... -->
</rdf:RDF>The same report could be expressed in Tim Berners-Lee's N3 language in
the following way.
Example 3.6. EARL report with a
compound assertor: N3 serialization.
@prefix earl: <http://www.w3.org/ns/earl#> .
@prefix foaf: <http://xmlns.com/foaf/0.1/> .
@prefix dct: <http://purl.org/dc/terms/> .
<http://www.example.org/earl/report#assertor01>
a earl:CompoundAssertor ;
<http://purl.org/dc/elements/1.1/title>
"John Doe and the W3C HTML Validator" ;
earl:helpAssertor
[ a foaf:Person ;
foaf:mbox <mailto:john@example.org> ;
foaf:name "John Doe"
] ;
earl:mainAssertor <http://validator.w3.org/about.html#> .
<http://validator.w3.org/about.html#>
a earl:Software ;
<http://purl.org/dc/elements/1.1/description>
"""W3C Markup Validation Service, a free service that checks
Web documents in formats like HTML and XHTML for conformance
to W3C Recommendations and other standards."""@en ;
<http://purl.org/dc/elements/1.1/title>
"W3C HTML Validator"@en ;
dct:hasVersion "0.7.1" .The next step is to define the tested
resource. For that, EARL offers the TestSubject class.
This class is a generic wrapper for things to be tested like Web resources
(Content) or software (Software). For our first example, let
us use the generic TestSubject class, which must have a Dublin Core date
property:
Example 3.7. EARL report with a
test subject.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:TestSubject rdf:about="http://example.org/resource/index.html">
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date"
>2006-02-14</dc:date>
<dc:title xml:lang="en">Project Description</dc:title>
</earl:TestSubject>
<!-- ... -->
</rdf:RDF>[Editor's note: Probably is more adequate
express URLs of TestSubject in terms of the HTTP in RDF Vocabulary.]
Which
expressed as N3 serialization looks like the following
example.
Example
3.8. EARL report with a test subject: N3
serialization.
@prefix earl: <http://www.w3.org/ns/earl#> .
<http://example.org/resource/index.html>
<http://purl.org/dc/elements/1.1/date>
"2006-02-14"^^<http://www.w3.org/2001/XMLSchema#date> .
<http://example.org/resource/index.html>
<http://purl.org/dc/elements/1.1/title> "Project Description"@en .
<http://example.org/resource/index.html> a earl:TestSubject .The
third step is to define the criterion used for testing the given resource. EARL defines
test criteria under the umbrella of the TestCriterion class. This class has
two subclasses, TestRequirement and TestCase, depending on
whether the criterion is a high level requirement, composed of many tests, or an atomic
test case. In our example, we are testing validity with XHTML 1.0 Strict, which could be expressed in
the following way via the TestRequirement class:
Example 3.9.
EARL report displaying the test requirement for validation.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:TestRequirement rdf:about="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<dc:description xml:lang="en">DTD for XHTML 1.0 Strict.</dc:description>
<dc:title xml:lang="en">XHTML 1.0 Strict Document Type Definition</dc:title>
</earl:TestRequirement>
<!-- ... -->
</rdf:RDF>Before we build up our assertion about our hypothetical
evaluation, let us construct the result of the test with the three errors highlighted by
the validator. This is done via the TestResult class. At this stage, we
will not discuss in detail the description of error locations: we will limit ourselves
to textual description of the errors.
Example 3.10. EARL report displaying different error
results from the validation test.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:TestResult rdf:ID="error1">
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Error -
Line 14 column 7: document type does not allow element <code>li</code>
here; missing one of <code>ul</code>, <code>ol</code> start-tag.</p>
</div>
</dc:description>
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
</earl:TestResult>
<earl:TestResult rdf:ID="error2">
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Error -
Line 15 column 6: end tag for <code>li</code> omitted, but OMITTAG
NO was specified.</p></div>
</dc:description>
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
</earl:TestResult>
<earl:TestResult rdf:ID="error3">
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Error -
Line 16 column 9: there is no attribute<code>alt</code>.</p></div>
</dc:description>
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
</earl:TestResult>
<!-- ... -->
</rdf:RDF>For the sake of completeness, we have included the three
validation errors. Finally, we are ready to put together all the pieces of the puzzle.
As discussed earlier, this is done via assertions. An Assertion is an
statement that links together who made the test, what was tested, which requirements
were analyzed for compliance and the result of the test. It must have the following
properties:
earl:assertedBy- The
Assertion must be asserted by an assertor. The assertor is a
human or software, or groups of these, that determine the result.
earl:subject- The thing that is being tested against some given requirements or test
cases.
earl:test- The requirement that is used to test a subject.
earl:result- The result of the test, i.e., whether the subject passes or fails the test case
(or there is some other result).
Example
3.11. A set of EARL assertions pointing to components defined in the previous
examples.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xml:base="http://www.example.org/earl/report#">
<earl:Assertion rdf:ID="ass3">
<earl:result rdf:resource="#error3"/>
<earl:test rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
<earl:assertedBy rdf:resource="#assertor01"/>
</earl:Assertion>
<earl:Assertion rdf:ID="ass1">
<earl:result rdf:resource="#error1"/>
<earl:test rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
<earl:assertedBy rdf:resource="#assertor01"/>
</earl:Assertion>
<earl:Assertion rdf:ID="ass2">
<earl:result rdf:resource="#error2"/>
<earl:test rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
<earl:assertedBy rdf:resource="#assertor01"/>
</earl:Assertion>
<!-- ... -->
</rdf:RDF>We can now put together all the elements of our first EARL
report:
Example
3.12. Our first complete EARL report.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:Assertion rdf:ID="ass1">
<earl:assertedBy rdf:resource="#assertor01"/>
<earl:result rdf:resource="#error1"/>
<earl:test rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
</earl:Assertion>
<earl:Assertion rdf:ID="ass2">
<earl:assertedBy rdf:resource="#assertor01"/>
<earl:result rdf:resource="#error2"/>
<earl:test rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
</earl:Assertion>
<earl:Assertion rdf:ID="ass3">
<earl:assertedBy rdf:resource="#assertor01"/>
<earl:result rdf:resource="#error3"/>
<earl:test rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
</earl:Assertion>
<earl:SingleAssertor rdf:about="http://validator.w3.org/about.html#">
<dct:hasVersion>0.7.1</dct:hasVersion>
<dc:description xml:lang="en">W3C Markup Validation Service, a free
service that checks Web documents in formats like HTML and XHTML
for conformance to W3C Recommendations and other standards.
</dc:description>
<dc:title xml:lang="en">W3C HTML Validator</dc:title>
</earl:SingleAssertor>
<earl:CompoundAssertor rdf:ID="assertor01">
<earl:helpAssertor>
<foaf:Person>
<foaf:mbox rdf:resource="mailto:john@example.org"/>
<foaf:name>John Doe</foaf:name>
</foaf:Person>
</earl:helpAssertor>
<earl:mainAssertor rdf:resource="http://validator.w3.org/about.html#"/>
<dc:title>John Doe and the W3C HTML Validator</dc:title>
</earl:CompoundAssertor>
<earl:TestSubject rdf:about="http://example.org/resource/index.html">
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date"
>2006-02-14</dc:date>
<dc:title xml:lang="en">Project Description</dc:title>
</earl:TestSubject>
<earl:TestRequirement rdf:about="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<dc:description xml:lang="en">DTD for XHTML 1.0 Strict.</dc:description>
<dc:title xml:lang="en">XHTML 1.0 Strict Document Type Definition</dc:title>
</earl:TestRequirement>
<earl:TestResult rdf:ID="error1">
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Error -
Line 14 column 7: document type does not allow element <code>li</code>
here; missing one of <code>ul</code>, <code>ol</code> start-tag.</p></div>
</dc:description>
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
</earl:TestResult>
<earl:TestResult rdf:ID="error2">
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Error -
Line 15 column 6: end tag for <code>li</code> omitted, but OMITTAG NO
was specified.</p></div>
</dc:description>
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
</earl:TestResult>
<earl:TestResult rdf:ID="error3">
<dc:description rdf:parseType="Literal">
<div xml:lang="en" xmlns="http://www.w3.org/1999/xhtml"><p>Error -
Line 16 column 9: there is no attribute<code>alt</code>.</p></div>
</dc:description>
<earl:outcome rdf:resource="http://www.w3.org/ns/earl#fail"/>
</earl:TestResult>
</rdf:RDF>In this section we have reviewed all the necessary steps to
build our first EARL report. Despite its verbosity, the report above is rather simple.
As mentioned already several times, Example 3.12 is only one of
the multiple serializations that allow us to express the corresponding RDF graph (see Figure 2 for a graphical representation of the first
assertion).
The previous section introduced us to the core
components of an EARL report.
In this section we will review the core classes and properties of EARL, together with some clarifications
on its usage. The decomposition of every class is followed by the set of RDF properties
the class must or may have.
4.1 Assertion
The
Assertion class is used to declare an statement about the results of a
test. As shown in Section 2, an Assertion binds together the
four components of EARL: the assertor, the subject, the test requirement and its result
(see Example 3.11). It is therefore the fundamental unit of an
EARL statement or set of
statements.
An Assertion must have exactly one of
the following properties:
- earl:assertedBy
- Person, software, or combinations thereof that determined the result of the
test. There are several ways to express these, as it will be seen later on. In the
EARL jargon, these are
references to
Assertors or subclasses of it.
- earl:subject
- The thing that is being tested against some given requirements or test cases. In
the EARL jargon, these are
references to
TestSubjects (therefore, its subclasses
Software and Content can be also referenced here).
- earl:test
- The requirement that is used to test a subject. In the EARL jargon, these are references
to
TestCriterion or subclasses of it.
- earl:result
- Reference to the result of the test, i.e., whether the subject passes or fails
the test case (or there is some other result). In the EARL jargon, these are references
to
TestResults.
An Assertion may also include the following
optional property:
- earl:mode
- The mode in which the test was performed: as an automated computer process, by a
human making a subjective judgement, or otherwise. This can be a reference to a
TestMode instance. More details on these modes are given in the
description of the TestMode class.
Figure 3 presents a graph of the Assertion
Class.
4.2 Assertor
The
Assertor class are the persons, tools or combinations thereof, that
performs the test reported in the assertions of the reports. There are two classes of
Assertors:
- Single Assertor: the assertor is unique, and could be a person, an organization,
a software tool, or any type of agent that can perform a test (class
SingleAssertor).
- Compound Assertor: a set of two or more assertors, from which one of them is the
main assertor, and the rest perform the role or auxiliary assertors (class
CompoundAssertor).
4.2.1 Single
Assertor
A SingleAssertor is one single entity or agent responsible
for the report. A SingleAssertor can be one of the following types:
foaf:Agent- The
Assertor is an Agent, as defined in the FOAF specification [FOAF]. Any subclass of it can be a SingleAssertor,
however, it is recommended to be one of foaf:Person or
foaf:Organization. It is not recommended to use the subclass
foaf:Group. Use a compound assertor for such cases.
foaf:Person- The
Assertor is a human being. This uses the
FOAF vocabulary term foaf:Person to describe a
person [FOAF]. There should be identifying information
including a name, and a uniquely identifying property such as email address or an
encrypted email address. At least, the properties foaf:name,
foaf:mbox or foaf:mbox_sha1sum should be
used. It is strongly recommended to use the later for privacy considerations.Example 4.1.
Example of a FOAF Person description.
<rdf:RDF
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xml:base="http://www.example.org/earl/report#">
<foaf:Person rdf:ID="jd">
<foaf:homepage rdf:resource="http://www.example.org/jdoe"/>
<foaf:mbox_sha1sum>a94d730850a2b4b71239756a0f301e6462263e07</foaf:mbox_sha1sum>
<foaf:family_name>Doe</foaf:family_name>
<foaf:givenname>John</foaf:givenname>
<foaf:title>Dr.</foaf:title>
<foaf:name>John Doe</foaf:name>
</foaf:Person>
</rdf:RDF>
foaf:Organization- The
Assertor represents an organization [FOAF].
There should be identifying information for the organization, including a name, and
a uniquely identifying property such as a home page. The properties
foaf:name, foaf:homepage should be used
to provide this information.Example 4.2. Example of a FOAF Organization
description.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xml:base="http://www.example.org/earl/report#">
<foaf:Organization rdf:ID="exOrg">
<foaf:homepage rdf:resource="http://www.example.org/"/>
<foaf:name>Example Organization</foaf:name>
</foaf:Organization>
</rdf:RDF>
earl:Software- The
Assertor is a piece of Software, such as a black box testing
tool, an SGML validator or an accessibility evaluation tool. See Section 4.3.1 for further details. Notice that
earl:Software can be both an assertor or a test subject. Each use case
must be deduced from the context of the EARL report.Example 4.3. Example of an EARL
Software class.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:Software rdf:about="http://example.org/svgValidator">
<dct:hasVersion>1.0</dct:hasVersion>
<dc:title xml:lang="en">SVG Validator</dc:title>
<dc:description xml:lang="en">An SVG validator in
all its flavours.</dc:description>
</earl:Software>
</rdf:RDF>
4.2.2 Compound Assertor
In this case, the
Assertor is a group of entities (i.e., persons, organizations and/or
software tools). Each group
must have at least one primary assertor and
may have one or more secondary assertors, identified, respectively, by
the properties:
- earl:mainAssertor
- The main assertor, which shall be in turn a person or a tool (see
SingleAssertor).
- earl:helpAssertor
- The assistant actor(s) to the main assertor, which also shall be in turn a
person or a tool (see
SingleAssertor).
Additionally, there are an optional description and title, identified by the
Dublin Core dc:title and dc:description properties,
respectively.
If it is not possible to determine the hierarchy of the assertors
(for instance, a group of accessibility experts carrying out a test in a Web site), all
of them are considered main assertors. This also applies to aggregation of test results
from different sources.
4.3 Test Subject
The
TestSubject class represents a generic placeholder for things that are
being tested. For every intended usage, it is recommended to create a subclass with
determined properties. EARL provides two concrete use cases, Content and
Software, specified in the following subsections. The use of TestSubject
for these purposes should be avoided. The TestSubject class
may have instances of the following properties:
- dc:title
- A human readable title for the test subject.
- dc:description
- A more thorough description of the test subject intended as well for human
consumption.
- dc:date
- Date referring to the subject. This date may not be an intrinsic part of the
subject. If exists, it shall be the creation date of the subject. In its absence, it
can be the date in which the subject was retrieved. For more specific issues on Web
content, refer to the
Content class. Whenever possible, the type of
date used should be identified via the corresponding XML Schema datatype. This
property could be used for versioning purposes.[Editor's note: It is
not better dct:hasVersion for that?.]
- dct:hasPart
- Property used to detail other subjects that are part of the main subject. Again,
more specific issues related to Web content are specified later.
- dct:isPartOf
- Property used to express relationships to other parent subjects of which this
subject belongs to.
Example
4.4. Example of a Test Subject class.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:http-header="http://www.w3.org/2006/http-header#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:http="http://www.w3.org/2006/http#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:TestSubject rdf:about="http://example.org/index.html">
<dc:title xml:lang="en">Example.org Home Page</dc:title>
<dc:description xml:lang="en">This is our entry page, please
select your option.</dc:description>
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date"
>2003-01-20</dc:date>
<dct:hasPart>http://example.org/css/layout.css</dct:hasPart>
<dct:hasPart>http://example.org/css/main.css</dct:hasPart>
<dct:hasPart>http://example.org/img/logo.png</dct:hasPart>
</earl:TestSubject>
</rdf:RDF>4.3.1 Software
The
Software class is used to describe any piece of software (from a full
application to different API components). It is not restricted to any type of tool, and
it could be an authoring tool, a user agent, or a testing tool. It must be noticed that
Software could be used as Assertor or as
TestSubject. In those cases, it could inherit properties from these
classes. A Software class must have the property:
- dc:title
- A human readable title or name for the software element.
And may have the properties:
- dc:description
- A more thorough description of the software and its capabilities intended as
well for human consumption.
- dct:hasVersion
- A distinct identifier for the version of the software used. Traditionally, it
consists of a combination of letters and digits with some separation character.
- foaf:homepage
- A URL where information about the tool or its manufacturer can be found.
The Software class is often used as Assertor, but in
the case of being a TestSubject (for instance, by being the subject of an
Authoring Tools Accessibility Guidelines compliance check), it inherits also its
optional properties: dc:date, dct:isPartOf and
dct:hasPart.
Example 4.5. Example of a Software class, for an
Assertor use.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:Software rdf:about="http://example.org/cooltool/">
<dc:title>Example Cool Tool</dc:title>
<dc:description xml:lang="en">This is the coolest tool
in the world.</dc:description>
<foaf:homepage>http://example.org/cooltool/</foaf:homepage>
<dct:hasVersion>1.0.1.a</dct:hasVersion>
</earl:Software>
</rdf:RDF>4.3.2 Content
The Content
class is a subclass of TestSubject used to describe a given content that
has been tested. It is generally intended for Web resources retrieved via any protocol
(like HTTP(S), FTP, etc.), or residing locally in the user's computer (i.e., file:
protocol). It must be noted that for HTTP resources, it is recommended to use the HTTP
Vocabulary in RDF [HTTP-RDF] to describe the context of the
resource, including its URL. This will disambiguate resources because HTTP allows
content negotiation with the requesting user agent, thus different resources could be
retrieved under the same URL.
[Editor's note: There is an issue on
whether to use rdf:about to identify the URL of the resource, or whether this can be an
internal pointer. Definition of non-HTTP resources unclear.]
The
Content class can be reused for other purposes (outside EARL) as well. It
should have instances of at least one of the following
properties:
- earl:sourceCopy
- A copy of the actual source of the content that has been tested.
This could include Literal, XML, or Data URI types depending on the content.
[Editor's note: there is an issue with data URIs.]
- earl:context
- The context under which the content was fetched. As mentioned earlier, it is
recommended to use the HTTP Vocabulary in RDF [HTTP-RDF]
to describe Web content served by HTTP(S).
For example, if we make the file from Example 3.2
well formed:
Example 4.6. Example of an XHTML file to be expressed
as a Content class.
<html lang="en" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>Example of project pages</title>
</head>
<body>
<h1>Project description</h1>
<h2>My project name</h2>
<p>The strategic goal of this project is to make you
understand EARL.</p>
<ul>
<li>Here comes objective 1.</li>
<li>Here comes objective 2.</li>
</ul>
<p>And goodbye ...</p>
</body>
</html>we could express it like this:
Example 4.7. XHTML file be expressed
as a Content class.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:Content rdf:about="http://example.org/resource/index.html">
<earl:sourceCopy rdf:parseType="Literal">
<html lang="en" xmlns="http://www.w3.org/1999/xhtml" xml:lang="en">
<head>
<title>Example of project pages</title>
</head>
<body>
<h1>Project description</h1>
<h2>My project name</h2>
<p>The strategic goal of this project is to make you understand EARL.</p>
<ul>
<li>Here comes objective 1.</li>
<li>Here comes objective 2.</li>
</ul>
<p>And goodbye ...</p>
</body>
</html>
</earl:sourceCopy>
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date"
>2006-02-14</dc:date>
<dc:title xml:lang="en">Project Description</dc:title>
</earl:Content>
</rdf:RDF>Notice that the RDF type XML Literal shall be used with
caution with HTML documents, which not necessarily are well-formed XML documents. The
same example, can be expressed via the use of contexts.
Example 4.8. Expressing Web content
via contexts.
<rdf:RDF
xmlns:foaf="http://xmlns.com/foaf/0.1/"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:http="http://www.w3.org/2006/http#"
xmlns:dc="http://purl.org/dc/elements/1.1/"
xml:base="http://www.example.org/earl/report#">
<earl:Content rdf:about="http://example.org/resource/index.html">
<earl:context>
<http:Request rdf:ID="req01"/>
</earl:context>
<dc:date rdf:datatype="http://www.w3.org/2001/XMLSchema#date"
>2006-02-14</dc:date>
</earl:Content>
<http:Connection rdf:ID="conn01">
<http:request>
<http:Request rdf:about="#req01"/>
</http:request>
<http:connectionAuthority>example.org:80</http:connectionAuthority>
</http:Connection>
<http:Request rdf:about="#req01">
<http:abs_path>/resource/index.html</http:abs_path>
</http:Request>
</rdf:RDF>Note that the Request class is linked to a
Connection class, which defines relevant parameters of the HTTP connection
used. More complex examples are shown in Section 5.
In case
where the Content class is used as a TestSubject, it inherits
its optional properties (see Section 4.3). The usage of
dc:date follows this convention:
- When existing, it should be the last modified date of the resource;
- otherwise, if available, the creation date; or
- if none of the above is available, the retrieval date.
4.4 Test Criterion
The TestCriterion
class is used to declare testable statements. A requirement is said to be testable if it
is possible to define an objective test to determine whether the requirement has been
met (i.e., it has passed or failed). Testable criterion could be software unit test
cases, conformance tests, accessibility guidelines, markup validation criteria, etc.
While TestCriterion can be globally used, it is recommended to use one of its
subclasses, TestRequirement or TestCase (see Sections 4.4.1 or 4.4.2 on their applicability).
It is not possible within
EARL to define the level of granularity of the Test Criterion. For that objective, it
would be necessary to specify some kind of test case description language, or even a
test suite description language as well. However, within EARL, it is possible to define
relation properties via Dublin Core properties and terms. In general,
TestCriterion may have the following properties:
- dc:title
- Human readable title for the criterion.
- dc:description
- A longer description of some of the criterion(-a) and its(their) components,
targeted to human consumption.
- dct:hasPart
- To be used when the criterion has subcomponents.
- dct:isPartOf
- To be used when the criterion is part of a bigger set of components.
To facilitate interoperability and exchange of reports, it is recommended to
use publicly available URIs. We present here an example on how
such criterion could be presented for the Web
Content Accessibility Guidelines 1.0 (WCAG 1.0).
Example 4.9. Expressing test criteria
for the Web Content Accessibility Guidelines 1.0.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<earl:TestCriterion rdf:about="http://www.w3.org/TR/WCAG10/#">
<dc:title>Web Content Accessibility Guidelines 1.0</dc:title>
<dc:description>The guidelines discuss accessibility issues
and provide accessible design solutions.</dc:description>
</earl:TestCriterion>
<earl:TestCriterion
rdf:about="http://www.w3.org/TR/WCAG10/#tech-text-equivalent">
<dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG10/#" />
<dc:title>Checkpoint 1.1</dc:title>
<dc:description>Provide a text equivalent for every non-text
element.</dc:description>
</earl:TestCriterion>
<earl:TestCriterion
rdf:about="http://www.w3.org/TR/WCAG10/#tech-redundant-server-links">
<dct:isPartOf rdf:resource="http://www.w3.org/TR/WCAG10/#" />
<dc:title>Checkpoint 1.2</dc:title>
<dc:description>Provide redundant text links for each active region
of a server-side image map.</dc:description>
</earl:TestCriterion>
</rdf:RDF>Of course, the level of granularity presented in this
example is not adequate for the typical output of compliance tools, for which test
criteria respond to a lower level.
4.4.1 Test
Requirement
A TestRequirement class is a high level set that is
being used to define compliance. As mentioned earlier, the level of granularity is not
defined, and different inclusion relationships and hierarchies can be defined. Examples
of TestRequirements could be as described in Example
4.9, the Web Content Accessibility Guidelines 1.0 or any of its checkpoints.
4.4.2 Test Case
A TestCase class represents an atomic
test, i.e., tests that are self-contained and independent of each other on a given
subject or set of subjects.
An example of TestCases could be the
tests defined in the Unified Web Evaluation
Methodology (UWEM). There, every test is defined for different content types, like
(X)HTML, CSS or media elements. Example 4.10 presents test
Test 1.1_HTML_01 from UWEM in EARL.
Example 4.10. Test 1.1_HTML_01 from the Unified Web
Evaluation Methodology in EARL.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:dct="http://purl.org/dc/terms/"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:dc="http://purl.org/dc/elements/1.1/">
<earl:TestRequirement rdf:about="http://www.wabcluster.org/UWEM1.1/#">
<dc:title>Unified Web Evaluation Methodology 1.1</dc:title>
</earl:TestRequirement>
<earl:TestCase rdf:about="http://www.wabcluster.org/UWEM1.1/#1.1_HTML_01">
<dct:isPartOf rdf:resource="http://www.wabcluster.org/UWEM1.1/#"/>
<dc:title>UWEM Test 1.1_HTML_01</dc:title>
<dc:description>Test targeted to check that non-text content
has a text equivalent.</dc:description>
</earl:TestCase>
</rdf:RDF>4.5 Test Mode
TestMode
is an optional component of the Assertion which specifies
how the testing was carried out. Whether human experts, tools or combinations thereof
were used to carry out the test should be expressed via SingleAssertor or
CompoundAssertor. Its values are predefined, and must be
one of the following:
- earl:manual
- Where the test was carried out by a human. This includes the case where the
human was aided by instructions or guidance provided by a software tool, but where
the human has carried out the actual test procedure.
- earl:semiAutomatic
- Where the test was partially carried out by the software tool, but human input
or judgment was still required to decide or help decide the outcome of the
test.
- earl:automatic
- Where the test was carried out automatically by the software tool and without
any human intervention.
- earl:notAvailable
- Whenever the test mode cannot be made public for some given reason or is not
available.
- earl:inferred
- [Editor's note: this value will be defined or dropped later.]
For example, Example 3.11 could be expanded to
express the mode of the assertions, which is in this case automatic:
Example 4.11.
Test assertions containing mode information.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xml:base="http://www.example.org/earl/report#">
<earl:Assertion rdf:ID="ass1">
<earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
<earl:result rdf:resource="#error1"/>
<earl:test
rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
<earl:assertedBy rdf:resource="#assertor01"/>
</earl:Assertion>
<earl:Assertion rdf:ID="ass2">
<earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
<earl:result rdf:resource="#error2"/>
<earl:test
rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
<earl:assertedBy rdf:resource="#assertor01"/>
</earl:Assertion>
<earl:Assertion rdf:ID="ass3">
<earl:mode rdf:resource="http://www.w3.org/ns/earl#automatic"/>
<earl:result rdf:resource="#error3"/>
<earl:test
rdf:resource="http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"/>
<earl:subject rdf:resource="http://example.org/resource/index.html"/>
<earl:assertedBy rdf:resource="#assertor01"/>
</earl:Assertion>
</rdf:RDF>4.6 Test Result
A
TestResult class is used to express the result of a test. Its primary use
is to provide a machine-readable result that can be processed or aggregated later on. It
shall be complemented with human-readable descriptions of the errors or warnings
found.
A TestResult must have exactly one instance
of the property:
- earl:outcome
- A machine-readable
OutcomeValue value that describes the result
(see Section 4.6.1).
And it should also include instances of the following
properties:
- dc:title
- A short title describing the error message. This generally corresponds to the
error type issued by the compliance tool.
- dc:description
- A human-readable thorough description of the error. This can also include markup
via an XML Literal.
A TestResult may also include instances of the
following properties:
- dc:date
- The date on which the test was conducted. This date may differ from the
dc:date property of the test subject.
- earl:pointer
- This property identifies a location (or a set of locations) within
the
TestSubject where the error occurs. [Editor's note: the Pointer
Methods in RDF document is currently under development by the ERT WG. This document
will be synchronized with it, once available.]
- earl:info
- This property presents additional information that complement the error
description. For example, it can include warning messages or information on how to
correct the error. Whenever feasible, it is recommended to use XML Literal values
for such additional messages.
[Editor's note: there is overlapping between dc:description and
earl:info.]
4.6.1 Outcome Value
The
OutcomeValue class is used to represent a set of predefined values which
are the outcome of the test (earl:outcome property). These values
are:
- earl:pass
- The subject passed successfully the test.
- earl:fail
- The subject did not meet the requirements of the test.
- earl:cannotTell
- The assertor cannot decide what the outcome of the test is. This result could be
used for example, as an intermediate stage when an automatic test requires further
human assessment.
- earl:notApplicable
- In this case, the test criterion is not applicable to the subject.
- earl:notTested
- The test has not been carried out yet. This is useful for workflow control or
quality assurance procedures.
This section will present some more complex examples on how to deal with
the preparation and handling of EARL reports.
The
process of creating an EARL report is closely linked to the testing
or quality assurance process. In general, it is recommended to follow these
steps:
- Describe precisely your testing process. In particular, identify the relevant
tasks as well as the actors that intervene in the process. Actors include not only
the assertors that perform the tests (either human or testing tools), but also users
of the reports like management, development team, etc.
- Identify the testing requirements. For example, for a typical Intranet, it could
be decided that all resources must be valid, no broken links shall be present,
accessibility compliance to WCAG 1.0 Level A is required, and
text resources shall be spell-checked.
- Identify the subjects to be tested. This means that the different resource types
(e.g., (X)HTML or other markup documents, style sheets, images, media documents,
applets, etc.) must be mapped to the different testing requirements.
- Define the monitoring process. Resources are not tested once, but several times.
To that end, a realistic timeline must be defined, in which testing cycles will fit.
Closely linked to this issue is the persistence and versioning of resources, so that
comparisons and monitoring techniques can be implemented.
- Finally, define your reporting criteria, which must be tailored to your
different audiences (see point 1).
From the above steps, it can be seen that EARL supports the testing process in
all its steps. However, it can be seen that the final assertions are not only referred
to atomic tests, but different aggregation possibilities can be implemented, adapted to
the different audiences and monitoring processes.
Let us see an example composed
of two reports. The test subject is an XHTML document with two nested tables. The first
top level table is a table used for content layout, whilst the second one contains real
tabular information. The first assertor is an accessibility compliance tool which is
able of issuing a warning for the first table, which needs human verification, and can
confirm that the second table is correctly used. The second assertor is an accessibility
expert, which can confirm the assertions of the tool. Both reports can be merged as
follows.
[Editor's note: Include here the reports.]
A note of
caution must be included here about security and privacy. EARL
reports can contain sensitive information, especially in regard to assertors and test
subjects. That can include for instance, usernames, passwords, or other sensitive
information embedded in the body of the testing subject. It is recommended to plan these
issues carefully, encrypt information whenever necessary and be aware of the target
audiences of any report.
Test subjects in EARL can become
complex whenever dealing with resources retrieved via the HTTP(S)
protocol. In many cases, a one-to-one mapping of a resource to a URI
is not possible due to different issues:
- Content negotiation, in which a Web server can send different resources when
reacting to different headers of an HTTP(S) request: user agent,
accepted language, etc.
- POST requests, in which different input parameters lead to different resources
or responses from the server.
- AJAX applications in which a mixture of Javascript and Web
Services can modify completely Web applications.
For such cases, it is necessary to use the HTTP Vocabulary
in RDF [HTTP-RDF]. Our first example is a POST request to the
URL http://www.example.org:80/webapp, with the following
pair of input parameters:
- Parameter
foo with value blah1, and
- Parameter
bar with value blah2.
Additionally, we use the HTTP header
User-Agent with the value 'My User Agent'. The TestSubject can
be then written as:
Example 5.5. Test subject for an
HTTP POST request.
<rdf:RDF
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:http-header="http://www.w3.org/2006/http-header#"
xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:http="http://www.w3.org/2006/http#"
xml:base="http://www.example.org/earl/report#">
<http:Connection rdf:ID="conn01">
<http:request>
<http:PostRequest rdf:ID="req01"/>
</http:request>
<http:connectionAuthority>www.example.org:80</http:connectionAuthority>
</http:Connection>
<http:PostRequest rdf:about="#req01">
<http:header rdf:parseType="Collection">
<http:MessageHeader rdf:ID="msg01"/>
<http:MessageHeader rdf:ID="msg02"/>
</http:header>
<http:version>1.1</http:version>
<http:body
rdf:resource="data:application/x-www-form-urlencoded;base64,Zm9vPWJsYWgxJmJhcj1ibGFoMg=="/>
<http:response>
<http:Response rdf:ID="resp01"/>
</http:response>
<http:abs_path>/webapp</http:abs_path>
</http:PostRequest>
<http:MessageHeader rdf:about="#msg01">
<http:fieldValue>My User Agent</http:fieldValue>
<http:fieldName rdf:resource="http://www.w3.org/2006/http-header#user-agent"/>
</http:MessageHeader>
<http:MessageHeader rdf:about="#msg02">
<http:fieldValue>www.example.org</http:fieldValue>
<http:fieldName rdf:resource="http://www.w3.org/2006/http-header#host"/>
</http:MessageHeader>
</rdf:RDF>Note that the string "Zm9vPWJsYWgxJmJhcj1ibGFoMg=="
in the data URI is the result of Base64-encoding the character
sequence "foo=blah1&bar=blah2".
[Editor's note:
Explain this example. Correct namespaces.]
<rdf:RDF xmlns:rdf="http://www.w3.org/1999/02/22-rdf-syntax-ns#"
xmlns:earl="http://www.w3.org/ns/earl#"
xmlns:http="http://www.w3.org/2006/http#"
xml:base="http://example.org/20070401/test#">
<!-- Main connection -->
<http:Connection rdf:ID="conn">
<http:connectionAuthority>
example.org:80
</http:connectionAuthority>
<http:request rdf:parseType="Collection">
<http:Request rdf:about="#request1" />
<http:Request rdf:about="#request2" />
<http:Request rdf:about="#request3" />
</http:request>
</http:Connection>
<!-- Requests -->
<http:GetRequest rdf:ID="request1">
<http:abs_path>/index.html</http:abs_path>
</http:GetRequest>
<http:GetRequest rdf:ID="request2">
<http:abs_path>/search.html</http:abs_path>
</http:GetRequest>
<http:GetRequest rdf:ID="request3">
<http:abs_path>/sitemap.html</http:abs_path>
<http:version>1.1</http:version>
<http:header rdf:parseType="Collection">
<http:MessageHeader>
<http:fieldName
rdf:resource="http://www.w3.org/2006/http-header#accept" />
<http:fieldValue rdf:parseType="Collection">
<http:HeaderElement>
<http:elementName>text/html</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>1.0</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
<http:HeaderElement>
<http:elementName>
application/xml
</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>0.9</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
<http:HeaderElement>
<http:elementName>*/*</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>0.01</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
</http:fieldValue>
</http:MessageHeader>
<http:MessageHeader>
<http:fieldName
rdf:resource="http://www.w3.org/2006/http-header#accept-language" />
<http:fieldValue rdf:parseType="Collection">
<http:HeaderElement>
<http:elementName>de-DE</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>1.0</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
<http:HeaderElement>
<http:elementName>de</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>0.75</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
<http:HeaderElement>
<http:elementName>en</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>0.25</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
<http:HeaderElement>
<http:elementName>*</http:elementName>
<http:param>
<http:Param>
<http:paramName>q</http:paramName>
<http:paramValue>0.01</http:paramValue>
</http:Param>
</http:param>
</http:HeaderElement>
</http:fieldValue>
</http:MessageHeader>
</http:header>
</http:GetRequest>
</rdf:RDF>[Editor's
note: To be added.]
This guide presented a thorough overview of
the Evaluation and Report Language (EARL). As mentioned in the
introduction, EARL must be seen as a generic framework that can
facilitate the creation and exchange of test reports. In this generality lies its
strength, as it can be applied to multiple scenarios and use cases, which may even lay
outside the world of software development and compliance testing.
The
EARL framework allow as well merging and aggregation of results in a
semantic manner, thus enabling different testing actors to share and improve
results.
Of course, there could be scenarios where EARL might
not be able to cope with their underlying complexity. However, its semantic nature
allows its extensibility via proprietary vocabularies based upon RDF,
without endangering the interoperability of the reports.
The Working Group is
looking forward to receiving feedback on the current version of the schema, and expects
from implementers of compliance tools issues and suggestions for
improvement.
- [DC]
- The Dublin Core Metadata
Element Set - DC Recommendation, 20 December
2004.
http://www.dublincore.org/documents/dces/
- [DCT]
- The Dublin Core
Metadata Terms - DC Recommendation, 13 June
2005.
http://www.dublincore.org/documents/dcmi-terms/
- [FOAF]
- FOAF Vocabulary Specification - Working
Draft, 3 June 2005.
http://xmlns.com/foaf/0.1/
- [HTTP-RDF]
- HTTP in RDF
- [IEEE-829]
- IEEE Standard
for Software Test Documentation (IEEE Std 829-1998). ISBN 0-7381-1444-8 SS94687.
Available at: http://ieeexplore.ieee.org/servlet/opac?punumber=5976
- [RDF]
- Resource
Description Framework (RDF) Model and Syntax Specification - W3C Recommendation,
22 February
1999.
http://www.w3.org/TR/1999/REC-rdf-syntax-19990222/
- [RDF-PRIMER]
- RDF Primer - W3C
Recommendation, 10 February
2004.
http://www.w3.org/TR/rdf-primer/
- [RDFS]
- RDF Vocabulary Description Language
1.0: RDF Schema - W3C
Recommendation, 10 February
2004.
http://www.w3.org/TR/rdf-schema/
- [RDF-XML]
- RDF/XML Syntax Specification
(Revised) - W3C Recommendation 10 February
2004.
http://www.w3.org/TR/rdf-syntax-grammar/
- [RDF-XML-DIFFS]
- Why RDF model is different from the XML
model - Paper by Tim Berners-Lee, September
1998.
http://www.w3.org/DesignIssues/RDF-XML
- [RFC2119]
- Key words for use in RFCs to
Indicate Requirement Levels - IETF RFC, March
1997.
http://www.ietf.org/rfc/rfc2119.txt
- [OWL]
- OWL Web Ontology Language - W3C
Recommendation, 10 February
2004.
http://www.w3.org/TR/owl-features/
- [WCAG10]
- Web Content Accessibility Guidelines
1.0 - W3C Recommendation, 5 May
1999.
http://www.w3.org/TR/WCAG10/
- [XML]
- [URI]
- [XPath]
- XML Path Language (XPath)
Version 1.0 – James Clark, Steve DeRose, W3C Recommendation, 16 November
1999
- To be completed.
[Editor's note: To
be determined.]
