[ contents ]
Copyright © 2012 W3C ® ( MIT , ERCIM , Keio ), All Rights Reserved. W3C liability , trademark and document use rules apply.
This document specifies an internationally harmonized methodology for evaluating the accessibility conformance of existing websites to the Web Content Accessibility Guidelines ( WCAG ) 2.0 . It defines an approach for conformance evaluation of entire websites as opposed to page-by-page evaluation that is already defined by WCAG 2.0. It addresses conformance evaluation in different contexts, including self-assessment and third-party evaluation. It is applicable to any website, including web applications and websites for mobile devices, and is independent of any particular evaluation tools, web browsers, and assistive technologies.
This document is a supporting resource for WCAG 2.0 and does not replace or supersede it in any way. It addresses the need for a standardized approach for evaluating entire websites after their development as opposed to evaluating individual collections of web pages during their development, which is equally important to ensure website accessibility. This work is part of W3C / WAI activities on Web Accessibility Evaluation and Testing that address different W3C / WAI guidelines and specifications .
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This
30
July
27
August
2012
Editors
Draft
of
Website
Accessibility
Conformance
Evaluation
Methodology
1.0
is
a
complete
draft
based
on
the
recently
same
approach
and
expanding
the
previously
published
W3C
First
Public
Working
Draft
located
at
http://www.w3.org/TR/2012/WD-WCAG-EM-20120327/
of
27
March
2012
.
The
This
Editors
Draft
addresses
the
following
comments
received,
to
seek
approval
for
publication
as
an
updated
Working
Draft:
This
document
is
intended
to
be
published
and
maintained
as
a
an
informative
W3C
Working
Group
Note
after
review
and
refinement.
It
includes
changes
following
the
resolutions
from
the
comments
we
received
on
the
Public
Working
Draft.
[@@@]
The
WCAG
2.0
Evaluation
Methodology
Task
Force
(
Eval
TF
)
invites
discussion
and
feedback
on
this
document
by
developers,
evaluators,
researchers,
and
others
with
interest
in
web
accessibility
evaluation.
Eval
TF
is
particularly
looking
for
feedback
on:
Please
send
comments
on
this
Editor
Editors
Draft
of
the
Website
Accessibility
Evaluation
Methodology
for
WCAG
2.0
to
public-wai-evaltf@w3.org
(publicly
visible
mailing
list
archive
).
These
comments
will
be
considered
in
the
internal
discussions
of
Eval
TF
.
Publication
as
Editor
Editors
Draft
does
not
imply
endorsement
by
the
W3C
Membership.
This
is
a
draft
document
and
may
be
updated,
replaced
or
obsoleted
by
other
documents
at
any
time.
It
is
inappropriate
to
cite
this
document
as
other
than
work
in
progress.
This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force ( Eval TF , a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG) , as part of the Web Accessibility Initiative (WAI) Technical Activity .
This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy . The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy .
Website
owners,
procurers,
suppliers,
developers,
and
others
are
frequently
tasked
with
assessing
the
conformance
of
existing
websites
to
the
Web
Content
Accessibility
Guidelines
(
WCAG
)
2.0
.
This
document
provides
a
standardized
methodology
for
evaluating
the
conformance
of
entire
websites
to
WCAG
2.0.
2.0,
as
opposed
to
individual
web
pages
.
It
is
developed
through
the
collaborative
W3C
Process
with
the
involvement
of
international
stakeholders.
It
is
a
supporting
document
for
WCAG
2.0
and
does
not
replace
or
supersede
it
in
any
way.
To ensure accessibility, it needs to be considered and addressed throughout the development process. Accessibility testing and evaluating conformance to WCAG 2.0 needs to be carried out during all stages of the process, including the design, implementation, and maintenance stages of a website .
The
guidance
in
this
document
focuses
on
only
one
aspect
of
evaluation
within
the
development
process,
which
is
the
accessibility
evaluation
of
an
already
existing
website
websites
.
This
is
useful
for
determining
the
level
of
accessibility
of
a
website
at
a
given
point
in
time,
such
as
at
the
time
of
releasing
or
purchasing
a
product.
It
is
also
useful
for
initial
conformance
assessments
when
getting
started
with
accessibility
and
for
continual
monitoring
of
website
accessibility.
The methodology described in this document supports conformance evaluation in different contexts including self-assessment and third-party assessment of websites . It is applicable to all websites , including web applications, websites intended to be used with mobile devices, and other types of websites . It is independent of the size of the website , the web technologies used to create the website , and of any particular web accessibility evaluation tools, web browsers, assistive technologies, and other software tools.
The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:
Other audiences for whom this methodology is also relevant include:
Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2.2 Required Expertise .
It is assumed that the reader of this document is familiar with the following W3C resources:
A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:
Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:
The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairments, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:
For the purposes of this document, the following terms and definitions apply:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
The content would not conform if that technology is turned off or is not supported
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility . In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website .
[
Editor
Note:
This
section
has
been
rewritten
and
needs
to
be
reviewed
by
Eval
TF.]
[
Review
Note:
Eval
TF
is
particularly
looking
for
feedback
on
this
section.]
The methodology defined by this document applies to full, self-enclosed websites . This includes websites of organizations and entities, persons, events, products, and services. It also includes web applications and mobile websites . Examples include:
A website may include several smaller sub-sites, such as an online shop, an area for each department within the organization, a blog area, and other areas that may each be considered to be a website . This methodology can be applied to make conformance claims for such individual sub-sites, in which case these sub-sites would be considered to be a website each, and to the main website in its entirety. However, this methodology may not be applied to a website excluding any of its parts.
Example
of
a
Website:
In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medicine Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.
In the example above, none of the depicted parts may be excluded from the scope of evaluation if a conformance claim is to be made for the university website according this methodology. This includes aggregated and embedded content from third-party sources such as online maps for the university campus and forms for credit card transactions. Excluding parts of the website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements full pages and complete processes .
Note:
According
to
WCAG
2.0
it
is
possible
to
make
a
defines
"
Statement
of
Partial
Conformance
"
for
individual
web
pages
that
are
known
not
to
conform
with
WCAG
2.0
due
to
third-party
content
and
and/or
languages
lacking
accessibility
support
.
However,
these
parts
must
still
Such
web
pages
may
not
be
included
in
excluded
from
the
scope
of
evaluation.
More
information
evaluation
according
to
this
methodology.
In
some
cases
this
means
that
the
website
as
a
whole
does
not
conform
with
WCAG
2.0
due
to
only
partially
conforming
web
pages
.
Section
3.5
Step
5:
Report
the
Evaluation
Findings
provides
more
guidance
on
reporting
evaluation
results
and
making
conformance
claims
is
provided
in
section
3.5.2
Step
5.b:
Provide
an
Accessibility
Statement
(Optional)
accessibility
statements
for
entire
websites
.
This methodology is applicable to all websites , including web applications, websites intended to be used with mobile devices, and other types of websites . Some particular types of websites include:
Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading .
The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools .
Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.
The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites . Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility .
The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation .
This section defines the stages and activities of the evaluation procedure. The stages are not necessarily sequential but rather iterative. Also the exact sequence of the activities carried out during the evaluation process depends on the type of website , the purpose of the evaluation, and the process used by the evaluator . Some of the activities overlap or be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:
Note: See also section 4. Considerations for Particular Situations that may influence how an evaluation procedure is carried out.
Requirement 1: The evaluation scope shall be defined according to Requirement 1.a , Requirement 1.b , Requirement 1.c , Requirement 1.d , and optionally Requirement 1.e .
During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner ) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see 3.2: Explore the Target Website ).
Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner ) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.
Requirement 1.a: The scope of the website shall be defined, with the definition not contradicting the constraints expressed in section 2.1 Scope of Applicability .
During this step the exact scope of the website (the web pages for which the evaluation will apply) is defined and documented. This scope definition may not contradict the terms established in section 2.1 Scope of Applicability . It is essential to carefully document particular aspects such as any use of third-party content and services, different languages, and parts of the website that may not be easily identifiable as such (for example an online shop that is not integrated into the website but considered to be part of it).
The outcome of this step is an unambiguous definition for the target website , so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of URIs that define the web pages that are within the scope of evaluation (part of the target website ).
Requirement 1.b: The goal of the evaluation shall be defined as " Basic Report ", " Detailed Report ", or " In-Depth Analysis ".
Defining the goal of the evaluation is particularly relevant to the auditing stage defined in 3.4 Step 4: Audit the Selected Sample and the reporting stage defined in 3.5 Step 5: Report the Evaluation Findings . Some of the evaluation goals include:
Requirement 1.c: The target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate for shall be defined.
Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.
Requirement
1.d:
The
target
users
of
the
website
and
minimum
set
of
web
browsers
and
assistive
technology
to
evaluate
for
shall
be
defined,
noting
that
the
this
definition
of
software
support
shall
not
conflict
with
the
WCAG
2.0
guidance
on
the
Level
of
Assistive
Technology
Support
Needed
for
"Accessibility
Support"
.
It is often not feasible for websites to support accessibility on every combination of web browser, assistive technology, and operating system that they run on, nor is it possible to test with every such combination of tools. It is necessary to determine the primary context in which a website is used and the minimum set of web browsers and assistive technology that must be supported by the website .
For example, a website may be public, only available within a closed network (intranet), or limited to authenticated users (extranet). Users of intranet and extranet websites may be known to have a certain level of training on using the website and may be using pre-determined web browsers and assistive technology. This is usually not the case for public websites .
This definition of target users and tools must meet the terms defined in WCAG 2.0 Level of Assistive Technology Support Needed for "Accessibility Support" and must be used throughout the evaluation. For example, it is not possible to evaluate some pages with one set of tools and other pages with another set. Accessibility support must be uniform throughout a single website .
Note: It is also not possible to single out or exclude individual groups of users, such as "people with visual impairments", in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.
[
Editor
Note:
This
section
has
been
edited
changed
significantly
from
30
July
2012
Editor
Draft
and
needs
to
be
reviewed
by
Eval
TF.]
review.]
Requirement
1.e:
The
WCAG
2.0
techniques
that
will
be
used
to
evaluate
the
conformance
to
WCAG
2.0
Success
Criteria
should
be
specified.
Techniques
are
an
effective
way
in
the
context
of
demonstrating
whether
particular
WCAG
2.0
Success
Criteria
are
met
or
informative
and
not
met.
However,
required
for
satisfying
the
WCAG
2.0
conformance
to
requirements
;
WCAG
2.0
Success
Criteria
are
written
as
testable
statements.
Techniques
provide
documented
ways
of
meeting
WCAG
2.0
applies
to
its
Success
Criteria
and
conformance
requirements
,
and
failures
in
meeting
them.
More
information
on
techniques
is
provided
in
WCAG
2.0
Layers
of
Guidance
.
W3C
/
WAI
provides
a
set
of
publicly
documented
Techniques
for
WCAG
2.0
.
However,
it
is
not
necessary
to
the
use
these
particular
techniques.
In
fact,
in
some
situations,
such
as
in
a
closed
network,
it
may
be
necessary
to
use
techniques
that
are
specifically
developed
for
such
situations.
Individuals
and
organizations
developing
techniques
must
employ
methods
for
establishing
the
technique's
ability
to
meet
the
WCAG
2.0
Success
Criteria.
It
is
good
practice
to
specify
the
sets
or
sources
of
techniques
that
are
intended
to
be
used
during
the
evaluation
at
this
stage
already,
already
to
ensure
consistent
expectation
between
the
evaluator
and
the
evaluation
commissioner
.
This
definition
is
typically
refined
in
later
stages
of
the
evaluation
process,
for
example
to
evaluate
web
content
using
particular
web
technologies
during
the
website
exploration
and
accessibility
features.
Note:
W3C
/
WAI
provides
a
set
of
publicly
documented
Techniques
for
evaluation
stages.
Section
3.4.2
Step
4.b:
Use
WCAG
2.0
Techniques
Where
Possible
(Optional)
but
other
provides
more
guidance
on
using
WCAG
2.0
techniques
may
be
used
too.
during
evaluation.
Requirement 2: The website to be evaluated shall be explored according to Requirement 2.a , Requirement 2.b , Requirement 2.c , and Requirement 2.d .
During
this
step
the
evaluator
explores
the
target
website
to
be
evaluated,
to
develop
a
better
understanding
of
the
website
use
use,
purpose,
and
functionality.
Carrying
out
initial
cursory
checks
during
this
stage
already
helps
identify
web
pages
that
are
relevant
for
more
detailed
evaluation
later
on.
For
example,
an
evaluator
may
identify
web
pages
that
seem
to
be
lacking
color
contrast,
consistent
navigation,
or
document
structures
(headings,
links,
etc.)
with
simple
checks
while
studying
the
website
,
and
note
them
down
as
candidates
for
selection
in
the
sampling
step
defined
in
3.3
Step
3:
Select
a
Representative
Sample
.
more
detailed
evaluation
later
on.
To carry out this step it is critical that the evaluator has access to all the relevant parts of the website . For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.
Note:
Involvement
of
the
website
owner
and/or
website
developer
can
be
helpful
to
help
identify
key
common
web
pages
,
functionality,
technologies,
and
other
aspects
of
the
implementation
that
makes
the
evaluation
procedure
more
efficient
and
effective.
However,
the
evaluator
is
responsible
for
an
objective
and
thorough
assessment.
[
Editor
Note:
This
paragraph
has
been
added
and
needs
approval.]
Requirement
2.a:
The
common
web
pages
of
the
website
and
templates
available
to
the
evaluator
shall
be
identified.
During
this
step
the
common
web
pages
of
the
website
and
the
templates
that
are
available
to
the
evaluator
and
used
to
create
the
website
are
identified
and
documented.
The
outcome
of
this
step
is
a
list
of
all
common
web
pages
and
templates
available
to
,
including
the
evaluator
common
states
of
these
web
pages
and
used
to
create
the
website
for
web
applications
.
These
will
be
part
of
the
sample
to
be
selected
through
the
steps
defined
in
3.3
Step
3:
Select
a
Representative
Sample
.
This step also helps understand key aspects of the website , such as the navigation and overall structure of the website .
Requirement
2.b:
The
key
functionalities
common
functionality
of
the
website
shall
be
identified.
During
this
step
the
key
functionalities
common
functionality
of
the
website
are
identified
and
documented,
to
provide
a
comprehensive
basis
for
the
sample
to
be
selected
through
the
steps
defined
in
3.3
Step
3:
Select
a
Representative
Sample
.
The
outcome
of
this
step
is
a
list
of
user
tasks
such
as
"selecting
and
purchasing
a
product
from
the
shop
area
of
the
website",
"filling
and
submitting
the
form
provided
on
the
website",
and
"registering
for
an
account
on
the
website".
Requirement 2.c: The variety of web page types shall be identified.
Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the user. The outcome of this step is a list of the different types of web pages on the website , to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample . Different web page types include:
Note:
This
step
is
intended
to
identify
instances
groups
of
web
pages
,
page
instances,
including
in
web
applications
,
rather
than
templates.
Templates
are
identified
as
part
of
3.2.1
Step
1.a:
Identify
Key
Web
Pages
of
the
Website
.
Requirement 2.d: The technologies relied upon to provide the website shall be identified.
During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in 3.3 Step 3: Select a Representative Sample . This includes base web technologies such as HTML and CSS , auxiliary web technologies such as Flash, Java, JavaScript, PDF , Silverlight and WAI-ARIA , as well as advanced web technologies such as SMIL and SVG . The outcome of this step is a list of web technologies that are relied upon .
Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference .
Note:
Where
possible,
it
is
often
also
useful
to
identify
the
libraries
and
components
used
to
create
the
website
,
such
as
Dojo,
JQuery,
and
others.
Particularly
for
web
applications,
much
of
the
accessibility
support
is
built
into
these
libraries
and
components,
and
evaluation
can
become
more
effective
and
efficient
when
these
are
identified.
[
Editor
Note:
This
paragraph
has
been
added
and
needs
approval.]
Note:
Only
the
technologies
that
are
relied
upon
to
provide
the
website
need
to
be
identified
for
evaluation.
This
relates
closely
to
the
WCAG
2.0
concepts
of
conforming
alternate
versions
and
non-interference
.
Requirement 3: A representative sample of web pages shall be selected from the website according to Requirement 3.a , Requirement 3.b , Requirement 3.c , and Requirement 3.d .
While ideally every web page of a website is evaluated, usually this is not possible on most websites . In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.
Exploration
of
the
target
website
in
3.2
Step
2:
Explore
the
Target
Website
(within
the
scope
set
in
3.1
Step
1:
Define
the
Evaluation
Scope
)
provides
sufficient
understanding
of
the
website
to
facilitate
selection
of
a
sample
of
web
pages
that
is
representative
of
the
entire
target
website
.
Depending
on
the
size
and
complexity
of
the
website
,
the
size
of
this
sample
will
vary.
For
example,
a
website
with
few
types
of
web
pages
that
are
all
generated
from
a
confined
set
of
templates,
templates
,
such
as
a
data
repository,
will
likely
require
a
smaller
sample
than
a
website
with
many
types
of
web
pages
that
are
authored
using
different
mechanisms.
Also
a
web
application
could
have
a
limited
number
of
web
pages
that
dynamically
generate
content
with
varying
types
of
appearance,
behavior,
and
information
that
each
need
to
be
sampled.
The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF , Silverlight, and WAI-ARIA , and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website .
Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample .
Requirement
3.a:
All
common
web
pages
and
templates
available
to
the
evaluator
shall
be
part
of
the
selected
sample
of
web
pages
.
All
common
web
pages
and
templates
that
are
available
to
,
including
the
evaluator
common
states
of
these
web
pages
and
used
for
creating
the
website
web
applications
,
must
be
part
of
the
selected
sample.
These
web
pages
are
identified
in
step
3.2.1
Step
2.a:
Identify
Key
Common
Web
Pages
of
the
Website
.
Requirement
3.b:
At
least
two
distinct
web
pages
(where
applicable)
applicable
and
available)
of
each
(1)
key
common
functionality
,
(2)
content,
design,
and
functionality,
and
(3)
web
technologies
shall
be
part
of
the
selected
sample
of
web
pages
.
From the variety and types of web pages identified in 3.2 Step 2: Explore the Target Website (within the scope of the evaluation as defined per 3.1 Step 1: Define the Evaluation Scope ), select at least two distinct web pages with the following features each (where applicable and available):
Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least two distinct web pages for each relevant feature identified on the website , though more web pages may be necessary depending on the complexity of the website .
Requirement 3.c: Other web pages relevant for people with disabilities and accessibility shall be part of the selected sample of web pages .
Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:
Requirement 3.d: All web pages that are part of a complete process shall be included.
The selected sample must include all web pages that belong to a series of web pages presenting a complete process . Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.
Requirement 4: The selected sample of web pages shall be audited according to Requirement 4.a , Requirement 4.b , Requirement 4.c , and optionally, Requirement 4.d and Requirement 4.e .
WCAG 2.0 defines five conformance requirements that need to be met for each web page in the sample selected per 3.3 Step 3: Select a Representative Sample . This includes checking whether each WCAG 2.0 Success Criterion in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) has been met or not met for each of these web pages .
Note:
Many
web
pages
will
have
repetitive
content,
such
as
the
header,
navigation,
and
other
common
components
that
may
not
need
to
be
re-evaluated
on
each
occurrence.
Depending
on
the
level
of
detail
for
reporting
defined
by
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
,
an
evaluator
may
not
need
to
continue
to
identify
successes
and
failures
in
meeting
the
conformance
target
for
each
these
repetitive
elements
on
every
web
page
.
Section
3.5
Step
5:
Report
the
Evaluation
Findings
provides
more
guidance
on
reporting.
Requirement 4.a: Each web page in the sample selected per 3.3 Step 3: Select a Representative Sample shall be checked for meeting each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) and each of the WCAG 2.0 conformance requirements .
For
each
web
page
in
the
sample
selected
per
3.3
Step
3:
Select
a
Representative
Sample
,
check
whether
each
of
the
WCAG
2.0
Success
Criteria
in
the
target
conformance
level
(per
3.1.3
Step
1.c:
Define
the
Conformance
Target
)
and
each
of
the
WCAG
2.0
conformance
requirements
have
been
met.
This
includes
that
all
accessibility
features,
especially
key
common
functionality
defined
per
3.2.2
Step
2.b:
Identify
Key
Functionalities
Common
Functionality
of
the
Website
,
are
usable
for
the
audiences
and
with
the
tools
defined
per
3.1.4
Step
4.d:
Define
the
Context
of
Website
Use
.
Note:
According
to
WCAG
2.0,
Success
Criteria
that
do
not
apply
to
the
which
there
is
no
matching
content
are
deemed
to
have
been
met.
satisfied
.
An
outcome
such
as
'Not
Applicable'
may
be
used
to
denote
the
particular
situation
where
Success
Criteria
were
satisfied
because
no
relevant
content
was
applicable.
To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.
This assessment also needs to be complemented with focused testing of particular situations including:
Note:
Templates
are
usually
used
to
create
many
web
pages
,
sometimes
entire
parts
of
a
website
.
While
evaluating
templates
is
optional
in
this
methodology,
in
some
contexts
it
can
be
helpful
to
check
templates
on
their
own.
Evaluating
the
templates
that
are
identified
per
3.3.1
Step
3.a:
Include
Key
Web
Pages
of
the
Website
may
identify
potential
issues
that
may
not
be
easily
identified
through
evaluating
individual
instances
of
web
pages
.
However,
issues
identified
in
templates
alone
do
not
necessarily
imply
that
these
issues
occur
on
the
website
and
need
to
be
validated
on
individual
instances
of
web
pages
.
Also,
identifying
no
issues
in
templates
does
not
necessarily
imply
that
no
issues
occur
on
on
individual
instances
of
web
pages
.
[
Editor
Note:
Need
agreement
on
this
note.]
[
Editor
Note:
This
section
has
been
completed
changed
significantly
from
30
July
2012
Editor
Draft
and
needs
to
be
approved
by
Eval
TF.]
review.]
Requirement 4.b: Where possible, applicable WCAG 2.0 techniques shall be used to demonstrate successes and failures in meeting the WCAG 2.0 Success Criteria relevant per 3.1.3 Step 1.c: Define the Conformance Target .
While
The
initial
sets
or
sources
of
WCAG
2.0
techniques
are
only
informative
to
be
used
during
evaluation
may
be
defined
in
section
3.1.5
Step
1.e:
Define
the
context
of
Techniques
to
be
Used
(Optional)
.
However,
during
evaluation
such
an
initial
set
may
often
need
to
be
refined
according
to
the
particular
situation,
such
as
for
evaluating
particular
web
technologies
and
accessibility
features.
WCAG
2.0,
they
provide
an
effective
way
2.0
defines
three
types
of
demonstrating
whether
techniques:
For
each
web
page
are
met,
it
can
be
assumed
that
a
WCAG
2.0
Success
Criterion
is:
In
some
situations,
for
example
WCAG
2.0
techniques
are
not
exhaustive
as
they
cannot
cover
every
possible
situation.
Also,
the
techniques
used
to
meet
WCAG
2.0
Success
Criteria
during
the
development
may
not
be
known
to
the
evaluator.
Particularly
for
newly
released
web
technologies
or
when
these
web
technologies
are
used
in
particular
contexts,
contexts
there
may
be
no
public
publicly
or
proprietary
documented
techniques
available
to
the
evaluator
.
In
this
case
the
The
evaluator
must
determine
whether
the
WCAG
2.0
Success
Criteria
are
met
without
the
use
be
considerate
of
techniques.
Otherwise
it
is
good
practice
(for
efficacy
and
justifiability)
to
use
existing
techniques
to
demonstrate
successes
and
failures
in
meeting
these
limitations
when
using
WCAG
2.0
Success
Criteria.
techniques
Note:
The
initial
sets
or
sources
of
Advisory
techniques
are
determined
per
3.1.5
Step
1.e:
Define
the
Techniques
to
be
Used
.
However,
these
may
need
to
not
be
extended
depending
on
fully
supported
by
assistive
technology.
If
they
are
used,
make
sure
that
these
work
with
the
findings
made
per
3.2
web
browsers
and
assistive
technology
defined
in
3.1.4
Step
2:
Explore
1.d:
Define
the
Target
Context
of
Website
Use
.
[
Editor
Note:
This
section
has
been
completed
changed
significantly
from
30
July
2012
Editor
Draft
and
needs
to
be
approved
by
Eval
TF.]
review.]
Requirement
4.c:
Each
use
of
the
web
technologies
used
to
create
Accessibility
features
provided
on
the
website
shall
be
checked
to
be
accessibility
supported
by
the
tools
defined
in
Step
1.d:
Define
the
Context
of
Website
Use
.
To
ensure
that
the
accessibility
features
provided
on
a
website
such
as
text-alternatives,
captions,
keyboard
access
are
actually
usable
in
practice,
each
use
of
the
web
technologies
used
to
create
the
website
these
accessibility
features
must
be
accessibility
supported
by
the
tools
defined
in
Step
1.d:
Define
the
Context
of
Website
Use
.
In some situations WCAG 2.0 techniques and repositories on accessibility support provide insights on the level of accessibility support for accessibility features in particular combinations of web technologies and tools. However, the evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.
Requirement 4.d: Every web page evaluated shall be archived for reference.
The web pages evaluated should be archived for future reference. At the very least, this should include the title and URI of the web page and the date of the evaluation. To avoid ambiguity, this should be complemented with any of the following:
Requirement 4.e: The tools and methods used to evaluate the web pages shall be recorded.
The tools and methods used to evaluate the web pages should be recorded for future reference. This includes versions of web accessibility evaluation tools, web browsers and add-ons, and other tools used during the evaluation. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation , this recording may apply globally for the entire evaluation, to individual web pages , or to individual checks carried out within the audited web pages .
Requirement 5: The evaluation findings shall be documented according to Requirement 5.a and optionally Requirement 5.b , Requirement 5.c , Requirement 5.d , Requirement 5.e , and Requirement 5.f .
This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting your findings is a key element of every evaluation and helps facilitate replicability of the results.
Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.
Requirement 5.a: Each outcome of the steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample shall be documented.
Document the outcomes of each of the previous steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample , to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:
Documentation
of
the
outcome
of
auditing
the
representative
sample
depends
on
the
level
of
detail
for
reporting
defined
by
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
.
Outcome
of
the
auditing
must
include
the
following
information
depending
on
the
set
goal
(optional
templates
for
reporting
example
reports
are
provided
in
Appendix
C:
Reporting
Templates
Example
Reports
):
Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility statement is provided per Step 5.b: Provide an Accessibility Statement (optional) . The level of confidentiality for evaluation reports is usually determined by the evaluation commissioner .
[
Editor
Note:
This
section
has
been
completed
and
needs
to
be
approved
by
Eval
TF.]
[
Review
Note:
Eval
TF
is
particularly
looking
for
feedback
on
this
section.]
Requirement 5.b: An accessibility statement shall be provided.
WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements . Accessibility statements for entire websites can be made according to this methodology when:
As per the requirements for WCAG 2.0 conformance claims , also accessibility statements according to this methodology must include the following information:
Accessibility statements according to this methodology can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support . In such cases the accessibility statements must also include information to identify the following:
[
Editor
Note:
This
section
has
been
completed
and
needs
to
be
approved
by
Eval
TF.]
[
Review
Note:
Eval
TF
is
particularly
looking
for
feedback
on
this
section.]
Requirement 5.c: A performance score shall be provided.
Performance scores can provide more granular measures on the level of conformance of a website than the conformance levels defined by WCAG 2.0 provide. However, scores alone do not provide sufficient context and information to understand the actual level of accessibility of a website . Performance scores according to this methodology may only be provided when the evaluation carried out conforms with this methodology as per 5. Conformance with this Methodology .
Depending on the requested level of detail, the performance score is calculated through one of the following approaches:
Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Web accessibility evaluation tools may be needed to help count and calculate the scores.
The approach used to calculate the score must be indicated together with the numeric ratio whenever the score is provided. For example, "X/Y Per Website", "X/Y Per Web Page", and "X/Y Per Instance" are valid statements to express the performance score.
Requirement 5.f: Machine-readable reports shall be provided.
Machine-readable reports facilitate processing the results by tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language ( EARL ) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL .
The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.
When getting started with improving website accessibility it is useful to do a conformance evaluation to assess the current state of the website . In such situations it is often useful to expand the scope of the evaluation beyond the planned conformance target to get a more complete picture of the accessibility performance on the website . Expanding the scope could include setting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to " In-Depth Analysis " and the conformance target in 3.1.3 Step 1.c. Define the Conformance Target to " WCAG 2.0 Level AAA".
Note: It is recommended to carry out preliminary reviews before carrying out a conformance evaluation to identify obvious errors and to develop a rough understanding of the overall performance of the website .
Evaluation of large websites according to this methodology may be broken down into the evaluation of individual website areas, such as the online shop, departments within an organization, blogging area, and other sub-sites. In this case, each sub-site must meet the terms established in section 2.1 Scope of Applicability and must be evaluated using this methodology. Evaluation of each sub-site must be carried out to at least the same conformance target (as per 3.1.3 Step 1.c. Define the Conformance Target ) as for the main website .
An additional evaluation must be carried out according to this methodology, whereby the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include at least all common web pages of the main website plus two randomly selected web pages from the representative sample selected for each sub-site. The reporting, scoring, and accessibility statements per 3.5 Step 5: Report the Evaluation Findings applies to the entire set of sampled web pages from the main website and all its sub-sites.
Sub-sites may be broken down into further sub-site where applicable. In this case this same clause of the methodology is applied.
Conformance
evaluation
according
to
this
methodology
may
be
re-run
after
a
short
period,
for
example
when
issues
are
identified
and
repaired
by
the
website
owner
or
website
developer
,
or
periodically
to
monitor
progress.
In
such
cases
the
selected
sample
(as
per
3.3
Step
3:
Select
a
Representative
Sample
)
must
include
a
different
set
of
exemplar
web
page
instances
(as
per
3.3.2
Step
3.b:
Include
Exemplar
Instances
of
Web
Pages
),
where
possible.
This
allows
a
portion
of
the
sample
to
remain
unchanged
(
common
web
pages
,
key
web
pages
and
functionality,
etc.),
common
functionality
),
which
facilitates
comparability
between
the
results.
Carrying
out
full
conformance
evaluation
of
multiple
websites
can
involve
a
lot
effort.
This
methodology
is
can
not
limited
to
be
undertaken
using
just
automated
evaluation,
which
can
only
evaluate
a
small
portion
of
the
accessibility
requirements,
and
it
also
requires
manual
evaluation
by
experts.
Limiting
the
goal
of
the
evaluation
in
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
to
"
Basic
Reporting
"
and
careful
selection
of
the
samples
in
3.3
Step
3:
Select
a
Representative
Sample
can
optimize
the
effort
required
for
each
evaluation.
The methodology defined by this document is flexible to allow its implementation in different situations and contexts. It is not required to carry out any of the steps and activities defined by this methodology in any particular sequence. However, it is required that the following requirements defined by this methodology are met:
Results, reports, accessibility statements, and performance scores can only be claimed to be in accordance with this methodology when the evaluation carried out meets these requirements.
This publication has been developed with support from the European Commission funded WAI-ACT Project ( IST 287725) .
Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.
[ Editor note: List needs correct formatting and updating.]
Editor note: A complete example template for evaluation reporting in human understandable language. Discussion proposal below built upon the idea to have three different levels of reporting. The template will have to be ok for official evaluations so some change is still necessary. The difference between option 1 and 2 is currently very small (open for discussion).This section needs appending to the requirements in section 4. this section may change in later drafts depending on how the other sections evolve.
This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three examples are:
The examples should at least have the following information:
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
General information about this checkpoint: link to how to meet and to understanding this success criteria.
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
Description of existing problems and barriers for users or link to descriptions on W3C/WAI website.
General information about this checkpoint: link to how to meet and to understanding this success criteria.
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..
General information about this checkpoint: link to how to meet and to understanding this success criteria.
Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Changes to the Public Working Draft of 27 March include:
A full disposition of comments of all the comments received on the 27 March Working Draft are available.