[ contents ]
Copyright © 2012 W3C ® ( MIT , ERCIM , Keio ), All Rights Reserved. W3C liability , trademark and document use rules apply.
[ Editor Note: This section has changed significantly from 27 August 2012 Editor Draft and needs review.]
This
document
specifies
an
internationally
harmonized
describes
a
methodology
for
evaluating
the
accessibility
conformance
of
existing
websites
to
the
Web
Content
Accessibility
Guidelines
(
WCAG
)
2.0
.
It
defines
an
approach
for
conformance
evaluation
of
entire
websites
as
opposed
to
page-by-page
provides
guidance
on
defining
the
evaluation
that
scope
and
parameters,
exploring
the
website
features
and
functionality,
sampling
representative
web
pages
where
it
is
already
defined
by
not
feasible
to
evaluate
all
web
pages
of
a
website,
applying
the
WCAG
2.0.
It
addresses
2.0
success
criteria
and
conformance
evaluation
requirements
in
different
contexts,
including
self-assessment
this
setting,
and
third-party
evaluation.
documenting
and
reporting
the
evaluation
findings.
It
extends
the
existing
guidance
for
WCAG
2.0
but
it
does
not
define
additional
WCAG
2.0
requirements
nor
does
it
replace
or
supersede
it
in
any
way.
This
methodology
only
addresses
accessibility
evaluation
of
websites
after
their
development,
for
example
to
validate
conformance
with
reasonable
confidence
or
to
better
understand
the
accessibility
performance
for
improvement.
It
is
applicable
to
any
website,
including
web
applications
and
websites
for
mobile
devices,
and
devices.
It
is
independent
of
any
particular
evaluation
tools,
tool,
web
browsers,
browser,
and
assistive
technologies.
This
document
is
a
supporting
resource
for
WCAG
2.0
technology,
and
does
not
replace
or
supersede
it
in
any
way.
It
addresses
different
contexts,
including
self-assessment
and
third-party
evaluation.
However,
ensuring
and
maintaining
accessibility
requires
that
accessibility
is
considered
throughout
the
need
for
a
standardized
approach
for
evaluating
entire
websites
after
their
development
as
opposed
to
evaluating
individual
collections
of
web
pages
process.
Complementary
resources
from
W3C
/
WAI
on
evaluation
during
their
development,
which
is
equally
important
to
ensure
website
accessibility.
provide
guidance
for
other
situations.
This work is part of complementary W3C / WAI activities on Web Accessibility Evaluation and Testing that address different W3C / WAI guidelines and specifications . It is developed through the collaborative W3C Process with the involvement of international experts and stakeholders.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This
27
August
4
September
2012
Editors
Draft
of
Website
Accessibility
Conformance
Evaluation
Methodology
1.0
is
a
complete
draft
based
on
the
same
approach
and
expanding
the
previously
published
W3C
First
Public
Working
Draft
of
27
March
2012
.
This
Editors
Draft
addresses
the
following
comments
received,
to
seek
approval
for
publication
as
an
updated
Working
Draft:
This
document
is
intended
to
be
published
and
maintained
as
an
informative
W3C
Working
Group
Note
after
review
and
refinement.
The
WCAG
2.0
Evaluation
Methodology
Task
Force
(
Eval
TF
)
invites
discussion
and
feedback
on
this
document
by
developers,
evaluators,
researchers,
and
others
with
interest
in
web
accessibility
evaluation.
Eval
TF
is
particularly
looking
for
feedback
on:
on
these
sections:
Please
send
comments
on
this
Editors
Draft
of
the
Website
Accessibility
Conformance
Evaluation
Methodology
for
WCAG
2.0
1.0
to
public-wai-evaltf@w3.org
(publicly
visible
mailing
list
archive
).
These
comments
will
be
considered
in
the
internal
discussions
of
Eval
TF
.
Publication as Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force ( Eval TF , a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG) , as part of the Web Accessibility Initiative (WAI) Technical Activity .
This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy . The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy .
[ Editor Note: This section has changed significantly from 27 August 2012 Editor Draft and needs review.]
Website
owners,
procurers,
suppliers,
developers,
and
others
are
frequently
tasked
with
assessing
often
need
to
evaluate
the
conformance
of
existing
websites
to
the
Web
Content
Accessibility
Guidelines
(
WCAG
)
2.0
.
This
document
provides
For
example,
such
evaluation
may
be
carried
out
as
part
of
releasing
or
purchasing
a
standardized
methodology
for
evaluating
the
website
to
validate
its
conformance
with
reasonable
confidence,
or
it
may
be
to
better
understand
and
monitor
the
level
of
entire
accessibility
provided
on
a
websites
website
for
improvement.
The
methodology
described
in
this
document
provides
one
possible
approach
to
WCAG
2.0,
as
opposed
to
individual
address
this
particular
aspect
of
web
pages
.
accessibility
evaluation.
It
is
developed
through
the
collaborative
does
not
specifically
explain
evaluation
in
other
situations
nor
does
it
provide
general
guidance
on
evaluation
with
W3C
WCAG
Process
with
the
involvement
of
international
stakeholders.
2.0.
It
is
a
supporting
document
extends
the
existing
guidance
for
WCAG
2.0
and
but
it
does
not
define
additional
WCAG
2.0
requirements
nor
does
it
replace
or
supersede
it
in
any
way.
[
Editor
Note:
This
section
has
changed
significantly
from
27
August
2012
Editor
Draft
and
evaluating
conformance
to
WCAG
2.0
needs
to
be
carried
out
during
all
stages
of
the
process,
including
the
design,
implementation,
and
maintenance
stages
of
a
website
.
review.]
The
guidance
methodology
described
in
this
document
focuses
provides
guidance
on
only
one
aspect
of
evaluation
within
the
development
process,
which
is
defining
the
accessibility
evaluation
of
already
existing
websites
.
This
is
useful
for
determining
scope
and
parameters,
exploring
the
level
of
accessibility
of
a
website
at
a
given
point
in
time,
such
as
at
the
time
of
releasing
or
purchasing
a
product.
It
is
also
useful
for
initial
conformance
assessments
when
getting
started
with
accessibility
features
and
for
continual
monitoring
functionality,
sampling
representative
web
pages
where
it
is
not
feasible
to
evaluate
all
web
pages
of
a
website
accessibility.
The
methodology
described
in
this
document
supports
,
applying
the
WCAG
2.0
success
criteria
and
conformance
evaluation
requirements
in
different
contexts
including
self-assessment
this
setting,
and
third-party
assessment
of
websites
.
documenting
and
reporting
the
evaluation
findings.
It
is
applicable
to
all
websites
,
including
web
applications,
websites
intended
to
be
used
with
mobile
devices,
and
other
types
of
websites
.
It
is
independent
of
the
size
of
the
website
,
the
web
technologies
used
to
create
the
website
,
and
of
any
particular
web
accessibility
evaluation
tools,
web
browsers,
assistive
technologies,
and
other
software
tools.
However, this methodology only addresses evaluation of website conformance to WCAG 2.0 after its development. Complementary resources from W3C / WAI on evaluation provide guidance on evaluation in other situations, and complementary W3C / WAI activities on Web Accessibility Evaluation and Testing address different W3C / WAI guidelines and specifications .
Also, WCAG 2.0 conformance requirements apply to individual web pages . However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page .
The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:
Other audiences for whom this methodology is also relevant include:
Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2.2 Required Expertise .
It
is
assumed
that
the
reader
of
this
document
is
familiar
with
the
following
related
resources
from
W3C
resources:
/
WAI
:
A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:
Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:
The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairments, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:
For the purposes of this document, the following terms and definitions apply:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
The content would not conform if that technology is turned off or is not supported
Content patterns that aremostly used to providefilled in by authors or thebasic components ofauthoring tool to produce content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
The methodology defined by this document is used for comprehensively assessing the conformance of websites to WCAG 2.0. A more cursory approach for exploring the accessibility of a website is described in Preliminary Review of Websites for Accessibility . In many cases it is advisable to carry out such preliminary reviews before applying this methodology, for example to identify obvious errors and to develop a rough understanding of the overall performance of the website .
[ Editor Note: This section has been slightly edited from 27 August 2012 Editor Draft .]
[ Review Note: Eval TF is particularly looking for feedback on this section.]
The methodology defined by this document applies to full, self-enclosed websites . This includes websites of organizations and entities, persons, events, products, and services. It also includes web applications and mobile websites . Examples include:
A
website
may
include
several
areas
with
smaller
sub-sites,
collections
of
related
web
pages
such
as
an
online
shop,
an
area
for
each
department
within
the
organization,
a
blog
area,
and
other
parts.
In
some
situations
such
areas
that
may
each
can
be
considered
to
be
a
full,
self-enclosed
website
.
each.
This
methodology
can
be
applied
to
make
conformance
claims
for
such
individual
sub-sites,
in
which
case
these
sub-sites
would
be
considered
to
be
a
(a
website
each,
within
another
website
)
and
to
the
main
website
in
its
entirety.
However,
this
methodology
may
not
be
applied
to
a
website
excluding
any
of
its
parts.
Excluding
parts
of
the
website
from
the
scope
of
evaluation
would
likely
conflict
with
the
WCAG
2.0
conformance
requirements
full
pages
and
complete
processes
,
or
significantly
distort
the
evaluation
results.
Example
of
a
Website:
In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medicine Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.
In
the
example
above,
none
of
the
depicted
parts
may
be
excluded
from
the
scope
of
evaluation
in
the
context
of
this
methodology,
if
a
conformance
claim
it
is
to
be
made
for
applied
to
the
university
website
according
this
methodology.
.
This
includes
any
aggregated
and
embedded
content
from
third-party
sources
such
as
online
maps
for
the
university
campus
and
forms
for
credit
card
transactions.
Excluding
transactions,
including
when
such
parts
of
the
website
originate
from
the
scope
of
evaluation
would
likely
conflict
with
the
WCAG
2.0
conformance
requirements
full
pages
and
complete
processes
.
third-party
sources.
Note:
WCAG
2.0
defines
"
Statement
of
Partial
Conformance
"
for
individual
web
pages
that
are
known
not
to
conform
with
WCAG
2.0
due
to
third-party
content
and/or
languages
lacking
accessibility
support
.
Such
web
pages
may
not
be
excluded
from
the
scope
of
evaluation
according
to
this
methodology.
In
some
cases
this
means
that
the
website
as
a
whole
does
not
conform
with
WCAG
2.0
due
to
only
partially
conforming
web
pages
.
Section
3.5
Step
5:
Report
the
Evaluation
Findings
provides
more
guidance
on
reporting
evaluation
results
and
making
accessibility
statements
for
entire
websites
.
This methodology is applicable to all websites , including web applications, websites intended to be used with mobile devices, and other types of websites . Some particular types of websites include:
Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading .
The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools .
Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.
The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, using the combined expertise of review teams provides better coverage for the required skills and helps identify accessibility barriers more effectively. While not required, it is strongly recommended to employ review teams for conformance evaluation of websites . Specific guidance is provided in Using Combined Expertise to Evaluate Web Accessibility .
The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation .
This section defines the stages and activities of the evaluation procedure. The stages are not necessarily sequential but rather iterative. Also the exact sequence of the activities carried out during the evaluation process depends on the type of website , the purpose of the evaluation, and the process used by the evaluator . Some of the activities overlap or be carried out in parallel. The following diagram illustrates the iterations between the stages defined in this section:
The diagram depicts each of the five steps defined in this section with arrows back and forth between each of two consecutive steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings.
Note: See also section 4. Considerations for Particular Situations that may influence how an evaluation procedure is carried out.
Methodology
Requirement
1:
The
Define
the
evaluation
scope
shall
be
defined
according
to
Methodology
Requirement
1.a
,
Methodology
Requirement
1.b
,
Methodology
Requirement
1.c
,
Methodology
Requirement
1.d
,
and
optionally
Methodology
Requirement
1.e
.
During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner ) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see 3.2: Explore the Target Website ).
Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner ) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.
Methodology
Requirement
1.a:
The
Define
the
scope
of
the
website
shall
be
defined,
with
the
definition
not
contradicting
the
constraints
expressed
in
section
2.1
Scope
of
Applicability
.
During this step the exact scope of the website (the web pages for which the evaluation will apply) is defined and documented. This scope definition may not contradict the terms established in section 2.1 Scope of Applicability . It is essential to carefully document particular aspects such as any use of third-party content and services, different languages, and parts of the website that may not be easily identifiable as such (for example an online shop that is not integrated into the website but considered to be part of it).
The outcome of this step is an unambiguous definition for the target website , so that for each web page it can be determined whether it is within the scope of evaluation or not. Where possible, it is recommended to use formalizations such as regular expressions or listings of URIs that define the web pages that are within the scope of evaluation (part of the target website ).
Methodology
Requirement
1.b:
The
Define
the
goal
of
the
evaluation
shall
be
defined
as
"
Basic
Report
",
"
Detailed
Report
",
or
"
In-Depth
Analysis
".
Defining the goal of the evaluation is particularly relevant to the auditing stage defined in 3.4 Step 4: Audit the Selected Sample and the reporting stage defined in 3.5 Step 5: Report the Evaluation Findings . Some of the evaluation goals include:
Methodology
Requirement
1.c:
The
Define
the
target
WCAG
2.0
conformance
level
("A",
as
"A",
"AA",
or
"AAA")
to
evaluate
for
shall
be
defined.
"AAA".
Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.
Methodology
Requirement
1.d:
The
Define
the
target
users
of
the
website
and
the
minimum
set
of
web
browsers
and
assistive
technology
to
evaluate
for
shall
be
defined,
noting
this
definition
shall
not
conflict
with
the
WCAG
2.0
guidance
on
the
Level
of
Assistive
Technology
Support
Needed
for
"Accessibility
Support"
.
It is often not feasible for websites to support accessibility on every combination of web browser, assistive technology, and operating system that they run on, nor is it possible to test with every such combination of tools. It is necessary to determine the primary context in which a website is used and the minimum set of web browsers and assistive technology that must be supported by the website .
For example, a website may be public, only available within a closed network (intranet), or limited to authenticated users (extranet). Users of intranet and extranet websites may be known to have a certain level of training on using the website and may be using pre-determined web browsers and assistive technology. This is usually not the case for public websites .
This definition of target users and tools must meet the terms defined in WCAG 2.0 Level of Assistive Technology Support Needed for "Accessibility Support" and must be used throughout the evaluation. For example, it is not possible to evaluate some pages with one set of tools and other pages with another set. Accessibility support must be uniform throughout a single website .
Note: It is also not possible to single out or exclude individual groups of users, such as "people with visual impairments", in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.
[
Editor
Note:
This
Eval
TF
is
particularly
looking
for
feedback
on
this
section.
We
would
like
to
know
your
opinion
on
the
level
of
inclusion
of
the
techniques.
Specifically
in
relation
to
section
has
changed
significantly
from
30
July
2012
Editor
Draft
3.4.2
Step
4.b:
Use
WCAG
2.0
Techniques
Where
Possible
(Optional)
and
needs
review.]
]
Methodology
Requirement
1.e:
The
Specify
the
WCAG
2.0
techniques
that
will
be
used
to
evaluate
WCAG
2.0
Success
Criteria
should
be
specified.
Criteria.
(Optional).
Techniques in the context of WCAG 2.0 are informative and not required for satisfying the WCAG 2.0 conformance requirements ; WCAG 2.0 Success Criteria are written as testable statements. Techniques provide documented ways of meeting WCAG 2.0 Success Criteria and failures in meeting them. More information on techniques is provided in WCAG 2.0 Layers of Guidance .
W3C / WAI provides a set of publicly documented Techniques for WCAG 2.0 . However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed for such situations. Individuals and organizations developing techniques must employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria.
It is good practice to specify the sets or sources of techniques that are intended to be used during the evaluation at this stage already to ensure consistent expectation between the evaluator and the evaluation commissioner . This definition is typically refined in later stages of the evaluation process, for example during the website exploration and evaluation stages. Section 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional) provides more guidance on using WCAG 2.0 techniques during evaluation.
Methodology
Requirement
2:
The
Explore
the
website
to
be
evaluated
shall
be
explored
according
to
Methodology
Requirement
2.a
,
Methodology
Requirement
2.b
,
Methodology
Requirement
2.c
,
and
Methodology
Requirement
2.d
.
During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use, purpose, and functionality.
Carrying out initial cursory checks during this stage helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website , and note them down for more detailed evaluation later on.
To carry out this step it is critical that the evaluator has access to all the relevant parts of the website . For example, it may be necessary to create accounts or otherwise provide access to restricted areas of a website that are part of the evaluation.
Note: Involvement of the website owner and/or website developer can be helpful to help identify common web pages , functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.
Methodology
Requirement
2.a:
The
Identify
the
common
web
pages
of
the
website
shall
be
identified.
.
During this step the common web pages of the website are identified and documented. The outcome of this step is a list of all common web pages , including the common states of these web pages for web applications . These will be part of the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample .
This step also helps understand key aspects of the website , such as the navigation and overall structure of the website .
Methodology
Requirement
2.b:
The
Identify
the
common
functionality
of
the
website
shall
be
identified.
.
During this step the common functionality of the website are identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample . The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".
Methodology
Requirement
2.c:
The
Identify
the
variety
of
web
page
types
shall
be
identified.
types.
Web pages with varying styles, layouts, structures, and functionality often have different implementations of accessibility features. They are also often generated by different templates and authored by different people. Web pages may also appear differently, behave differently, and contain different content depending on the user. The outcome of this step is a list of the different types of web pages on the website , to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample . Different web page types include:
Note: This step is intended to identify groups of web page instances, including in web applications .
Methodology
Requirement
2.d:
The
Identify
the
technologies
relied
upon
to
provide
the
website
shall
be
identified.
.
During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in 3.3 Step 3: Select a Representative Sample . This includes base web technologies such as HTML and CSS , auxiliary web technologies such as Flash, Java, JavaScript, PDF , Silverlight and WAI-ARIA , as well as advanced web technologies such as SMIL and SVG . The outcome of this step is a list of web technologies that are relied upon .
Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference .
Note: Where possible, it is often also useful to identify the libraries and components used to create the website , such as Dojo, JQuery, and others. Particularly for web applications, much of the accessibility support is built into these libraries and components, and evaluation can become more effective and efficient when these are identified.
Review Note: EvalTF is specifically looking for review of section 3.3 Step 3 on sampling. We temporarily use the term "with reasonable confidence" in relation to sampling of representative web pages and would like your input on how to better achieve that in this section of the methodology.
Methodology
Requirement
3:
A
Select
a
representative
sample
of
web
pages
shall
be
selected
from
the
website
according
to
Methodology
Requirement
3.a
,
Methodology
Requirement
3.b
,
Methodology
Requirement
3.c
,
and
Methodology
Requirement
3.d
.
While ideally every web page of a website is evaluated, usually this is not possible on most websites . In cases where all web pages can be evaluated, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps.
[ Editor Note: Small change made to below paragraph. Review needed. Added "with reasonable confidence".]
Exploration of the target website in 3.2 Step 2: Explore the Target Website (within the scope set in 3.1 Step 1: Define the Evaluation Scope ) provides sufficient understanding of the website to facilitate selection of a sample of web pages that is representative with reasonable confidence of the entire target website . Depending on the size and complexity of the website , the size of this sample will vary. For example, a website with few types of web pages that are all generated from a confined set of templates , such as a data repository, will likely require a smaller sample than a website with many types of web pages that are authored using different mechanisms. Also a web application could have a limited number of web pages that dynamically generate content with varying types of appearance, behavior, and information that each need to be sampled.
The web pages identified through the exploration will typically relate to more than one design aspect. For example, web pages with particular functionality such as scripting, multimedia, and forms will typically also use related technologies such as Flash, JavaScript, PDF , Silverlight, and WAI-ARIA , and in many cases these web pages may have different design to others. Careful selection of web pages can significantly reduce the required sample size while maintaining appropriate representation of the entire website .
Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample .
Methodology
Requirement
3.a:
All
Include
all
common
web
pages
shall
be
part
of
into
the
selected
sample
of
web
pages
.
All common web pages , including the common states of these web pages for web applications , must be part of the selected sample. These web pages are identified in step 3.2.1 Step 2.a: Identify Common Web Pages of the Website .
Methodology
Requirement
3.b:
At
Include
at
least
two
distinct
web
pages
(where
applicable
and
available)
of
each
(1)
common
functionality
,
(2)
content,
design,
and
functionality,
and
(3)
web
technologies
shall
be
part
of
into
the
selected
sample
of
web
pages
.
From the variety and types of web pages identified in 3.2 Step 2: Explore the Target Website (within the scope of the evaluation as defined per 3.1 Step 1: Define the Evaluation Scope ), select at least two distinct web pages with the following features each (where applicable and available):
Note: A selected web page could have any number of these features. For example, a selected web page could be used to represent the use of forms and scripting at the same time. The important aspect is to select at least two distinct web pages for each relevant feature identified on the website , though more web pages may be necessary depending on the complexity of the website .
Methodology
Requirement
3.c:
Other
Include
other
web
pages
relevant
for
people
with
disabilities
and
accessibility
shall
be
part
of
into
the
selected
sample
of
web
pages
.
Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:
Methodology
Requirement
3.d:
All
Include
all
web
pages
that
are
part
of
a
complete
process
shall
be
included.
.
The selected sample must include all web pages that belong to a series of web pages presenting a complete process . Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.
Methodology
Requirement
4:
The
Audit
the
selected
sample
of
web
pages
shall
be
audited
according
to
Methodology
Requirement
4.a
,
Methodology
Requirement
4.b
,
Methodology
Requirement
4.c
,
and
optionally,
Methodology
Requirement
4.d
and
Methodology
Requirement
4.e
.
WCAG 2.0 defines five conformance requirements that need to be met for each web page in the sample selected per 3.3 Step 3: Select a Representative Sample . This includes checking whether each WCAG 2.0 Success Criterion in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) has been met or not met for each of these web pages .
Note: Many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Depending on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation , an evaluator may not need to continue to identify successes and failures in meeting the conformance target for these repetitive elements on every web page . Section 3.5 Step 5: Report the Evaluation Findings provides more guidance on reporting.
Methodology
Requirement
4.a:
Each
Check
each
web
page
in
the
sample
selected
per
3.3
Step
3:
Select
a
Representative
Sample
shall
be
checked
for
meeting
each
of
the
WCAG
2.0
Success
Criteria
in
the
target
conformance
level
(per
3.1.3
Step
1.c:
Define
the
Conformance
Target
)
and
each
of
the
WCAG
2.0
conformance
requirements
.
For each web page in the sample selected per 3.3 Step 3: Select a Representative Sample , check whether each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) and each of the WCAG 2.0 conformance requirements have been met. This includes that all accessibility features, especially common functionality defined per 3.2.2 Step 2.b: Identify Common Functionality of the Website , are usable for the audiences and with the tools defined per 3.1.4 Step 4.d: Define the Context of Website Use .
Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied . An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable.
To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.
This assessment also needs to be complemented with focused testing of particular situations including:
Note:
Templates
are
usually
often
used
to
create
many
web
pages
,
sometimes
entire
parts
of
a
website
.
While
evaluating
templates
is
optional
in
this
methodology,
in
some
contexts
it
can
be
helpful
to
check
templates
on
their
own.
Evaluating
templates
may
identify
potential
issues
that
may
not
be
easily
identified
through
evaluating
individual
instances
of
web
pages
.
However,
issues
identified
in
templates
alone
do
not
necessarily
imply
that
these
issues
occur
on
the
website
and
need
to
be
validated
on
individual
instances
of
web
pages
.
Also,
identifying
no
issues
in
templates
does
not
necessarily
imply
that
no
issues
occur
on
instances
of
web
pages
.
[
Editor
Note:
Need
agreement
on
this
note.]
Methodology
Requirement
4.b:
Where
possible,
use
applicable
WCAG
2.0
techniques
shall
be
used
to
demonstrate
successes
and
failures
in
meeting
the
WCAG
2.0
Success
Criteria
relevant
per
3.1.3
Step
1.c:
Define
the
Conformance
Target
.
(Optional).
The initial sets or sources of WCAG 2.0 techniques to be used during evaluation may be defined in section 3.1.5 Step 1.e: Define the Techniques to be Used (Optional) . However, during evaluation such an initial set may often need to be refined according to the particular situation, such as for evaluating particular web technologies and accessibility features.
WCAG 2.0 defines three types of techniques:
For each web page a WCAG 2.0 Success Criterion is:
WCAG 2.0 techniques are not exhaustive as they cannot cover every possible situation. Also, the techniques used to meet WCAG 2.0 Success Criteria during the development may not be known to the evaluator. Particularly for newly released web technologies or when these web technologies are used in particular contexts there may be no publicly or proprietary documented techniques available to the evaluator . The evaluator must be considerate of these limitations when using WCAG 2.0 techniques
Note: Advisory techniques may not be fully supported by assistive technology. If they are used, make sure that these work with the web browsers and assistive technology defined in 3.1.4 Step 1.d: Define the Context of Website Use .
Methodology
Requirement
4.c:
Accessibility
Check
if
accessibility
features
provided
on
the
website
shall
be
checked
to
be
are
accessibility
supported
by
the
tools
defined
in
Step
1.d:
Define
the
Context
of
Website
Use
.
To ensure that the accessibility features such as text-alternatives, captions, keyboard access are actually usable in practice, each of these accessibility features must be accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use .
In some situations WCAG 2.0 techniques and repositories on accessibility support provide insights on the level of accessibility support for accessibility features in particular combinations of web technologies and tools. However, the evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.
Methodology
Requirement
4.d:
Every
Archive
every
evaluated
web
page
evaluated
shall
be
archived
for
reference.
(Optional)
The
Archive
web
pages
evaluated
should
be
archived
for
future
reference.
At
the
very
least,
this
should
include
the
title
and
URI
of
the
web
page
and
the
date
of
the
evaluation.
To
avoid
ambiguity,
complement
this
should
be
complemented
with
any
of
the
following:
Methodology
Requirement
4.e:
The
Record
the
tools
and
methods
used
to
evaluate
the
web
pages
shall
be
recorded.
.
(Optional).
The
For
future
reference,
record
Evaluation
tools
and
methods
used
to
evaluate
the
web
pages
should
be
recorded
for
future
reference.
.
This
includes
versions
of
web
accessibility
evaluation
tools,
web
browsers
and
add-ons,
and
other
tools
used
during
the
evaluation.
Depending
on
the
level
of
detail
for
reporting
defined
by
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
,
this
recording
may
apply
globally
for
the
entire
evaluation,
to
individual
web
pages
,
or
to
individual
checks
carried
out
within
the
audited
web
pages
.
Methodology
Requirement
5:
The
Document
the
evaluation
findings
shall
be
documented
according
to
Methodology
Requirement
5.a
and
optionally
Methodology
Requirement
5.b
,
Methodology
Requirement
5.c
,
Methodology
Requirement
5.d
,
Methodology
Requirement
5.e
,
and
Methodology
Requirement
5.f
.
This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting your findings is a key element of every evaluation and helps facilitate replicability of the results.
Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.
Methodology
Requirement
5.a:
Each
Document
each
outcome
of
the
steps
defined
in
3.1
Step
1:
Define
the
Evaluation
Scope
,
3.2
Step
2:
Explore
the
Target
Website
,
3.3
Step
3:
Select
a
Representative
Sample
,
and
3.4
Step
4:
Audit
the
Selected
Sample
shall
be
documented.
.
Document the outcomes of each of the previous steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample , to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:
Documentation of the outcome of auditing the representative sample depends on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation . Outcome of the auditing must include the following information depending on the set goal (optional example reports are provided in Appendix C: Example Reports ):
Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility statement is provided per Step 5.b: Provide an Accessibility Statement (optional) . The level of confidentiality for evaluation reports is usually determined by the evaluation commissioner .
[ Review Note: Eval TF is particularly looking for feedback on this section.]
[ Editor Note: The title of this section and the requirement have changed. EvalTF need for agreement with this change. Also see discussion in DoC ]
Methodology
Requirement
5.b:
An
accessibility
statement
shall
be
provided.
Provide
an
Accessibility
Statement
according
to
this
Methodology.
(Optional).
WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements . Accessibility statements for entire websites can be made according to this methodology when:
As per the requirements for WCAG 2.0 conformance claims , also accessibility statements according to this methodology must include the following information:
Accessibility statements according to this methodology can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support . In such cases the accessibility statements must also include information to identify the following:
[ Review Note: Eval TF is particularly looking for feedback on this section.]
Methodology
Requirement
5.c:
A
Provide
a
performance
score
shall
be
provided.
score.
(Optional).
Performance scores can provide more granular measures on the level of conformance of a website than the conformance levels defined by WCAG 2.0 provide. However, scores alone do not provide sufficient context and information to understand the actual level of accessibility of a website . Performance scores according to this methodology may only be provided when the evaluation carried out conforms with this methodology as per 5. Conformance with this Methodology .
Depending on the requested level of detail, the performance score is calculated through one of the following approaches:
Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Web accessibility evaluation tools may be needed during the evaluation process to help count and calculate the scores.
The approach used to calculate the score must be indicated together with the numeric ratio whenever the score is provided. For example, "X/Y Per Website", "X/Y Per Web Page", and "X/Y Per Instance" are valid statements to express the performance score.
Methodology
Requirement
5.f:
Machine-readable
reports
shall
be
provided.
Provide
machine-readable
reports.
(Optional).
Machine-readable reports facilitate processing the results by tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language ( EARL ) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL .
The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.
When getting started with improving website accessibility it is useful to do a conformance evaluation to assess the current state of the website . In such situations it is often useful to expand the scope of the evaluation beyond the planned conformance target to get a more complete picture of the accessibility performance on the website . Expanding the scope could include setting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to " In-Depth Analysis " and the conformance target in 3.1.3 Step 1.c. Define the Conformance Target to " WCAG 2.0 Level AAA".
Note: It is recommended to carry out preliminary reviews before carrying out a conformance evaluation to identify obvious errors and to develop a rough understanding of the overall performance of the website .
Evaluation of large websites according to this methodology may be broken down into the evaluation of individual website areas, such as the online shop, departments within an organization, blogging area, and other sub-sites. In this case, each sub-site must meet the terms established in section 2.1 Scope of Applicability and must be evaluated using this methodology. Evaluation of each sub-site must be carried out to at least the same conformance target (as per 3.1.3 Step 1.c. Define the Conformance Target ) as for the main website .
An additional evaluation must be carried out according to this methodology, whereby the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include at least all common web pages of the main website plus two randomly selected web pages from the representative sample selected for each sub-site. The reporting, scoring, and accessibility statements per 3.5 Step 5: Report the Evaluation Findings applies to the entire set of sampled web pages from the main website and all its sub-sites.
Sub-sites may be broken down into further sub-site where applicable. In this case this same clause of the methodology is applied.
Conformance evaluation according to this methodology may be re-run after a short period, for example when issues are identified and repaired by the website owner or website developer , or periodically to monitor progress. In such cases the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include a different set of exemplar web page instances (as per 3.3.2 Step 3.b: Include Exemplar Instances of Web Pages ), where possible. This allows a portion of the sample to remain unchanged ( common web pages and common functionality ), which facilitates comparability between the results.
Carrying out full conformance evaluation of multiple websites can involve a lot effort. This methodology can not be undertaken using just automated evaluation, which can only evaluate a small portion of the accessibility requirements, it also requires manual evaluation by experts. Limiting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to " Basic Reporting " and careful selection of the samples in 3.3 Step 3: Select a Representative Sample can optimize the effort required for each evaluation.
[ Editor Note: This section has changed, please review]
The
methodology
defined
by
this
document
is
flexible
to
allow
its
implementation
in
different
situations
and
contexts.
It
is
not
required
to
carry
out
any
of
the
steps
and
activities
defined
by
this
methodology
in
any
particular
sequence.
However,
it
is
required
that
For
proper
application
of
this
methodology,
use
all
the
following
requirements
defined
by
this
methodology
are
met:
Methodology
Requirements:
Results,
reports,
accessibility
statements,
and
performance
scores
can
only
be
claimed
to
be
in
accordance
with
this
methodology
when
the
evaluation
carried
out
meets
these
requirements.
Methodology
Requirements.
This publication has been developed with support from the European Commission funded WAI-ACT Project ( IST 287725) .
Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.
[ Editor note: List needs correct formatting and updating.]
Editor
note:
A
complete
example
template
for
evaluation
reporting
in
human
understandable
language.
Discussion
proposal
below
is
built
upon
the
idea
to
have
three
different
levels
of
reporting.
The
template
will
have
to
be
ok
for
official
evaluations
so
some
change
is
still
necessary.
The
difference
between
option
1
and
2
is
currently
very
small
(open
for
discussion).This
section
needs
appending
to
the
requirements
Methodology
Requirements
in
section
4.
this
section
may
change
in
later
drafts
depending
on
how
the
other
sections
evolve.
This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three examples are:
The examples should at least have the following information:
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
General information about this checkpoint: link to how to meet and to understanding this success criteria.
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
Description of existing problems and barriers for users or link to descriptions on W3C/WAI website.
General information about this checkpoint: link to how to meet and to understanding this success criteria.
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..
General information about this checkpoint: link to how to meet and to understanding this success criteria.
Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Changes to the Public Working Draft of 27 March include:
A full disposition of comments of all the comments received on the 27 March Working Draft are available.