[ contents ]
Copyright © 2012 W3C ® ( MIT , ERCIM , Keio ), All Rights Reserved. W3C liability , trademark and document use rules apply.
This document ( WCAG-EM ) describes a methodology for evaluating the conformance of websites to the Web Content Accessibility Guidelines ( WCAG ) 2.0 . It provides guidance on defining the evaluation scope and parameters, exploring the website features and functionality, sampling representative web pages where it is not feasible to evaluate all web pages of a website, applying the WCAG 2.0 success criteria and conformance requirements in this setting, and documenting and reporting the evaluation findings. It complements the existing guidance for WCAG 2.0 but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.
This
methodology
only
addresses
accessibility
evaluation
of
existing
websites
after
their
development,
,
for
example
to
validate
conformance
with
reasonable
confidence
or
to
better
understand
the
accessibility
performance
for
improvement.
It
is
applicable
to
any
website,
website
,
including
web
applications
and
websites
for
mobile
devices.
It
is
independent
of
any
particular
evaluation
tool,
web
browser,
and
assistive
technology,
and
it
addresses
different
contexts,
including
self-assessment
and
third-party
evaluation.
However,
ensuring
and
maintaining
accessibility
requires
that
accessibility
is
considered
throughout
the
development
process.
Complementary
resources
from
W3C
/
WAI
on
evaluation
provide
guidance
for
other
situations.
This work is part of complementary W3C / WAI activities on Web Accessibility Evaluation and Testing that address different W3C / WAI guidelines and specifications . It is developed through the collaborative W3C Process with the involvement of international experts and stakeholders.
This section describes the status of this document at the time of its publication. Other documents may supersede this document. A list of current W3C publications and the latest revision of this technical report can be found in the W3C technical reports index at http://www.w3.org/TR/.
This
15
September
13
December
2012
Editors
Draft
of
Website
Accessibility
Conformance
Evaluation
Methodology
1.0
is
based
on
the
same
approach
and
expanding
the
previously
published
W3C
First
Public
Working
Draft
of
27
March
20
September
2012
.
This
Editors
Draft
addresses
the
following
comments
received,
to
seek
approval
for
publication
as
an
updated
Working
Draft:
This
document
is
intended
to
be
published
and
maintained
as
an
informative
W3C
Working
Group
Note
after
review
and
refinement.
The
WCAG
2.0
Evaluation
Methodology
Task
Force
(
Eval
TF
)
invites
discussion
and
feedback
on
this
document
by
developers,
evaluators,
researchers,
and
others
with
interest
in
web
accessibility
evaluation.
Eval
TF
is
particularly
looking
for
feedback
on
these
sections:
the
sections
about
2.1
Scope
of
Applicability
3.3
Step
3:
Select
a
Representative
Sample
3.5.2
Step
5.b:
Provide
an
Accessibility
Statement
(Optional)
and
3.5.3
Step
5.c:
Provide
Performance
Scores
(Optional)
5
Application
of
this
Methodology
.
Please send comments on this Editors Draft of Website Accessibility Conformance Evaluation Methodology 1.0 to public-wai-evaltf@w3.org (publicly visible mailing list archive ). These comments will be considered in the internal discussions of Eval TF .
Publication as Editors Draft does not imply endorsement by the W3C Membership. This is a draft document and may be updated, replaced or obsoleted by other documents at any time. It is inappropriate to cite this document as other than work in progress.
This document has been produced by the WCAG 2.0 Evaluation Methodology Task Force ( Eval TF , a joint task force of the Web Content Accessibility Guidelines Working Group (WCAG WG) and Evaluation and Repair Tools Working Group (ERT WG) , as part of the Web Accessibility Initiative (WAI) Technical Activity .
This document was produced by two groups operating under the 5 February 2004 W3C Patent Policy . The groups do not expect this document to become a W3C Recommendation. W3C maintains a public list of any patent disclosures for WCAG WG and public list of any patent disclosures for ERT WG made in connection with the deliverables of each group; these pages also include instructions for disclosing a patent. An individual who has actual knowledge of a patent which the individual believes contains Essential Claim(s) must disclose the information in accordance with section 6 of the W3C Patent Policy .
Website owners, procurers, suppliers, developers, and others often need to evaluate the conformance of existing websites to the Web Content Accessibility Guidelines ( WCAG ) 2.0 . For example, such evaluation may be carried out as part of releasing or purchasing a website to validate its conformance with reasonable confidence, or it may be to better understand and monitor the level of accessibility provided on a website for improvement. The methodology described in this document provides one possible approach to address this particular aspect of web accessibility evaluation. It does not specifically explain evaluation in other situations nor does it provide general guidance on evaluation with WCAG 2.0. It extends the existing guidance for WCAG 2.0 but it does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way.
The
methodology
described
in
this
document
(
WCAG-EM
)
provides
guidance
on
defining
the
evaluation
scope
and
parameters,
exploring
the
website
features
and
functionality,
sampling
representative
web
pages
where
it
is
not
feasible
to
evaluate
all
web
pages
of
a
website
,
applying
verifying
the
application
of
the
WCAG
2.0
success
criteria
and
conformance
requirements
in
this
setting,
and
documenting
and
reporting
the
evaluation
findings.
It
is
applicable
to
all
websites
,
including
web
applications,
websites
intended
to
be
used
with
mobile
devices,
and
other
types
of
websites
.
It
is
independent
of
the
size
of
the
website
,
the
web
technologies
used
to
create
the
website
,
and
of
any
particular
web
accessibility
evaluation
tools,
web
browsers,
assistive
technologies,
and
other
software
tools.
[
For
Review:
Changed
lines
below
according
to
discussion
in
DoC
and
during
Telco
of
6
december
2012.
We
could
move
or
copy
this
methodology
only
addresses
evaluation
also
to
the
scope
of
applicability
section.
Also
changed
sentence
in
Abstract
to
existing
websites.
Please
review
the
text
below
and
check
if
this
covers
comments
20,
23,
24,
25,
26,
27
and
28
and
Comment
7
and
Comment
155
.
Input
from
EOWG
still
necessary
to
replace
"@@@".
The
main
purpose
of
this
Methodology
is
to
evaluate
an
existing
website
's
conformance
to
WCAG
2.0
after
its
development.
Complementary
resources
from
W3C
/
WAI
on
evaluation
2.0.
This
can
also
be
a
website
provide
guidance
on
that
is
still
in
development.
When
developing
a
website
,
accessibility
should
be
considered
early
and
throughout
the
design
and
development
process.
We
specifically
want
to
encourage
evaluation
throughout
development.
Related
information
outside
the
scope
of
this
document
is
provided
in
other
situations,
@@@(New
link
from
EOWG
to
be
provided
here)
and
complementary
in
Evaluating
Websites
for
Accessibility
,
W3C
/
WAI
activities
on
Web
Accessibility
Evaluation
and
Testing
address
different
Activities
,
and
W3C
/
WAI
guidelines
Guidelines
and
specifications
Techniques
.
Also, WCAG 2.0 conformance requirements apply to individual web pages . However, on most websites it is not feasible to evaluate every web page with reasonable effort. To ensure conformance of a website to WCAG 2.0 it is necessary to ensure that this is considered throughout the development process. Following this methodology helps assess the level of conformance with reasonable confidence where it is not feasible to evaluate every web page .
The primary target audience for this methodology are individuals and organizations, wanting to evaluate the conformance of existing websites to WCAG 2.0. This includes but is not limited to:
Other audiences for whom this methodology is also relevant include:
Users of the methodology are assumed to have expertise in WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web; read more in section 2.2 Required Expertise .
It is assumed that the reader of this document is familiar with the following related resources from W3C / WAI :
A multi-page resource suite that outlines different approaches for evaluating websites for accessibility. The following resources are particularly important in this context:
Internationally recognized standard explaining how to make web content more accessible to people with disabilities. The following resources are particularly important in this context:
The following documents introduce the essential components of web accessibility, inclusive design for people with disabilities and people with aging-related impairments, and overlap with design for mobile devices. The documents help managers, designers, developers, policy makers, researchers, and others optimize their efforts in these overlapping areas:
For the purposes of this document, the following terms and definitions apply:
When a web page is one of a series of web pages presenting a process (i.e., a sequence of steps that need to be completed in order to accomplish an activity), all web pages in the process conform at the specified level or better. (Conformance is not possible at a particular level if any page in the process does not conform at that level or better.)
[
Review
Editor
Note:
We
are
searching
for
input
on
this
definition.
We
previously
used
the
terms
key
functionality
and
primary
Changed
definition
following
decision
in
DoC
.
Note:
Other
functionality
is
not
out
of
scope
--
the
sections
in
this
definition,
which
"common
functionality"
currently
occurs
is
intended
to
help
identify
critical
pages
but
are
searching
for
a
better
term.]
is
not
intended
to
limit
evaluation
to
these
pages
alone.
The content would not conform if that technology is turned off or is not supported
Content patterns that are filled in by authors or the authoring tool to produce content for end users (e.g., document templates, content management templates, presentation themes). Often templates will pre-specify at least some authoring decisions.
A non-embedded resource obtained from a single URI using HTTP plus any other resources that are used in the rendering or intended to be rendered together with it by a user agent
The
methodology
defined
by
this
document
is
used
for
comprehensively
assessing
the
conformance
of
websites
to
WCAG
2.0.
A
more
cursory
approach
for
exploring
the
accessibility
of
a
website
is
described
in
Preliminary
Review
of
Websites
for
Accessibility
.
In
many
cases
it
is
advisable
to
carry
out
such
preliminary
reviews
before
applying
this
methodology,
for
example
to
identify
obvious
errors
and
to
develop
a
rough
understanding
of
the
overall
performance
of
the
website
.
website.
[ Review Note: Eval TF is particularly looking for feedback on this section.]
The
methodology
defined
by
this
document
applies
to
full,
self-enclosed
websites
.
websites.
This
includes
websites
of
organizations
and
entities,
persons,
events,
products,
and
services.
It
also
includes
web
applications
and
mobile
websites
.
websites.
Examples
include:
A
website
may
include
areas
with
smaller
collections
of
related
web
pages
such
as
an
online
shop,
an
area
for
each
department
within
the
organization,
a
blog
area,
and
other
parts.
In
some
situations
such
areas
can
be
considered
to
be
a
full,
self-enclosed
website
each.
This
methodology
can
be
applied
to
such
individual
sub-sites
(a
website
within
another
website
)
website)
and
to
the
main
website
in
its
entirety.
However,
this
methodology
may
not
be
applied
to
a
website
excluding
any
of
its
parts.
Excluding
parts
of
the
website
from
the
scope
of
evaluation
would
likely
conflict
with
the
WCAG
2.0
conformance
requirements
full
pages
and
complete
processes
,
or
significantly
distort
the
evaluation
results.
Example
of
a
Website:
In the diagram above, the university website is made of several sub-sites: "Information for Students", "Information for Lecturers", a "Courseware Application", and a "Library Application". The "Courseware Application" includes "Physics Courses", "Medicine Courses", and "History Courses" that are aggregated into the application. Also, the university website has individual web pages such as legal notices, sitemap, and others that are not part of any particular sub-site.
In
the
example
above,
none
of
the
depicted
parts
may
be
excluded
from
the
scope
of
evaluation
in
the
context
of
this
methodology,
if
it
is
to
be
applied
to
the
university
website
.
website.
This
includes
any
aggregated
and
embedded
content
such
as
online
maps
for
the
university
campus
and
forms
for
credit
card
transactions,
including
when
such
parts
originate
from
third-party
sources.
Although the primary purpose of this document is to evaluate an existing website, we encourage evaluation throughout development process.
Note:
WCAG
2.0
defines
"
Statement
of
Partial
Conformance
"
for
individual
web
pages
that
are
known
not
to
conform
with
WCAG
2.0
due
to
third-party
content
and/or
languages
lacking
accessibility
support
.
Such
web
pages
may
not
be
excluded
from
the
scope
of
evaluation
according
to
this
methodology.
In
some
cases
this
means
that
the
website
as
a
whole
does
not
conform
with
WCAG
2.0
due
to
partially
conforming
web
pages
.
pages.
Section
3.5
Step
5:
Report
the
Evaluation
Findings
provides
more
guidance
on
reporting
evaluation
results
and
making
accessibility
statements
for
entire
websites
.
websites.
This
methodology
is
applicable
to
all
websites
,
websites,
including
web
applications,
websites
intended
to
be
used
with
mobile
devices,
and
other
types
of
websites
.
websites.
Some
particular
types
of
websites
include:
Users of the methodology defined by this document are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web. This includes understanding of the relevant web technologies, barriers that people with disabilities experience, assistive technologies and approaches that people with disabilities use, and evaluation techniques and tools to identify potential barriers for people with disabilities. In particular, it is assumed that users of this methodology are deeply familiar with the resources listed in section 1.3 Background Reading .
The methodology defined by this document is independent of any particular web accessibility evaluation tool, web browsers, and other software tools. However, web accessibility evaluation tools can significantly assist an evaluator during the evaluation process and contribute to more effective evaluations. For example, some web accessibility evaluation tools can scan entire websites to help identify relevant pages for manual evaluation, and other tools can assist manual evaluation in a variety of ways. Specific guidance is provided in Selecting Web Accessibility Evaluation Tools .
Note: Web accessibility evaluation tools can only automatically check a limited set of accessibility requirements that are automatable. Conformance evaluation requires manual testing and judgment by experienced evaluators.
The
methodology
defined
by
this
document
can
be
carried
out
by
an
individual
evaluator
with
the
skills
described
in
section
2.2
Required
Expertise
.
However,
using
the
combined
expertise
of
review
teams
provides
better
coverage
for
the
required
skills
and
helps
identify
accessibility
barriers
more
effectively.
While
not
required,
it
is
strongly
recommended
to
employ
review
teams
for
conformance
evaluation
of
websites
.
websites.
Specific
guidance
is
provided
in
Using
Combined
Expertise
to
Evaluate
Web
Accessibility
.
The methodology defined by this document can be carried out by an individual evaluator with the skills described in section 2.2 Required Expertise . However, involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluator alone. While not required, it is strongly recommended to involve real people covering a wide range of abilities during the evaluation process. Specific guidance is provided in Involving Users in Web Accessibility Evaluation .
This
section
defines
the
stages
and
activities
of
the
evaluation
procedure.
The
stages
are
not
necessarily
sequential
but
rather
iterative.
sequential.
Also
the
exact
sequence
of
the
activities
carried
out
during
the
evaluation
process
depends
on
the
type
of
website
,
website,
the
purpose
of
the
evaluation,
and
the
process
used
by
the
evaluator
.
Some
of
the
activities
overlap
or
may
be
carried
out
in
parallel.
The
following
diagram
illustrates
the
iterations
between
the
stages
defined
in
this
section:
The diagram depicts each of the five steps defined in this section with arrows back and forth between each of two consecutive steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings.
Note: See also section 4. Considerations for Particular Situations that may influence how an evaluation procedure is carried out.
Methodology Requirement 1: Define the evaluation scope according to Methodology Requirement 1.a , Methodology Requirement 1.b , Methodology Requirement 1.c , Methodology Requirement 1.d , and optionally Methodology Requirement 1.e .
During this step the overall scope of the evaluation is defined. It is a fundamental step that affects the subsequent steps in the evaluation procedure and is ideally carried out together with the evaluation commissioner (who may or may not be the website owner ) to ensure common expectations about the scope of the evaluation. Some exploration throughout this stage may be necessary to understand the full scope of the website and the required evaluation (see 3.2: Explore the Target Website ).
Note: Involvement of the website owner and/or website developer (in addition to the evaluation commissioner ) is not required but often helps identify use cases, functionality, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.
Methodology
Requirement
1.a:
Define
the
scope
of
the
website
.
website.
During
this
step
the
exact
scope
of
the
website
(the
web
pages
for
which
the
evaluation
will
apply)
is
defined
and
documented.
This
scope
definition
may
not
contradict
the
terms
established
in
section
2.1
Scope
of
Applicability
.
It
is
essential
to
carefully
document
particular
aspects
such
as
any
use
of
third-party
content
and
services,
services
developed
externally,
different
languages,
and
parts
of
the
website
that
may
not
be
easily
identifiable
as
such
(for
example
an
online
shop
that
is
not
integrated
into
the
website
but
considered
to
be
part
of
it).
The
outcome
of
this
step
is
an
unambiguous
definition
for
the
target
website
,
website,
so
that
for
each
web
page
it
can
be
determined
whether
it
is
within
the
scope
of
evaluation
or
not.
Where
possible,
it
is
recommended
to
use
formalizations
such
as
regular
expressions
or
listings
of
URIs
that
define
the
web
pages
that
are
within
the
scope
of
evaluation
(part
of
the
target
website
).
website).
Methodology Requirement 1.b: Define the goal of the evaluation as " Basic Report ", " Detailed Report ", or " In-Depth Analysis ".
Defining the goal of the evaluation is particularly relevant to the auditing stage defined in 3.4 Step 4: Audit the Selected Sample and the reporting stage defined in 3.5 Step 5: Report the Evaluation Findings . Some of the evaluation goals include:
Methodology Requirement 1.c: Define the target WCAG 2.0 conformance level as "A", "AA", or "AAA".
Part of initiating the evaluation process is to define the target WCAG 2.0 conformance level ("A", "AA", or "AAA"). In many situations WCAG 2.0 Level AA is the generally accepted level of accessibility.
[ Review Note: Eval TF is particularly looking for feedback on requiring uniform accessibility support throughout a single website to be in accordance with this methodology.]
Methodology Requirement 1.d: Define the target users of the website and the minimum set of web browsers and assistive technology for evaluation.
It
is
often
not
feasible
for
websites
to
support
accessibility
on
every
combination
of
web
browser,
assistive
technology,
and
operating
system
that
they
run
on,
nor
is
it
possible
to
test
with
every
such
combination
of
tools.
It
is
necessary
to
determine
the
primary
context
in
which
a
website
is
used
and
the
minimum
set
of
web
browsers
and
assistive
technology
that
must
be
supported
by
the
website
.
website.
For
example,
a
website
may
be
public,
only
available
within
a
closed
network
(intranet),
or
limited
to
authenticated
users
(extranet).
Users
of
intranet
and
extranet
websites
may
be
known
to
have
a
certain
level
of
training
on
using
the
website
and
may
be
using
pre-determined
web
browsers
and
assistive
technology.
This
is
usually
not
the
case
for
public
websites
.
websites.
This definition of target users and tools needs to meet the terms defined in WCAG 2.0 Level of Assistive Technology Support Needed for "Accessibility Support" and needs to be supported throughout the website. For example, if one part of a website is accessible using one set of tools that is different from a set of tools that is needed to access another part of the same website, then the website is effectively not accessible for some users. Accessibility support needs to be uniform throughout a single website.
Note: It is also not possible to single out or exclude individual groups of users, such as "people with visual impairments", in the target audience. All WCAG 2.0 Success Criteria must be considered throughout the evaluation.
[ Review Note: Eval TF is particularly looking for feedback on this section. We would like to know your opinion on the level of inclusion of the techniques. Specifically in relation to section 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional) .]
Methodology Requirement 1.e: Specify the WCAG 2.0 techniques that will be used to evaluate WCAG 2.0 Success Criteria. (Optional).
Techniques in the context of WCAG 2.0 are informative and not required for satisfying the WCAG 2.0 conformance requirements ; WCAG 2.0 Success Criteria are written as testable statements. Techniques provide documented ways of meeting WCAG 2.0 Success Criteria and failures in meeting them. More information on techniques is provided in WCAG 2.0 Layers of Guidance .
W3C / WAI provides a set of publicly documented Techniques for WCAG 2.0 . However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed for such situations. Individuals and organizations developing techniques must employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria.
It is good practice to specify the sets or sources of techniques that are intended to be used during the evaluation at this stage already to ensure consistent expectation between the evaluator and the evaluation commissioner . This definition is typically refined in later stages of the evaluation process, for example during the website exploration and evaluation stages. Section 3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible (Optional) provides more guidance on using WCAG 2.0 techniques during evaluation.
Methodology Requirement 2: Explore the website to be evaluated according to Methodology Requirement 2.a , Methodology Requirement 2.b , Methodology Requirement 2.c , and Methodology Requirement 2.d .
During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website use, purpose, and functionality.
Carrying
out
initial
cursory
checks
during
this
stage
helps
identify
web
pages
that
are
relevant
for
more
detailed
evaluation
later
on.
For
example,
an
evaluator
may
identify
web
pages
that
seem
to
be
lacking
color
contrast,
consistent
navigation,
or
document
structures
(headings,
links,
etc.)
with
simple
checks
while
studying
the
website
,
website,
and
note
them
down
for
more
detailed
evaluation
later
on.
To
carry
out
this
step
it
is
critical
that
the
evaluator
has
access
to
all
the
relevant
parts
of
the
website
.
website.
For
example,
it
may
be
necessary
to
create
accounts
or
otherwise
provide
access
to
restricted
areas
of
a
website
that
are
part
of
the
evaluation.
Note: Involvement of the website owner and/or website developer can be helpful to help identify common web pages , functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough assessment.
Methodology
Requirement
2.a:
Identify
the
common
web
pages
of
the
website
.
website.
During this step the common web pages of the website are identified and documented. The outcome of this step is a list of all common web pages , including the common states of these web pages for web applications . These will be part of the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample .
This
step
also
helps
understand
key
aspects
of
the
website
,
website,
such
as
the
navigation
and
overall
structure
of
the
website
.
website.
Methodology
Requirement
2.b:
Identify
the
common
functionality
of
the
website
.
website.
During this step the common functionality of the website are identified and documented, to provide a comprehensive basis for the sample to be selected through the steps defined in 3.3 Step 3: Select a Representative Sample . The outcome of this step is a list of user tasks such as "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website".
Methodology Requirement 2.c: Identify the variety of web page types.
Web
pages
with
varying
styles,
layouts,
structures,
and
functionality
often
have
different
implementations
of
accessibility
features.
They
are
also
often
generated
by
different
templates
and
authored
by
different
people.
Web
pages
may
also
appear
differently,
behave
differently,
and
contain
different
content
depending
on
the
user.
The
outcome
of
this
step
is
a
list
of
the
different
types
of
web
pages
on
the
website
,
website,
to
provide
a
comprehensive
basis
for
the
sample
to
be
selected
through
the
steps
defined
in
3.3
Step
3:
Select
a
Representative
Sample
.
Different
web
page
types
include:
Note: This step is intended to identify groups of web page instances, including in web applications .
Methodology
Requirement
2.d:
Identify
the
technologies
relied
upon
to
provide
the
website
.
website.
During this step the web technologies relied upon to provide the website are identified and documented, to provide a basis for the sample selection through the steps defined in 3.3 Step 3: Select a Representative Sample . This includes base web technologies such as HTML and CSS , auxiliary web technologies such as Flash, Java, JavaScript, PDF , Silverlight and WAI-ARIA , as well as advanced web technologies such as SMIL and SVG . The outcome of this step is a list of web technologies that are relied upon .
Only the web technologies that are relied upon to provide the website need to be identified for evaluation. This relates closely to the WCAG 2.0 concepts of conforming alternate versions and non-interference .
Note:
Where
possible,
it
is
often
also
useful
to
identify
the
libraries
and
components
used
to
create
the
website
,
website,
such
as
Dojo,
JQuery,
and
others.
Particularly
for
web
applications,
much
of
the
accessibility
support
is
built
into
these
libraries
and
components,
and
evaluation
can
become
more
effective
and
efficient
when
these
are
identified.
[ Review Note: EvalTF is specifically looking for review of section 3.3 Step 3 on sampling. We temporarily use the term "with reasonable confidence" in relation to sampling of representative web pages and would like your input on how to better achieve that in this section of the methodology. We plan to add a random sampling approach in a next version of the methodology.]
Methodology Requirement 3: Select a representative sample of web pages from the website according to Methodology Requirement 3.a , Methodology Requirement 3.b , Methodology Requirement 3.c , and Methodology Requirement 3.d .
While
ideally
every
web
page
of
a
website
is
evaluated,
usually
this
is
not
possible
on
most
websites
.
websites.
In
cases
where
all
web
pages
can
be
evaluated,
this
sampling
procedure
can
be
skipped
and
the
selected
sample
is
considered
to
be
the
entire
website
in
the
remaining
steps.
Exploration
of
the
target
website
in
3.2
Step
2:
Explore
the
Target
Website
(within
the
scope
set
in
3.1
Step
1:
Define
the
Evaluation
Scope
)
provides
sufficient
understanding
of
the
website
to
facilitate
selection
of
a
sample
of
web
pages
that
is
representative
with
reasonable
confidence
of
the
entire
target
website
.
website.
Depending
on
the
size
and
complexity
of
the
website
,
website,
the
size
of
this
sample
will
vary.
For
example,
a
website
with
few
types
of
web
pages
that
are
all
generated
from
a
confined
set
of
templates
,
such
as
a
data
repository,
will
likely
require
a
smaller
sample
than
a
website
with
many
types
of
web
pages
that
are
authored
using
different
mechanisms.
Also
a
web
application
could
have
a
limited
number
of
web
pages
that
dynamically
generate
content
with
varying
types
of
appearance,
behavior,
and
information
that
each
need
to
be
sampled.
The
web
pages
identified
through
the
exploration
will
typically
relate
to
more
than
one
design
aspect.
For
example,
web
pages
with
particular
functionality
such
as
scripting,
multimedia,
and
forms
will
typically
also
use
related
technologies
such
as
Flash,
JavaScript,
PDF
,
Silverlight,
and
WAI-ARIA
,
and
in
many
cases
these
web
pages
may
have
different
design
to
others.
Careful
selection
of
web
pages
can
significantly
reduce
the
required
sample
size
while
maintaining
appropriate
representation
of
the
entire
website
.
website.
Note: In this step the evaluator identifies the comprehensive sample of web pages for evaluation. However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample .
Methodology
Requirement
3.a:
Include
all
common
web
pages
into
the
selected
sample
of
web
pages
.
pages.
All common web pages , including the common states of these web pages for web applications , must be part of the selected sample. These web pages are identified in step 3.2.1 Step 2.a: Identify Common Web Pages of the Website .
Methodology
Requirement
3.b:
Include
at
least
two
distinct
web
pages
(where
applicable
and
available)
of
each
(1)
common
functionality
,
(2)
content,
design,
and
functionality,
and
(3)
web
technologies
into
the
selected
sample
of
web
pages
.
pages.
From the variety and types of web pages identified in 3.2 Step 2: Explore the Target Website (within the scope of the evaluation as defined per 3.1 Step 1: Define the Evaluation Scope ), select at least two distinct web pages with the following features each (where applicable and available):
Note:
A
selected
web
page
could
have
any
number
of
these
features.
For
example,
a
selected
web
page
could
be
used
to
represent
the
use
of
forms
and
scripting
at
the
same
time.
The
important
aspect
is
to
select
at
least
two
distinct
web
pages
for
each
relevant
feature
identified
on
the
website
,
website,
though
more
web
pages
may
be
necessary
depending
on
the
complexity
of
the
website
.
website.
Methodology
Requirement
3.c:
Include
other
web
pages
relevant
for
people
with
disabilities
and
accessibility
into
the
selected
sample
of
web
pages
.
pages.
Websites frequently include web pages that are relevant for people with disabilities and accessibility but do not explicitly match the criteria described in the previous sections. These include:
Methodology Requirement 3.d: Include all web pages that are part of a complete process .
The selected sample must include all web pages that belong to a series of web pages presenting a complete process . Also, no web page in the selected sample may be part of a process without all other web pages that are part of that process to be also included into the selected sample.
Methodology Requirement 4: Audit the selected sample of web pages according to Methodology Requirement 4.a , Methodology Requirement 4.b , Methodology Requirement 4.c , and optionally, Methodology Requirement 4.d and Methodology Requirement 4.e .
WCAG
2.0
defines
five
conformance
requirements
that
need
to
be
met
for
each
web
page
in
the
sample
selected
per
3.3
Step
3:
Select
a
Representative
Sample
.
This
includes
checking
whether
each
WCAG
2.0
Success
Criterion
in
the
target
conformance
level
(per
3.1.3
Step
1.c:
Define
the
Conformance
Target
)
has
been
met
or
not
met
for
each
of
these
web
pages
.
pages.
Note:
Many
web
pages
will
have
repetitive
content,
such
as
the
header,
navigation,
and
other
common
components
that
may
not
need
to
be
re-evaluated
on
each
occurrence.
Depending
on
the
level
of
detail
for
reporting
defined
by
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
,
an
evaluator
may
not
need
to
continue
to
identify
successes
and
failures
in
meeting
the
conformance
target
for
these
repetitive
elements
on
every
web
page
.
page.
Section
3.5
Step
5:
Report
the
Evaluation
Findings
provides
more
guidance
on
reporting.
Methodology Requirement 4.a: Check each web page in the sample selected per 3.3 Step 3: Select a Representative Sample for meeting each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) and each of the WCAG 2.0 conformance requirements .
For each web page in the sample selected per 3.3 Step 3: Select a Representative Sample , check whether each of the WCAG 2.0 Success Criteria in the target conformance level (per 3.1.3 Step 1.c: Define the Conformance Target ) and each of the WCAG 2.0 conformance requirements have been met. This includes that all accessibility features, especially common functionality defined per 3.2.2 Step 2.b: Identify Common Functionality of the Website , are usable for the audiences and with the tools defined per 3.1.4 Step 4.d: Define the Context of Website Use .
Note: According to WCAG 2.0, Success Criteria to which there is no matching content are deemed to have been satisfied . An outcome such as 'Not Applicable' may be used to denote the particular situation where Success Criteria were satisfied because no relevant content was applicable.
To carry out an evaluation effectively, it is often useful to construct and apply personas, use cases, and scenarios of users with a variety of abilities and using different web browsing techniques, including assistive technology and adaptive strategies. It is critical to consider the broadest possible spectrum of use cases to help identify issues that may occur to different audiences. It is strongly recommended to also involve real users during this process, to help identify issues that may not be easily identified through expert testing alone. See Involving Users in Evaluating Web Accessibility for more guidance.
This assessment also needs to be complemented with focused testing of particular situations including:
Note:
Templates
are
often
used
to
create
many
web
pages
,
pages,
sometimes
entire
parts
of
a
website
.
website.
While
evaluating
templates
is
optional
in
this
methodology,
in
some
contexts
it
can
be
helpful
to
check
templates
on
their
own.
Evaluating
templates
may
identify
potential
issues
that
may
not
be
easily
identified
through
evaluating
individual
instances
of
web
pages
.
pages.
However,
issues
identified
in
templates
alone
do
not
necessarily
imply
that
these
issues
occur
on
the
website
and
need
to
be
validated
on
individual
instances
of
web
pages
.
pages.
Also,
identifying
no
issues
in
templates
does
not
necessarily
imply
that
no
issues
occur
on
instances
of
web
pages
.
pages.
Methodology Requirement 4.b: Where possible, use documented WCAG 2.0 techniques and failures to help assess successes and failures in meeting the WCAG 2.0 Success Criteria relevant per 3.1.3 Step 1.c: Define the Conformance Target (Optional) .
Reminder: Techniques and failures in the context of WCAG 2.0 are only informative. They can help assess if WCAG 2.0 Success Criteria are met by providing documented ways of meeting them and commonly occurring failures in meeting them. However, as per the WCAG 2.0 conformance requirements , only the Success Criteria must be met. And you can use any techniques that meet the Success Criteria, whether they are documented yet by the WCAG WG or not.
The initial sets or sources of WCAG 2.0 techniques and failures to be used during evaluation may be defined in section 3.1.5 Step 1.e: Define the Techniques to be Used (Optional) . However, during evaluation such initial sets may often need to be refined according to the particular situation, such as for evaluating particular web technologies and accessibility features that are identified on the website being evaluated.
WCAG 2.0 techniques are documented ways for meeting or for going beyond what is required by individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is met on a web page when:
Conversely, failures are documented ways of not meeting individual WCAG 2.0 Success Criteria. A WCAG 2.0 Success Criterion is not met on a web page when a failure applies to any instance of web content that is addressed by the WCAG 2.0 Success Criterion.
WCAG 2.0 techniques are not the only way to meet WCAG 2.0 Success Criteria, and WCAG 2.0 failures are not the only way to fail WCAG 2.0 Success Criteria. WCAG 2.0 techniques and failures are not exhaustive as they cannot cover every possible situation. Also, the techniques used to meet WCAG 2.0 Success Criteria during the development may not be known to the evaluator . Particularly for newly released web technologies, or when these web technologies are used in particular contexts, there may be no publicly or proprietary documented techniques and failures available to the evaluator . The evaluator must consider these limitations when using WCAG 2.0 techniques and failures to evaluate conformance with WCAG 2.0.
Methodology Requirement 4.c: Check if accessibility features provided on the website are accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use .
To ensure that the accessibility features such as text-alternatives, captions, keyboard access are actually usable in practice, each of these accessibility features must be accessibility supported by the tools defined in Step 1.d: Define the Context of Website Use .
In some situations WCAG 2.0 techniques and repositories on accessibility support provide insights on the level of accessibility support for accessibility features in particular combinations of web technologies and tools. However, the evaluator is responsible for the accuracy of the assessment of accessibility support and the resulting evaluation.
Methodology Requirement 4.d: Archive every evaluated web page for reference. (Optional)
Archive web pages for future reference. At the very least, include the title and URI of the web page and the date of the evaluation. To avoid ambiguity, complement this with any of the following:
Methodology
Requirement
4.e:
Record
the
tools
and
methods
used
to
evaluate
the
web
pages
.
pages.
(Optional).
For
future
reference,
record
Evaluation
tools
and
methods
used
to
evaluate
the
web
pages
.
pages.
This
includes
versions
of
web
accessibility
evaluation
tools,
web
browsers
and
add-ons,
and
other
tools
used
during
the
evaluation.
Depending
on
the
level
of
detail
for
reporting
defined
by
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
,
this
recording
may
apply
globally
for
the
entire
evaluation,
to
individual
web
pages
,
pages,
or
to
individual
checks
carried
out
within
the
audited
web
pages
.
pages.
Methodology
Requirement
5:
Document
the
evaluation
findings
according
to
Methodology
Requirement
5.a
and
optionally
Methodology
Requirement
5.b
,
Methodology
Requirement
5.c
,
and
Methodology
Requirement
5.d
,
Methodology
Requirement
5.e
,
and
Methodology
Requirement
5.f
.
This section describes how to report the evaluation findings that have been gathered during the previous steps. Reporting your findings is a key element of every evaluation and helps facilitate replicability of the results.
Note: Individual pieces of the reporting are gathered throughout the evaluation process, not necessarily at the end of it.
Methodology Requirement 5.a: Document each outcome of the steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample .
Document the outcomes of each of the previous steps defined in 3.1 Step 1: Define the Evaluation Scope , 3.2 Step 2: Explore the Target Website , 3.3 Step 3: Select a Representative Sample , and 3.4 Step 4: Audit the Selected Sample , to ensure justifiable, transparent, and repeatable evaluation results. In particular, this includes documenting:
Documentation of the outcome of auditing the representative sample depends on the level of detail for reporting defined by 3.1.2 Step 1.b: Define the Goal of the Evaluation . Outcome of the auditing must include the following information depending on the set goal (optional example reports are provided in Appendix C: Example Reports ):
Note: While such a report is required for conformance with this methodology, it is not required for any parts of this report to be made public unless an optional public accessibility statement is provided per Step 5.b: Provide an Accessibility Statement (optional) . The level of confidentiality for evaluation reports is usually determined by the evaluation commissioner .
[ Review Note: Eval TF is particularly looking for feedback on this section.]
Methodology Requirement 5.b: Provide an Accessibility Evaluation Statement. (Optional).
WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements . Accessibility evaluation statements for entire websites can be made according to this methodology when:
As per the requirements for WCAG 2.0 conformance claims , also accessibility evaluation statements must include the following information:
Accessibility evaluation statements can also be made when only partial conformance has been achieved according to the requirements defined in third-party content and languages lacking accessibility support . In such cases the accessibility evaluation statements must also include information to identify the following:
[ Review Note: Eval TF is particularly looking for feedback on this section.]
Methodology Requirement 5.c: Provide a performance score. (Optional).
Performance
scores
can
provide
more
granular
measures
on
the
level
of
conformance
of
a
website
than
the
conformance
levels
defined
by
WCAG
2.0
provide.
However,
scores
alone
do
not
provide
sufficient
context
and
information
to
understand
the
actual
level
of
accessibility
of
a
website
.
website.
Performance
scores
according
to
this
methodology
may
only
be
provided
when
the
evaluation
carried
out
conforms
with
this
methodology
as
per
5.
Conformance
with
this
Methodology
.
Depending on the requested level of detail, the performance score is calculated through one of the following approaches:
Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met. Web accessibility evaluation tools may be needed during the evaluation process to help count and calculate the scores.
The approach used to calculate the score must be indicated together with the numeric ratio whenever the score is provided. For example, "X/Y Per Website", "X/Y Per Web Page", and "X/Y Per Instance" are valid statements to express the performance score.
Methodology
Requirement
5.f:
5.d:
Provide
machine-readable
reports.
(Optional).
Machine-readable reports facilitate processing the results by tools, for example to help monitor accessibility of a website over time. The Evaluation and Report Language ( EARL ) is a machine-readable format that was specifically designed for this purpose. It is recommended to use EARL for providing machine-readable reports. See also Understanding Metadata to learn more about uses of metadata, including machine-readable reports, such as EARL .
The methodology defined by this document is flexible to allow its implementation in different situations and contexts. This section provides guidance for applying this methodology in some of these situations.
When
getting
started
with
improving
website
accessibility
it
is
useful
to
do
a
conformance
evaluation
to
assess
the
current
state
of
the
website
.
website.
In
such
situations
it
is
often
useful
to
expand
the
scope
of
the
evaluation
beyond
the
planned
conformance
target
to
get
a
more
complete
picture
of
the
accessibility
performance
on
the
website
.
website.
Expanding
the
scope
could
include
setting
the
goal
of
the
evaluation
in
3.1.2
Step
1.b:
Define
the
Goal
of
the
Evaluation
to
"
In-Depth
Analysis
"
and
the
conformance
target
in
3.1.3
Step
1.c.
Define
the
Conformance
Target
to
"
WCAG
2.0
Level
AAA".
Note:
It
is
recommended
to
carry
out
preliminary
reviews
before
carrying
out
a
conformance
evaluation
to
identify
obvious
errors
and
to
develop
a
rough
understanding
of
the
overall
performance
of
the
website
.
website.
Evaluation
of
large
websites
according
to
this
methodology
may
be
broken
down
into
the
evaluation
of
individual
website
areas,
such
as
the
online
shop,
departments
within
an
organization,
blogging
area,
and
other
sub-sites.
In
this
case,
each
sub-site
must
meet
the
terms
established
in
section
2.1
Scope
of
Applicability
and
must
be
evaluated
using
this
methodology.
Evaluation
of
each
sub-site
must
be
carried
out
to
at
least
the
same
conformance
target
(as
per
3.1.3
Step
1.c.
Define
the
Conformance
Target
)
as
for
the
main
website
.
website.
An additional evaluation must be carried out according to this methodology, whereby the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include at least all common web pages of the main website plus two randomly selected web pages from the representative sample selected for each sub-site. The reporting, scoring, and accessibility statements per 3.5 Step 5: Report the Evaluation Findings applies to the entire set of sampled web pages from the main website and all its sub-sites.
Sub-sites may be broken down into further sub-site where applicable. In this case this same clause of the methodology is applied.
Conformance evaluation according to this methodology may be re-run after a short period, for example when issues are identified and repaired by the website owner or website developer , or periodically to monitor progress. In such cases the selected sample (as per 3.3 Step 3: Select a Representative Sample ) must include a different set of exemplar web page instances (as per 3.3.2 Step 3.b: Include Exemplar Instances of Web Pages ), where possible. This allows a portion of the sample to remain unchanged ( common web pages and common functionality ), which facilitates comparability between the results.
Carrying out full conformance evaluation of multiple websites can involve a lot effort. This methodology can not be undertaken using just automated evaluation, which can only evaluate a small portion of the accessibility requirements, it also requires manual evaluation by experts. Limiting the goal of the evaluation in 3.1.2 Step 1.b: Define the Goal of the Evaluation to " Basic Reporting " and careful selection of the samples in 3.3 Step 3: Select a Representative Sample can optimize the effort required for each evaluation.
The methodology defined by this document is flexible to allow its implementation in different situations and contexts. It is not required to carry out any of the steps and activities defined by this methodology in any particular sequence. For proper application of this methodology, use all the following Methodology Requirements:
Results, reports, accessibility statements, and performance scores can only be claimed to be in accordance with this methodology when the evaluation carried out meets these Methodology Requirements.
This publication has been developed with support from the European Commission funded WAI-ACT Project ( IST 287725) .
Past and present active participants of the WCAG 2.0 Evaluation Methodology Task Force (Eval TF) include: Shadi Abou-Zahra; Frederick Boland; Denis Boudreau; Amy Chen; Vivienne Conway; Bim Egan; Michael Elledge; Wilco Fiers; Detlev Fischer; Elizabeth Fong; Vincent François; Alistair Garrison; Emmanuelle Gutiérrez y Restrepo; Katie Haritos-Shea; Martijn Houtepen; Peter Korn; Maureen Kraft; Aurelien Levy; Kerstin Probiesch; Donald Raikes; Corominas Ramon; Roberto Scano; Samuel Sirois; Sarah J Swierenga; Eric Velleman; Konstantinos Votis; Kathleen Wahlbin; Elle Waters; Richard Warren; Léonie Watson.
Review Note: A complete example template for evaluation reporting in human understandable language. Discussion proposal below is built upon the idea to have three different levels of reporting. The template will have to be ok for official evaluations so change is still necessary. The difference between option 1 and 2 is currently very small (open for discussion).This section needs appending to the Methodology Requirements in section 4. this section may change in later drafts depending on how the other sections evolve.
This Appendix proposes three different levels of reporting following the discussion in clause 5 on levels of evaluation. The three examples are:
The examples should at least have the following information:
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
General information about this checkpoint: link to how to meet and to understanding this success criteria.
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
Description of existing problems and barriers for users or link to descriptions on W3C/WAI website.
General information about this checkpoint: link to how to meet and to understanding this success criteria.
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Results
Guideline X (heading)
Checkpoint: SC XX (subheading)
Result: pass/fail
Character: global/regional (or another wording) if regional: a list of pages where the problem exists
Description of existing problems and barriers for users or link to descriptions on W3C/WAI website..
General information about this checkpoint: link to how to meet and to understanding this success criteria.
Action: Description of techniques for meeting the SC (could be techniques which are already in the techniques document or new techniques which are not in the document, but with which the SC can be met).
List of user agents, including assistive technologies that where used during the evaluation of the website and their versions.
List of pages in the sample
Changes
to
since
the
Public
Working
Draft
of
27
March
include:
A full disposition of comments of all the comments received on the 27 March Working Draft are available.