Disposition of Comments

Website Accessibility Conformance Evaluation Methodology (WCAG-EM),
Editor Draft, 29 November 2013

This is a disposition of comments received on the Website Accessibility Conformance Evaluation Methodology (WCAG-EM), Editor Draft of 29 November 2013. This page is intended for internal discussion by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF).

This is a disposition of comments received on the 29 November 2013 Editor Draft of the Website Accessibility Conformance Evaluation Methodology (WCAG-EM). The latest Working Draft is provided at http://www.w3.org/TR/WCAG-EM/. The latest Editor Draft is provided at http://www.w3.org/WAI/ER/methodology/.

Contents

  1. Overall Comments
  2. Comments on Abstract
  3. Comments on Introduction
  4. Comments on Using this Methodology
  5. Comments on Scope of Applicability
  6. Comments on Step 1
  7. Comments on Step 2
  8. Comments on Step 3
  9. Comments on Step 4
  10. Comments on Step 5

Overall Comments

ID Status Commenter Location Current text Suggested Change Rationale Resolution
45 Closed Michael Cooper Throughout Section headings Add section numbers hard to tell from minor size differences what are sub sections and what are new sections

Resolution: Add spacing and size (as permitted by PubRules) rather than to add section numbers.

Rationale: Had section numbers initally but it got very confusing (eg. "4.3 Step 3" etc.).
46 Closed Kerstin Probiesch Throughout Term "Optional" Change to "not required" or "not mandatory" or "not facultativ" or add a definition of "optional" in the "Terms and Definitions" section, which clearly says that all sections which are marked as optional are not facultativ. I think the term "optional" could be misleading and there is no definition of this in the "Terms and Definitions" section.

Resolution: No change.

Rationale: The term "optional" is commonly used and not more difficult to understand than "not required" or "not mandatory" or "not facultativ".
85 Closed Samuel Martin - - Appendix C Example Reports has been removed; Add a report example at some point. Provide users with applied examples of the methodology. I understand this maybe will be addressed in future versions.

Resolution: No change

Rationale: This has moved from the appendix to step 5.a. In that section there is a minimum overview of what should be recorded/reported. The evaluator and the evaluation commissioner can agree further on details to be included or the form.

Comments on Abstract

ID Status Commenter Location Current text Suggested Change Rationale Resolution
1 Closed Bim Egan Abstract Second paragraph; both first and second sentences Change one of the instances of "primarily"; one of the instances could use the word "mainly" Contains the adjective "primarily" in both sentences

Resolution: Remove "primarily" from the first sentence.

Rationale: It is actually an expected expertise later on in the document.
2 Closed Michael Cooper, Loretta Guarino Reid, Detlev Fischer, Samuel Martin Abstract "It describes a procedure to evaluate websites and includes considerations to guide evaluators and to promote good practice". Put back some of the previous text "It provides guidance on defining parameters for the evaluation scope; on exploring and understanding websites including their key features and functionality; on sampling representative web pages from websites where it is not feasible to evaluate all web pages of a website; on evaluating the conformance of these selected web pages to the target level of WCAG 2.0 conformance; and on documenting and reporting the findings from such evaluation." Current text is too glossy

Resolution: Add to existing text "It provides guidance on good practice in defining the evaluation scope, exploring the target website, selecting representative samples from websites where it is not feasible to evaluate all content, auditing the selected samples, and recording the evaluation findings."

Rationale: Put back some of the text with consideration for Detlev's point.
3 Closed Gregg Vanderheiden, Alistair Garrison Abstract It does not provide instructions for evaluating web content feature by feature, which is addressed by the WCAG 2.0 techniques layer. It does not provide instructions for evaluating web content feature by feature, which is addressed by WCAG 2.0 success criteria. The techniques are not designed to be used for evaluation. they are neither necessary nor sufficient -- so they cannot be used to evaluate.

Resolution: Change as proposed.

Rationale: More accurate.

See also Comment 9.
4 Closed Alistair Garrison Abstract It is primarily designed for evaluating existing websites, for example to learn about them and to monitor their level of accessibility. Make bold. It is the only place in the whole document that the documents actual purpose is defined - if you miss this sentence, you will not understand the purpose of the document just from the text.

Resolution: No change.

Rationale: Not good to bold in abstract as all sentences are important; have an entire Purposes for this Methodology section saying that in more detail.

Comments on Introduction

ID Status Commenter Location Current text Suggested Change Rationale Resolution
5 Closed Sharron Rush Introduction "needs to". Replace with "must," "should consider," or "will want to". I understand that there are constraints around some of the words, but the "need to" is so repetitive it becomes distracting.

Resolution: No change.

Rationale: There are only three occurrences of "needs to" and none in this section; will double-check though.
6 Closed Bim Egan, Loretta Guarino Reid, Gregg Vanderheiden, Alistair Garrison Introduction "... and highlights considerations that evaluators to apply these steps in the context of a particular website". Rephrase "that evaluators to apply". This sentence seems incomplete

Resolution: Change to "... and highlights considerations for evaluators to apply these steps in the context of a particular website".

Rationale: Typo (used "that" instead of "for").

See also comment 11 and comment 48.
7 Closed David MacDonald Introduction It also defines how optional conformance claims can be made to cover individual web pages, series of web pages such as a multi-page form, and multiple related web pages such as a website". Add "a" in front of "series of web pages". Grammar

Resolution: Change as suggested.

Rationale: Grammar.
8 Closed David MacDonald Introduction List of background reading on "Evaluating Websites for Accessibility" Add "Involving web accessibility experts". -

Resolution: No change.

Rationale: Actually this methodology is the guidance of accessibility experts.
9 Closed Kerstin Probiesch, Loretta Guarino Reid, Detlev Fischer, Gregg Vanderheiden Introduction "The methodology relies on WCAG 2.0 techniques such as the Techniques for WCAG 2.0 documented by W3C/WAI, but is not limited to this set of techniques". Delete this sentence. This is in contradiction with the idea and the concept of the Techniques Document. Not if a technique is used is important but wether an SC is met or not. Therefore a methodology for evaluating can't rely on techniques. What the WG has published here: http://www.w3.org/WAI/WCAG20/wcag2faq.html#techsnot is also relevant for evaluators. Even with "but is not limited to this set of techniques." the paragraph places too much emphasis on the techniques.

Resolution: Change to "The methodolody can be using in conjunction with techniques for meeting WCAG 2.0 success criteria, such as the Techniques for WCAG 2.0 documented by W3C/WAI, but does not require this or any other specific set of techniques".

Rationale: Agree with the concern about the use of the word "relies" but important to explain the role of the techniques in conjunction with this methodology.

See also Comment 3.
11 Closed Michael Cooper Introduction Throughout. Put back some of the previous text "This methodology does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance. Following this methodology may not identify every possible occurrence of support for or violation of WCAG 2.0 conformance. It is one possible approach for evaluating the conformance of existing websites to WCAG 2.0 in different evaluation contexts, including self-assessment and third-party evaluation. It is primarily intended for use by experienced accessibility evaluators". As best I can follow the paper trail, this was removed in response to a comment requesting simplifying the wording. However, I seem to recall the disclaimer this paragraph provides was very important to the WCAG WG. I don't think it should have been simplified out, unless I mis-remember the WCAG view. I think the para needs to be restored. I'm not asking for the Scope section to be restored, just stick the para somewhere. It could use some simplification, just not all the way to nil.

Resolution: Change second paragraph to read "This methodology, describes the steps that are common to website conformance evaluation processes. It highlights considerations for evaluators to apply these steps in the context of a particular website. It does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance. Following this methodology will help evaluators to apply good practice, avoid commonly made mistakes, and achieve more comparable results."

Rationale: Re-inserts lost bits without duplicating information.

See also comment 6, comment 48, and Eval TF minutes of 12 December 2013.
12 Closed Michael Cooper Introduction "website". "web site". Grammar

Resolution: No change.

Rationale: This is current WAI spelling convention.
13 Closed Loretta Guarino Reid, Gregg Vanderheiden, Alistair Garrison Introduction "though in the majority of use cases it does not directly result into conformance claims". "though in the majority of use cases it does not directly result in conformance claims". Grammar

Resolution: Change as suggested.

Rationale: Grammar.
14 Closed Loretta Guarino Reid Introduction "The information below related to web accessibility essentials, evaluation, and WCAG 2.0 is important for using this methodology". "The information below, related to web accessibility essentials, evaluation, and WCAG 2.0, is important for using this methodology". Add commas

Resolution: Change as suggested.

Rationale: Grammar.
15 Closed Detlev Fischer Introduction "Some web page states are ancilliary or treated similarly to individual web pages in the context of this methodology". "In the context of this methodology, web page states can be treated as ancilliary to pages (i.e., recorded as additional state of a page in a page sample) or as individual web pages". Current wording is rather fuzzy

Resolution: Change to "In the context of this methodology, web page states can be treated as ancillary to web pages (i.e., recorded as additional state of a web page in a web page sample) or as individual web pages".

Rationale: Proposed change with minor tweaks.

See also comment 52.
16 Closed Gregg Vanderheiden Introduction "Also, this methodology does not in any way add to or change the requirements defined by the normative WCAG 2.0 standard". "This methodology does not in any way add to or change the requirements defined by the normative WCAG 2.0 standard". -

Resolution: Change as suggested.

Rationale: "Also" not necessary.
17 Closed Gregg Vanderheiden Introduction "WCAG 2.0 defines conformance requirements for individual web pages that are known to satisfy each conformance requirement, rather than for entire websites". "WCAG 2.0 defines conformance requirements for individual web pages (and in some cases, sets of web pages), but does not describe how to evaluate entire websites". Doesn't read well. Not clear what it is saying.

Resolution: Change as suggested.

Rationale: Clearer language.
18 Closed Gregg Vanderheiden, Samuel Martin Introduction "Core Functionality". "Dependent Components" or "Dependent Functionality" (with according changes to the definition). I understand the purpose of the term "core functionality" but its use bothers me very much, since it has been abused so completely in every other domain of accessibility. For the W3C to define or endose the term is extremely troubling. I would advise talking about "High Frequency pages" -- and "Pages needed to complete processes". And woud really speak against the use of the term CORE. It is not needed, and it is extraodinarily dangerous - both for web page evaluation and dangerous to accessibilty overall.

Resolution: Change the term to "Essential Functionality".

Rationale: Eval TF agrees with the concern but not with the terms "dependent functionality" or "dependent functionality".

See also comment 27 and Eval TF minutes of 12 December 2013 and of 15 January.
19 Closed Alistair Garrison Introduction "Web page states". "Web page DOM states". It is important to tell the reader that it is the DOM which is changing, after the page has loaded, not the web page (which could be construed as web page source code).

Resolution: Add "(changes to the Document Object Model - DOM)" to the sentence "Web pages with dynamic content can have different states".

Rationale: Avoids (further) complicating the term but make the term definition clearer.
20 Closed Howard Kramer Introduction List in "Purposes for this Methodology". Combine 5th and 6th bullet. Repetitive.

Resolution: Change as suggested.

Rationale: Mistake carried over from previous draft.
47 Closed Denis Boudreau Introduction "The activities carried out within these steps are influenced by many aspects such as [...] how much knowledge the evaluators have of how the website was or is being developed (e.g. white-box, black-box, or grey-box testing) [...]". I am wondering if the white-box, black-box, grey-box testing mention is relevant.

Resolution: Remove as suggested.

Rationale: Doesn't really add much.
48 Closed with ID11 Denis Boudreau Introduction "This methodology, the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0, describes the steps that are common to website conformance evaluation processes and highlights considerations that evaluators to apply these steps in the context of a particular website". "The Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0 describes the steps that are common to website conformance evaluation processes. It highlights considerations that evaluators need to take into account to apply these steps in the context of a particular website". sentence is long and weird.

Resolution: Change to "This methodology, describes the steps that are common to website conformance evaluation processes. It highlights considerations for evaluators to apply these steps in the context of a particular website.".

Rationale: Makes the sentence shorter and clearer.

See also comment 6, and comment 11.
49 Closed Denis Boudreau Introduction "Following this methodology will help evaluators to apply good practice, avoid commonly made mistakes, and achieve more comparable results". Remove the "to". -

Resolution: Change as suggested.

Rationale: Grammar.

See also Eval TF minutes of 9 January 2014.
50 Closed Samuel Martin Introduction Term titles in terms and definitions. Upper-case only the first word or lower-case both. Common orthographic conventions.

Resolution: Change as suggested.

Rationale: Grammar.

See also Eval TF minutes of 9 January 2014.
51 Closed Samuel Martin Introduction Web Page States. Web Page State (singular). -

Resolution: No change.

Rationale: The intent is not to define what an instance of state is but that a web page can have different states.
52 Closed Samuel Martin Introduction Ancilliary. Ancillary. Typo

Resolution: Change as suggested.

Rationale: Typo.

See also comment 15.

Comments on Using this Methodology

ID Status Commenter Location Current text Suggested Change Rationale Resolution
10 Closed Kerstin Probiesch, Detlev Fischer Using this Methodology Section "Review Teams (Optional)". Delete the entire section. That Review Teams are helping to identify barriers "more effectively" is at a first glance a logical statement, but because there are no systematic data which provide evidence it is just a thesis or an opinion. Two examples: a. is the evaluation of a Review Team with some months experience "more effectively" than the evaluation carried out by an individual evaluator with 5, 6, 7 years or more experience? b. what if a Review Team are testing their own interpretation: are the results of this Review Team better than the results of an individual evaluator who follows for examples decisions of the WCAG Working Group? As this is an opinion which is not substantiated by systematical data it can't be "highly recommended". When we speak about evaluation we of course speak about accessibility, but we also speak about business models and about a lot of money. Evaluators are in competition with one another Individual Evalators are in competition with Testing Organizations. WCAG-EM is for example explicitly mentioned in the "European Accessibility Requirements for Public Procurement of Products and Services in the ICT Domain" especially in this document: http://bit.ly/1e6VHXu (page 90, note 2). It is likely that the preference of Review Teams over Individual Evaluators will have a negative impact on the free competition. We should give guidance on how to evaluate web pages and should obstain from all (as long as they are not facts) what is likely to prefer one group of evaluators.

Resolution: Change "the use of review teams is highly recommended" to "the use of review teams is recommended" in the section but keep it.

Rationale: The referenced document reflects current W3C/WAI position.

See also Eval TF minutes of 12 December 2013.
21 Closed Richard Warren Using this Methodology "However, web accessibility evaluation tools significantly assist evaluators during the evaluation process and contribute to more effective evaluation". "Many checks are not automatable, however, web accessibility evaluation tools can significantly assist evaluators during the evaluation process and contribute to more effective evaluation". Re-arrange the text in the section "Evaluation Tools" to make it more clear that automated testing will never be sufficient on its own.

Resolution: Change as suggested.

Rationale: Better reflects the role of evaluation tools.
22 Closed Kathleen Anderson Using this Methodology Section "Involving Users (Optional)". Make non-optional Testing a website for accessibility should be given the same importance as testing a website for all other functionality, and no website should go live without user testing.

Resolution: No change.

Rationale: This goes beyond WCAG requirements. There are also no clear techniques and procedures for testing with users.
23 Closed Alistair Garrison Using this Methodology "evaluation of websites with WCAG 2.0". "evaluation of websites against WCAG 2.0". -

Resolution: Change to "evaluating websites using WCAG 2.0".

Rationale: We avoid using "against".
37 Closed David MacDonald, Kerstin Probiesch Using this Methodology "Involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluators alone". "Involving people with disabilities and people with aging-related impairments provides a clearer picture of how the site actually works for people with disabilities. It can result in a more rounded and useful assessment, and therefore better usability and overall accessibility of the site". Although I almost always include people with disabilities in evaluations, and they often identify things that can be improved on a web site's accessibility/usability, it rarely results in identifying strict WCAG failures that were not found in the "expert review" ... Let's leave evaluators out of this sentence.

Resolution: Change to "Involving people with disabilities including people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by expert evaluation alone".

Rationale: Previous comments reaffirmed the importance of involving users. It is something that can be further discussed and refined after publication as a working draft.

See also Eval TF minutes of 12 December 2013.
44 Closed Alistair Garrison Using this Methodology "Users of this methodology are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web". Change in-line list to an actual list. It makes it easier for people to read.

Resolution: No change.

Rationale: It is not really meant as a list.
54 Closed Kerstin Probiesch Using this Methodology "Users of this methodology are assumed to be knowledgeable of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web". "Users of this methodology are assumed to be knowledgeable of WCAG 2.0 which includes the Success Criteria, the role of techniques http://www.w3.org/TR/WCAG20-TECHS/intro.html) in general and the overall concept of WCAG 2.0, accessible web design, assistive technologies, and of how people with different disabilities use the Web". -

Resolution: Add "WCAG 2.0 Overview" document to the list in "Background Reading".

Rationale: That document better explains the different parts of WCAG 2.0.
68 Closed with ID37 Samuel Martin Using this Methodology "people with disabilities and people with aging-related impairments". Introduce "aging-related impairments" somewhere else, and clarify its relation to people with disabilities. I understand the reference to people with disabilities encompasses people with aging-related impairments, but that should be made clear to the reader

Resolution: Change to "people with disabilities, including people with aging-related impairments, [...]".

Rationale: This document is not the best place to explain these terms and their relationships. See ID37

Comments on Scope of Applicability

ID Status Commenter Location Current text Suggested Change Rationale Resolution
53 Closed Kathy Wahlbin Scope of Applicability "Note: Websites using responsive design techniques (i.e. adapting the presentation according to user hardware, software, and preferences) as opposed to redirecting the user to a different location are not considered to be independent website versions". "Note: Websites using responsive design techniques (i.e. adapting the presentation according to user hardware, software, and preferences) as opposed to redirecting the user to a different location are not considered to be independent website versions unless the site at the different breakpoints utilizes different code. In that case, the website at the different breakpoints could be considered as individual websites each for evaluation". Clarify what is needed for evaluating responsive websites.

Resolution: Add a section on "Websites using responsive design" under "Particular Types of Websites" to elaborate on the need to test the different adaptations and breakpoints using a variety of user agents (tablets, phones, etc.).

Rationale: More information about responsive design is needed but adding it to this particular note will not be as useful.

See also Eval TF minutes of 9 January 2014.
55 Closed Kerstin Probiesch Scope of Applicability "When a target website is defined for evaluation, it is essential that all web pages, web page states, and functionality within the scope of this definition are considered for evaluation". should be made clear that also PDF and other formats have to be considered for evaluation and that not only HTML is meant -

Resolution: Change the note in the definition of "web page" to read "Web pages may include multimedia content, interactive components, and rich and mobile web applications. Web pages are not limited to HTML and can be PDF documents and any other format."

Rationale: Explaining this point in the definition of the term "web page" has a more global impact throughout the document.
56 Closed Samuel Martin Scope of Applicability "Examples of Websites". Add another example corresponding to the website of an organization department. The case where the responsibility for content publishing of each department is distributed, is typical and relevant enough to be mentioned among the list of examples (even though is later dealt with in detail, in the discussion on websites with separable areas)

Resolution: No change.

Rationale: The reference to "organizational department" was removed because it was not immediately clear that it only applies to such cases where they are separable areas.
57 Closed Samuel Martin Scope of Applicability "This includes (...) forms for online payments, and discussion boards, including when such parts originate from third-party sources". Add this information outside the example as well. Make the information more explicit.

Resolution: Move the second paragraph of the example to outside the example.

Rationale: That would make the text part of the main section rather than of the example alone.

58 Closed Samuel Martin Scope of Applicability "Example of Website Enclosure" and "Particular Types of Websites" sections. Exchange the order of these two subsections. "Example of Website Enclosure" relies on the definition of "Website with separable areas" (included in "Particular Types of Websites").

Resolution: No change.

Rationale: The term "separable areas" is not used anywhere in "Example of Website Enclosure".
59 Closed Samuel Martin Scope of Applicability "Web applications will typically require (...) larger web page samples to reflect the different types of content, functionality, and processes." Check whether this is substantiated or just assumed. In my understanding, the amount of "different types of content, functionality and processes" does not depend on being a web application per se. On the contrary, if the same functions are offered through a static site and a dynamic web application, the same complexity should be expected. Besides, even some web applications are more homogeneous than static web sites, in that all the contents are generated from the same templates and scripts, and then a smaller sample is enough. Step 3 talks about "How interactive the content is", which is a much more objective foundation to determine that a sample should be larger, rather than just relying on the fact the functionality is provided through a web application.

Resolution: No change.

Rationale: This section refers to web applications that "tend to be more complex and interactive" such as "webmail clients, document editors, and online shops". These *typically* tend to require larger samples.
60 Closed Samuel Martin Scope of Applicability "are each independent of one another in usage, where using one version does not require or depend on using another version of the website" "are independent of one another in use, that is, using one version does not require or..." clarify the wording.

Resolution: Change as suggested.

Rationale: Better language.
61 Closed Samuel Martin Scope of Applicability "In this context evaluators often have easier access to the website developers and maintainers, the development environment and authoring tools, and the materials used for development." "... and the hosting platform where the website is deployed". Being able to "get into the kitchen" eases accessibility evaluation. Not only is it a matter of snooping around development processes, but it also facilitates access to later stages. E.g. self-assessment may ease accessing restricted areas (which is unrelated to accessing development).

Resolution: Change to "In this context evaluators often have easier access to the website developers and maintainers, the development and hosting environments, the authoring tools, and the materials used for development and maintenance."

Rationale: Clearer meaning.
62 Closed Samuel Martin Scope of Applicability "Third-party evaluators typically use pure black-box testing and … Therefore it is more difficult, often impossible, to make conformance claims based on such an evaluation alone." - First, black-box testing means 'no access to the internals of the software', instead of 'no access to the development environment (and authoring tools)'. Second, in the specific context of website evaluation accessibility, third-party evaluators usually do have access to a large part of the source code (e.g. HTML, CSS, Javascript), which even sometimes represents all the source code (e.g. in static websites, where there is no server-side scripting environment). Thus these tests are rather grey-box (or even white-box). Third, WCAG success criteria are defined from the perspective of the end-user, so black-box testing should not be in principle excluded as a way to test conformance. For instance, the evaluator does not need to know whether device-independent scripting events have been used -i.e. black box-, they just need to check the functionality is available from a keyboard -white box-.

Resolution: Change section to read "Third-party evaluators typically have little information about how a website was developed, its internal software, and all its areas and functionality. Therefore it is more difficult, often impossible, to make conformance claims for entire websites based on such an evaluation alone. However, such evaluation can be effective for validating conformance claims and statements about websites."

Rationale: Clearer meaning.

63 Closed Samuel Martin Scope of Applicability in "Example of Web site enclosure", it is said that the scope of evaluation includes "aggregated and embedded content (...) including when such parts originate from third-party sources". Later on, in "Particular Evaluation Contexts", it is said that "third-party content is not under the control of the website or web service providers (...) non-conforming content needs to be clearly identified in the web pages in which it appears". Distinguish between these two uses of "third-party content". One refers to content integrated from external sources, but which is part of the scope, so as to consider complete processes. The other refers to user-generated contents, which can be either repaired or excluded from the scope through a statement of partial conformance. The same term "third-party" is used for two different concepts that require two different approaches, this may confuse the readers.

Resolution: No change.

Rationale: Section "Principle of website enclosure" states that all third-party content in scope must be included in the evaluation while section "evaluating third-party content" explains how to go about with this type of content; the sections do not contradict.
64 Closed Samuel Martin Scope of Applicability Bullet list under "Re-Running Website Evaluation". Add links from each item to the relevant steps. In the previous working draft, the phrases "as per step x.x" appeared at the end of each item. They have been removed to make the structure clearer, but at the same time the whole section has been shifted and now appears before the methodology steps. With the current wording, "exemplar web page instances" are mentioned before they are explained in the methodology.

Resolution: Link to particular section 3.c

Rationale: In the previous text all links lead to the same section (step 3), which wasn't really useful. That is why we took the links out.
66 Closed Samuel Martin Scope of Applicability - Add an explanation of the security and privacy issues that may be involved when following this methodology. Security and privacy issues may appear when following this methodology, and evaluators should be aware of that, so as to take whatever measures (out of the scope of WCAG-EM) they consider. Different kinds of standards usually add a section on potential security risks. In any case, not sure if it fits in "Particular Evaluation Contexts", or they should be added directly to Step 2 and Step 5.b respectively.

Resolution: Add "Granting evaluators such access may require particular security and privacy precautions" to the corresponding note in Step 2, and a new "Note: Records of the evaluation specifics may include sensitive information such as internal code, passwords, and copies of data. They may need particular security and privacy precautions to secure." to section 5.b.

Rationale: Better to address directly in the corresponding sections. further not role of document but of evaluator and evaluation commissioner to point to legal issues besides W3C legal and copyright notices.

See also comment 41.
67 Closed Michael Cooper Scope of Applicability - Think it would be useful to add examples of what is *not* in scope, e.g., single pages, aggregated sites, etc. -

Resolution: No change.

Rationale: Both single pages and aggregated websites are in scope; the intent is to specifically address *any* type of website.

Comments on Step 1

ID Status Commenter Location Current text Suggested Change Rationale Resolution
24 Closed Sharron Rush Step 1 1b. Add "A clear definition of goals at this point will bring greater clarity to subsequent steps in the evaluation process. For example, clear evaluation goals are particularly relevant...etc". Suggest to change the wording of Step 1b to avoid the cognitive dissonance of suddenly being referred to a step that is two steps ahead of where we are

Resolution: No change.

Rationale: The link in Step 1b is the WCAG 2.0 definition for "conformance level", that readers are expected to be familiar with.
25 Closed Richard Warren Step 1 1d, 1e. - Steps 1d and 1e. I understand, and approve of, the concept but I am not sure if the wording is right and cannot think of how to improve it at this stage.

Resolution: No change.

Rationale: There will be more opportunity to further review and discussion after publication as a working draft.
26 Closed Gregg Vanderheiden, Kerstin Probiesch Step 1 "W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some evaluators might use other methods (inline with the requirements for custom techniques) to evaluate conformance to WCAG 2.0. W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria." "W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that ONE WAY TO MEET THE WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some AUTHORS might use other methods (IN LINE with the requirements for custom techniques) to CREATE conformance to WCAG 2.0 AND EVALUATORS SHOULD ACCEPT VIABLE ALTERNATIVE TECHNIQUES AS WELL." Techniques are not designed or provided for evaluation. They are provided as example ways to meet SC.

Resolution: No change.

Rationale: The text says that techniques can be used to HELP evaluation. This seems to be a correct statement. Also, we use the term "techniques" to refer to sufficient techniques, advisory techniques, and failures as currently used by WCAG WG.
69 Closed Samuel Martin Step 1 "Initial exploration of the target website during this stage may be necessary to [...]". "Initial exploration of the target website during this step may be necessary to [...]". Confusing when the word "stage" is used to refer a "step" within the document.

Resolution: Change as suggested.

Rationale: More precise.
72 Closed Samuel Martin Step 1 figure in the introduction of "Evaluation Procedure". make step 5 cover the rest of the steps in parallel. Step 5 happens in parallel to the others, instead of taking place after the others (as the figure is currently implying).

Resolution: We propose to keep this figure for the next PWD.

Rationale: More precise but also more complex. We will discuss this for the next version of the document. See the minutes of the 24 January 2014 EvalTF telco.
73 Closed, possibly open after test-run Samuel Martin Step 1 Use of the word baseline. Ask for public feedback on the concept of baseline as it is used in WCAG-EM. The concept of "baseline" has been historically subject to much contention during the development of WCAG 2.0, until it was finally dismissed. In WCAG-EM, it is used in a different context, but with several aspects in common. Risks exist, such as evaluating against an artificially restricted subset of user agents that support the accessibility features. Later on, in Step 4, it is not clear enough whether all the user agents (including assistive technologies) included in the chosen baseline should be employed to audit websites. I understand the concept may be necessary here, but it should be treated with caution.

Resolution: Check explicitly with WCAG WG.

Rationale: No issues raised so far.
74 Closed Samuel Martin Step 1 Bullet list in step 1.e. Add "Adherence to specific documentation or reporting templates". Additional requirements may also be targeting non-functional process constraints. One of the most typical examples is the integration of accessibility evaluation within a larger (e.g. corporate QA) process, whose documentation system must be used.

Resolution: Change as suggested.

Rationale: Better coverage.

Comments on Step 2

ID Status Commenter Location Current text Suggested Change Rationale Resolution
27 Closed Sharron Rush, Gregg Vanderheiden Step 2 2b. - Step 2b"common functionality" is particularly jarring here. Common to what? Perhaps the phrase can be replaced with "critical functionality," although I do not understand why "basic" is not acceptable.

Resolution: Change to "essential functionality" as per comment 18.

Rationale: Term changed; ("common functionality" was typo that should have read "core functionality").
70 Closed Samuel Martin Step 2 "The initial exploration carried out in this step is typically refined in the later stages [...]". "The initial exploration carried out in this step is typically refined in the later steps [...]". Confusing when the word "stage" is used to refer a "step" within the document.

Resolution: Change to "The initial exploration carried out during this step is typically refined in the later [...]".

Rationale: More precise.
71 Closed Samuel Martin Step 2 "Carrying out initial cursory checks during this stage helps identify [...]". "Carrying out initial cursory checks during this step helps identify [...]". Confusing when the word "stage" is used to refer a "step" within the document.

Resolution: Change as suggested.

Rationale: More precise.
75 Closed Kathy Wahlbin Step 2 Steps 2.a and 2.c. Remove references to "web page states" from Step 2.a. Both of these two section talk about web page states. Remove this from step 2a since it is detailed in step 2c. This would simplify the steps.

Resolution: Change "including web page states" to "which may be web page states" in "methodology requirement 2.a" and remove "and web page states" from the last sentence in the section.

Rationale: More precise.
76 Closed Kathy Wahlbin Step 2 "identfy its common web pages". "identify its common web pages". Typo.

Resolution: Change as proposed.

Rationale: Typo.
77 Closed Kerstin Probiesch Step 2 "Web pages using varying technologies such as HTML, CSS, JavaScript and WAI-ARIA;". "Web pages using varying technologies and formats such as HTML, CSS, JavaScript, WAI-ARIA, PDF etc.". -

Resolution: Change to "Web pages using varying technologies such as HTML, CSS, JavaScript, WAI-ARIA, PDF, etc.".

Rationale: Accepted with minor tweak.
90 Closed Samuel Martin Step 2 The last paragraph has been removed since the previous working draft "This step also helps understand key aspects of the website, such as the navigation and overall structure of the website." Re-add it as a "Note". Identifying common web pages has a value by itself, but it is also helpful to gain an overall vision of the website organization at this early stage of the evaluation.

Resolution: No change.

Rationale: The entire section (step 2) serves this purpose thus the opening sentence "During this step the evaluator explores the target website to be evaluated, to develop an initial understanding of the website and its use, purpose, and functionality".
91 Closed Samuel Martin Step 2 "Identify an initial list of core functionality". "... core functionalities". grammatical agreement ("list of" + plural).

Resolution: No change.

Rationale: Functionality is plural form.
92 Closed Samuel Martin Step 2 "it might be less easy to identify that it also has a currency conversion function that is core to the particular context of the online shop". Replace the currency converter example, maybe with "... that it also has a personalization function to customize options such as currency and country". Or just choose another functionality for the example (maybe my specific suggestion would also fall under step 2.e). A currency converter as that one would not be presented as "core" in that context. Indeed, there is a bullet list that develops the example within the same step, but the currency converter is not even listed there among the core functionalities.

Resolution: Change "For example, an online shop is expected to have a payment function though it might be less easy to identify that it also has a currency conversion function that is core to the particular context of the online shop" to read "For example, it may be easier to identify the functionality for purchasing products in an online shop than the functionality provided for vendors to sell products through the shop".

Rationale: Better example.

See also comment 18.
93 Closed Samuel Martin Step 2 Bullet list in Step 2.b. Highlight as example as done in other sections. To differentiate from other bullet lists.

Resolution: No change

Rationale:This would attract much more attention to the list than is the intention. We do think however that an example may be helpful here.
94 Closed Samuel Martin Step 2 "Web pages that are created using different templates and by different authors (if this is known to the evaluator);". Split into two items: "Web pages that are created using different templates (if this is known to the evaluator);" and "Web pages that are authored by different people (if this is known to the evaluator);" plus add "Web pages that are created using different coding styles". These are two different points and concepts. Sometimes it will be possible to distinguish according to one criteria but not the other, or vice versa. In addition, variety of coding styles should also be mentioned, to be consistent with the list later presented in step 3. The variety of coding styles is always discernible by the evaluators, whether they know the reason (e.g. different authors) or not.

Resolution: Change as proposed.

Rationale: More clarity.
95 Closed Samuel Martin Step 2 "auxiliary web technologies such as Java, JavaScript and WAI-ARIA". "auxiliary web technologies such as JavaScript and WAI-ARIA". as of December 2014, it does not seem client-side Java is used enough as an auxiliary web technology to deserve being mentioned in such a short list.

Resolution: Change as proposed.

Rationale: More realistic.
96 Closed Samuel Martin Step 2 "as well as specific web technologies such as SMIL and SVG". add PDF. relevance of PDF; plus consistency with the introduction (where HTML and PDF are the only examples of technologies mentioned).

Resolution: Change as proposed.

Rationale: More accuarate.
97 Closed in ID96 Samuel Martin Step 2 "as well as specific web technologies such as SMIL and SVG". add PDF. relevance of PDF; plus consistency with the introduction (where HTML and PDF are the only examples of technologies mentioned).

Resolution: Change as proposed.

Rationale: More accuarate. Closed in ID96
98 Closed Samuel Martin Step 2 "Some websites include web pages and web page states that are specifically relevant for people with disabilities and accessibility of the website. (...) This includes: (...) Particularly popular web pages and those that are commonly used as entry points to the website;". either change the first sentence, so that it does not specifically refer to people with disabilities, or move the mentioned bullet item to its own step 2.f. The presented example (popular web pages and common entry points) is neither "specifically relevant for people with disabilities" nor included in "some websites". On the contrary, they are relevant for everybody, and they are included in any website (each will have their own popular pages and common entry points, but all will have some). It makes perfect sense that they are included in the sample, but they do not fit under the current heading. Another option is to move them to a different step.

Resolution: Remove bullet item.

Rationale: Adding too much confusion for quite little value.

Comments on Step 3

ID Status Commenter Location Current text Suggested Change Rationale Resolution
28 Closed David MacDonald, Kerstin Probiesch Step 3 - Calculate sample size based on size of the website. These are established international statistical sample sizes. Then with that baseline we can talk about increasing (or decreasing) the sample size based on the other factors of complexity, age, consistency etc...

Resolution: Add editor note and ask for more input on this issue.

Rationale: The suggested sample sizes are based on uninformed sampling rather than informed structured sampling. Moreover, it is unclear how to calculate the "size of a website" especially for dynamic applications with few "web pages" but unforeseen number of "web page states".

See also mailing list discussion on sample size and Eval TF minutes of 12 December 2013.
29 Closed David MacDonald, Kerstin Probiesch Step 3 - I think we should have an added section in the different types samples 3f templates. Choose a page using each type of template -

Resolution: No change.

Rationale: Step 3c refers to the web page types, including templates, identified in Step 2c. Previous comments indicated that too much emphasis on templates is not desired.
30 Closed Richard Warren Step 3 - - The implication is now that ALL websites require selecting a representative sample. This is not so. Ideally all pages should be evaluated. If, and only if, the site is too large for a full evaluation to be viable should we need to select a representative sample.

Resolution: No change.

Rationale: The first paragraph already says "In cases where it is feasible to evaluate all web pages, this sampling procedure can be skipped and the selected sample is considered to be the entire website in the remaining steps of the conformance evaluation procedure". This sentence is already the result of much discussion.

See also Eval TF minutes of 12 December 2013.
31 Closed Richard Warren Step 3 - - Also I am not happy with the step 3e (select a random sample). There have been many discussions about this and no definitive conclusion. I suggest that we leave the method of selecting the random sample up to the evaluator - who MUST record his/her method in step 5 so that it can be replicated.

Resolution: No change.

Rationale: There is no required method for selecting a random sample. Step 5 requires evaluators to record the method used for random sampling.

See also Eval TF minutes of 12 December 2013.
32 Closed Detlev Fischer Step 3 "Include all common web pages and web page states that were identified in Step 2.a: Identify Common Web Pages of the Website into the selected sample for evaluation". "Include all common web pages and web page states that were identified in Step 2.a: Identify Common Web Pages of the Website into the selected sample for evaluation. Web page states and the method to call them up can be recorded together with the base page. Alternatively, web page states can constitute a separate page". I think we should make it clear to evaluators that they will often need to call up additional states to be evaluated, and the idea of replicability requires that the way to call up states is documented. I am not happy with my own wording though, so maybe someone else has a better suggestion?

Resolution: No change.

Rationale: Aspects of recording are already addressed in Step 5 rather than in the individual sections to avoid complexity and repetition.

33 Closed Gregg Vanderheiden Step 3 "Core functionality" "Dependent components" Step 2b"common functionality" is particularly jarring here. Common to what? Perhaps the phrase can be replaced with "critical functionality," although I do not understand why "basic" is not acceptable.

Resolution: Change to "essential functionality" as per comment 18.

Rationale: Term changed.

See also Eval TF minutes of 12 December 2013.
34 Closed Gregg Vanderheiden Step 3 3d Put this step AFTER the random selection step: "IF ANY OF THE ABOVE PROCESSES HAVE PUT A PAGE INTO YOUR EVALUATION SAMPLE, WHERE THAT PAGE IS PART OF A PROCESS, THEN ALL OF THE PAGES INVOLVED IN THAT PROCESS MUST ALSO BE ADDED TO THE EVALUATION SAMPLE." In step 3d you don't actually say anywhere that ALL processes should be included in the sample. Just that if you select one page in a process- you must include the whole process. Maybe do DON'T WANT to say that all processes are included. (some site may have an almost endless number of them).

Resolution: No change.

Rationale: Indeed, previous comments indicated that including all processes is not desired and often impossible. Also including entire processes for the random sample bloats it unnecessarily; it is only desired for the structured sample.
78 Closed David MacDonald, Gregg Vanderheiden, Samuel Martin Step 3 distinctinstance - Typo

Resolution: Change to "distinct instance".

Rationale: Typo.
79 Closed David MacDonald Step 3 Constistent - Typo

Resolution: Change to "Consistent".

Rationale: Typo.
80 Closed David MacDonald Step 3 Nethods - Typo

Resolution: Change to "Methods".

Rationale: Typo.
99 Closed Samuel Martin Step 3 typcially typically Typo

Resolution: Change as suggested.

Rationale: Typo.
100 Closed Samuel Martin Step 3 "websites with content that is rich in functionality require larger samples to cover the different tasks that can be carried out on a website and the different states that individual web pages can have;" "websites with content that is rich in interaction require larger samples to cover the functions provided by a website and the different states that individual web pages can have;" what is relevant for rich web applications is that they have rich interactivity, not whether they provide complex or simpler functions. Likewise, it is not a matter of being able to carry out many tasks (this is more related to the size of the site), but a matter of covering the whole site (same as in a static site).

Resolution: Change as suggested.

Rationale: More accurate.
101 Closed Samuel Martin Step 3 "content that is aggregated from different sources or that is processed during runtime". clarify the meaning of "during runtime". It is not clear whether it means the content is processed (on the server) upon user accessed, or whether it is processed (on either the server or the client) during/after user access.

Resolution: Change to "processed as it is served (at runtime)".

Rationale: More accurate.
102 Closed Samuel Martin Step 3 "websites that are available in different versions, are served according to users and their preferences, and adapt to access devices require larger samples to cover these different situations;". "... or adapt to ...". express alternatives, not restrictions.

Resolution: Change as suggested.

Rationale: More accurate.
103 Closed Samuel Martin Step 3 "instance(s)" of web pages (three times within the step). Consider defining an "instance" above. in Step 2.c, it is said that "The outcome of this step is a list of the different types (as opposed to specific instances) of web pages and web page states", but then in Step 3.c. it is not clear what "exemplar instances" refer to.

Resolution: No change.

Rationale: Seems clear from context and there are several notes already to explain it more clearly.

Comments on Step 4

ID Status Commenter Location Current text Suggested Change Rationale Resolution
35 Closed Michael Elledge Step 4 - Several grammar edits. Grammar.

Resolution: Change as suggested.

Rationale: Grammar.
36 Closed David MacDonald, Kerstin Probiesch, Loretta Guarino Reid, Gregg Vanderheiden Step 4 In such cases, an evaluator may use an identifier such as "not applicable" to denote the particular situations where Success Criteria are satisfied because no matching content is presented. "not present", "No matching content"? "satisfied by omission"? "automatically satisfied"? The working group went to great lengths to keep that term away from the evaluation process. ALL SC apply to a website. If the site does not have MEDIA then all of the pages meet the SC. The SC are not NA. Again, once evaluators start labelling things NA -- all sorts of other reasons are used for the designation.

Resolution: Change to "not present".

Rationale: More accurate.

See also Eval TF minutes of 19 December 2013.
38 Closed Gregg Vanderheiden Step 4 Step 4.e "Methodology Requirement 4.e: Check that each web page and web page state in the randomly selected sample does not show types of content and outcomes that are not represented in the structured sample.
The purpose of this step is to ensure that the overall sample includes all of the page types. This is done by comparing the structured and random sample to see there are new types of content (as identified in Step 2: Explore the Target Website) found in the random sample than in the structured sample. If there are then the structured sample should be expanded and a new random sample taken until the random sample produces no new types of content. At this point it is a fair assumption that the sample is representative of the website (absent any other knowledg to the contrary)."		 Do not use the word "correlate" (rewording suggested in the survey); correlation not achievable.

Resolution: Take the following sentence out: "The evaluation outcomes of the structured and random sample correlate when they are sufficiently large and representative. "

Rationale: Agree with the point made regarding the formal meaning of correlation.
81 Closed Kathy Walhbin Step 4 - Add a new step to include information on how to use assistive technology in testing. This step could be marked as optional. In section 1.c, we ask people to define the web browser, assistive technologies and other user agents for which features provided on the website are to be accessibility supported but then we do not have language about this in the auditing of the selected sample. I feel that this should be included.

Resolution: Change "Check that all features are supported by the baseline defined in Step 1.c: Define an Accessibility Support Baseline" to read "Check that all features are supported by the web browsers and assistive technologies defined in Step 1.c: Define an Accessibility Support Baseline".

Rationale: Better clarification of the role of user agents.

See also Eval TF minutes of 19 December 2013.

Comments on Step 5

ID Status Commenter Location Current text Suggested Change Rationale Resolution
39 Closed David MacDonald, Kerstin Probiesch, Alistair Garrison Step 5 "Conformance level satisfied: Level A, AA or AAA as per Step 1.b. Define the Conformance Target". "Conformance Level (Level A, AA, AAA) with a fair degree of confidence" or "Conformance level evaluated". I don't think an organization can claim absolute WCAG conformance based on this methodology, as this phrase appears to indicate. At least not as it is defined currently in WCAG which requires EVERY page to conform. I think it might expose them to legal action.

Resolution: Change to "Conformance level evaluated: Level A, AA or AAA as per Step 1.b. Define the Conformance Target".

Rationale: More accurate reflection of the intended meaning.
40 Closed Kerstin Probiesch Step 5 "Name of the evaluator (responsible person, organization, or other entity)". "Names of the evaluators (responsible person, members of the review teams, organization, or other entity)". Transparency (especially because WCAG-EM is part of "European Accessibility Requirements for Public Procurement of Products and Services in the ICT Domain")

Resolution: Link word "evaluator" to its definition and remove explanation here. Also for evaluation commissioner

Rationale: The fact that an evaluator (keyword) can be a team of people is already reflected in the definition and does not need to be repeated here.
41 Closed Alan Smith Step 5 Step 5.b "Specific test credentials (useids, etc.) required to replicate a unique data set or workflow". Often data sets are unigue to a particular userid and drive a web page design, especially in financial institution web pages/apps. Since we do have a bullet for Description of settings...Perhaps "test credentials" wording would be best served by putting it in that bullet?

Resolution: Change as suggested.

Rationale: Important addition.

See also comment 66.
42 Closed Detlev Fischer, Samuel Martin Step 5 Per Instance Score Drop this and allow other options for scoring. As discussed in Eval TF minutes of 5 December 2013.

Resolution: Change as proposed.

Rationale: Was previously agreed on but not reflected in the Editor Draft.
43 Closed Gregg Vanderheiden, Detlev Fischer, Kerstin Probiesch, Samuel Martin Step 5 5d Drop entire section. Scoring has always proven to be problemmatic and misleading when actually tried -- and unless and until you have data showing that it actaully works based on evaluation of diverse sites -- we should not have this in the document. THis is alway seductively attractive but ive never seen it work.

Resolution: No change; review note added for further feedback and discussion.

Rationale: Comments so far have been both in favor and against scoring. It is unclear that the use is harmful when appropriately contextualized as in this document. We have changed from conformance score to aggregation score and limited the calculation to website, dropping page and instance.
82 Closed David MacDonald Step 5 Currently the following performance scoring approaches are provided by this methodology. Currently, the following performance scoring approaches are provided by this methodology. Grammar.

Resolution: Changes made.

Rationale: Section rewritten for clarity.
83 Closed Samuel Martin Step 5 - Re-add "The type of scoring system used has to be indicated along with the score whenever such a score is provided". The measure of a magnitude needs the units into which it is expressed so as to make sense; and the measurement procedure followed so as to be reproducible.

Resolution: Re-added.

Rationale: More precise.
84 Closed Samuel Martin Step 5 "and divide this by the sum of all applicable WCAG 2.0 Success Criteria across all web pages and web page states in the selected sample." Make the meaning of the sentence more accurate. The first-time reader may interpret that "the sum of criteria across all web pages" means the number of applicable criteria, not that number multiplied (I mean, roughly) by the number of web pages.

Resolution: Changes made.

Rationale: Section rewritten for clarity.
89 Closed Samuel Martin Step 5 Performance scores Only count success criteria reflected in the content. Less skew of the data.

Resolution: Changes made.

Rationale: Counting only success criteria to which there is matching content.
86 Closed Samuel Martin Step 5 "Performance scores are not intended for inter-site comparisons." add strong emphasis to the sentence. Even though the document states these are just the approaches provided by the methodology (thus, WCAG-EM does not claim this is the ultimate methodology), Eval TF should be aware that the scoring proposed herein may soon become the "default" scoring system, as the public will recognize it as sanctioned by the W3C WAI. Thus, the goal of the performance score should be made clear-cut. Otherwise, we risk everybody will start establishing site rankings based on the WCAG-EM performance score (what they will do, anyway).

Resolution: Text removed.

Rationale: Section rewritten for clarity.
87 Closed Samuel Martin Step 5 Performance scores Several suggestions to improve the wording (comments #37-44 in Samuel Martin's comments. Confusing instructions and terminology (especially "applicable Success Criteria").

Resolution: Changes made.

Rationale: Section rewritten for clarity.
88 Closed Samuel Martin Step 5 "The outcomes of Step 4: Audit the Selected Sample can be aggregated over the entire sample". Add before "Depending on the desired granularity of the report documentation, ...". Make it clear aggregation is not a mere choice, but it responds to more general requirements of the evaluation process..

Resolution: Changes made.

Rationale: Section rewritten for clarity.