Disposition of Comments

Replace "one possible approach to evaluate websites" with something that gives it more credibility and more strongly recommends it, e.g., "one established approach".

"See brainstorms in [approach - EOWG minutes 5 April]."

Resolution: Reworded sentence.

Rationale: Balance between sounding exclusive and too tentative.

Replace "However, this methodology does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance." with: "This methodology does not replace the need for quality assurance measures implemented throughout design, development, and maintenance."

Resolution: Sentence removed.

Rationale: Section has been rewritten.

Change the term "Common Functionality"

"See brainstorms in [approach - EOWG minutes 5 April]."

Resolution: Replace "Common Functionality" with "Essential Functionality"

Rationale: Latest decision in comment 18 on 29 November 2013 Editor Draft.

Common functionality

• "Critical functionality (taken from note 2)
• Fundamental functionality (taken from the definition)
• Purpose-defining functionality (again, based on the definition)"

"Given the definition (functionality that if removes fundamentally changes the purpose of the website), the term 'common' is not strong enough. I believe that the term should emphasize the essential or fundamental character of this functionality that is, by definition, more important than non-fundamental functionality."

Resolution: See ID3

Rationale: See ID3

"As I have suggested alternatives for "common functionality" (such as "critical functionality", "fundamental functionality" or "purpose-defining functionality" this step should be modified to be consistent with the new term."

Resolution: See ID3

Rationale: See ID3

Common Functionality

Change "Common Functionality" in "Core Functionality".

"It better fits the description: "all functionality of a website that, if removed, fundamentally changes the use or purpose of the website for users""

Resolution: See ID3

Rationale: See ID3

'Complete processes' should be singular: 'complete process'.

Suggestion for the definition: "Complete process: sequence of steps that need to be completed in order to accomplish an activity."

"The text is not a definition as it just quotes text from WCAG 2.0."

Resolution: No change.

Rationale: The term and the definition are indeed directly taken from WCAG.

Common web pages

Change 'common web pages' in "general-purpose web pages" and the definition in "web pages whose purpose is general and not specific to the website they belong".

"I think that this definition (web pages that are relevant to the entire website) lacks precision. I think that the concept that defines these pages is that their underlying functionality is not specific for a given website, but is general to many web sites. But I am not completely sure about this."

Resolution: No change.

Rationale: See also earlier comment on september 2012 Public Working Draft. The commenter there indicates: "The definition does not follow the usual definitions of the meaning for "common".

We define "common" here as relevant to the entire website and typically linked from all web pages. The definition then points to some examples of web pages that are widely available on websites like the homepage, 404 page etc. "General purpose.." seems to indicate more action while some of the web pages in the example can be just pages with text that are very common. "Core" is to heavy and also the term is used earlier for Core Functionality which could cause confusion.

"Given their relevance to WCAG-EM, I suggest adding two definitions from WCAG that are essential to the process of conformance evaluation: "conformance" and "satisfies a success criterion"."

"Satisfies a success criterion means that a success criterion is satisfied (passes) unless it fails and to me has great implications in the evaluation of accessibility: a success criterion passes for a page unless it is demonstrated that it fails."

"Conformance: satisfying all the requirements of a given standard, guideline or specification."

"Satisfies a success criterion: the success criterion does not evaluate to 'false' when applied to the page."

Resolution: Added term conformance from WCAG 2.0:

"From WCAG 2.0 definition for "conformance":
"satisfying all the requirements of a given standard, guideline or specification"."

Rationale: Adding these terms does not seem necessary for a good understanding of the text however the term "conformance" is important in the context and used many times. The term Satisfies a success criterion is not literally used in WCAG-EM.

"This is a vague notion b/c it relies on “process” which is itself not defined, and difficult to define".

"I would try to redefine it in terms of goals that users try or may want to try to achieve. All pages (from a starting point) that have to be visited so that a certain goal, out of a given list of goals/use cases, have to be audited."

Resolution: No change to the definition but further explanations in Step 3.d.

Rationale: The definition is directly from WCAG 2.0. The goal seems to be covered by "activity" in the sentence: "a sequence of steps that need to be completed in order to accomplish an activity". Further clarification is provided in the corresponding section.

Closed

"The auditor has not enough info regarding which templates were used, if any."

Resolution: No change

Rationale: This is one of the reasons why evaluating Templates is optional (optionality specified in Note in section 4.a).

We also added a Note to the definition: "Note: Involvement of the website owner and/or website developer can be useful to help identify common web pages, functionality, technologies, and other aspects of the implementation that makes the evaluation procedure more efficient and effective. However, the evaluator is responsible for an objective and thorough evaluation.".

"self enclosed? the web is by definition not self enclosed"

Resolution: No change

Rationale: The definition of a website is explained in more detail later on, in particular in Methodology Requirement 1.a.

"This does not say anything of what can be inside this container. For something like gmail, I need to test many different states in which the “page” can be. How does the methodology help me decide which states to consider?"

Resolution: No change to the definition but extensive explanations on web page states have been added throughout the document.

Rationale: We used this definition from WCAG. The web page states are further addressed throughout the document.

"Format it as a list."

Resolution: Change made.

Rationale: This section has been rewritten.

"Re-write the list to be more clear, and to focus on the most common uses first. fyi, see Who WCAG-EM is for in Overview page."

"As written, the repetition of words makes it quite difficult to parse. Also, we are concerned about having "website developers" as the first item. See: Shadi (for discussion in EOWG f2f before CSUN)."

Resolution: Change made.

Rationale: This section has been rewritten.

"Delete the sentence starting with "Complementary resources on ..." and add those resources to the next section "Background Reading"."

Resolution: Change made.

Rationale: This section has been rewritten.

"Unlink the headings and add those links in the lists. "

linked headings discussion - EO meeting minutes of 4/5

Resolution: Change made.

Rationale: This section has been rewritten.

Replace the words "need" and "needs".

"Probably replacing with "should" would be simplest; however, we understand there issues with that word. See brainstorms in needs - EOWG minutes 5 April."

Resolution: Removed as many instances of "need" as possible.

Rationale: Need to avoid the term "should" as it is part of formal (normative) language.

"From the second sentence, remove the words "Later on"."

later on discussion from EO meeting 5 April

Resolution: Sentence removed.

Rationale: Section has been rewritten.

"Tighten it up."

"It should be a short summary. Best if the whole this is not exactly the same wording as later in doc."

Resolution: Edited

Rationale: Section has been rewritten.

"This methodology is a thorough, robust approach for evaluating the conformance of websites to WCAG 2.0. Before applying this methodology to a website, it is usually good to do a preliminary evaluation to identify obvious accessibility barriers and develop a rough understanding of the overall performance of the website. Easy Checks - A First Review of Web Accessibility describes a simple, quick approach to show you if a web page has obvious accessibility barriers."

Reword to be more positive and be in-line with Easy Checks (which will replace the old Preliminary Eval page).

For example, something like: This methodology is a thorough, robust approach for evaluating conformance to WCAG 2.0. Before applying this methodology, it is usually best to do a preliminary evaluation to identify obvious accessibility barriers and get a general idea of the level of accessibility of a website. A simple, quick approach for getting a general idea if a web page has accessibility barriers is provided in Easy Checks - A First Review of Web Accessibility.

Resolution: Change made.

Rationale: Section has been rewritten.

"Needs clarification"

Resolution: Change made.

Rationale: Section has been rewritten.

"Examples could be: visiting a museum virtually; a small site dealing with a specific product developped by a company; collaborative area for the inhabitants of a city (with possibility to exchange things, look for baby sitters or someone who could work in the garden: this means third-party services)."

"A site with a specific topic considered as a subdomain. Areas with advertisement, areas with RSS feeds and pop-up windows, iframes. Specific content in Flash, applet, widget, chatting room."

Resolution: Examples have been considered.

Rationale: Section has been rewritten.

Proposal text: "In some cases websites may have clearly separable areas where using one area does not require or depend on using another area of the website. An example could be a service company website with three areas: a public area for information purposes, a customer area for providing services to existing customers and an employee area for providing services to people working in the company. Such areas can be considered as individual websites rather than sub-sites for the purpose of this document."

Resolution: Change made.

Rationale: Section has been rewritten.

Web Applications section

"Suggested abridged version: Websites with a limited number of pages which dynamically generate content are considered as a single website (or part of a larger site). Each state of a web page and content-generated web page for the purpose of this document should be included as a candidate for sampling."

"Rewrite to be more clear."

Resolution: Change made.

Rationale: Section has been rewritten.

Remove the word "Optional".

"I don't understand why this section is labelled as "optional". It just describes the relationship between the methodology and evaluation tools."

Resolution: No change

Rationale: In previous versions we decided to make more clear what parts of the document are necessary an what parts are optional. It is not necessary to use evaluation tools, review teams and to involve users. That is why these sections are optional. Previously (without "optional") some readers interpreted these sections as obligatory causing potential confusion.

Remove the word "Optional" and deleting the third sentence ("While not required, it is recommended to employ review teams for conformance evaluation of websites.").

"I don't understand why this section is labelled as "optional". This section "Using this methodology" does not seem to me a place to define "requirements" for users of the methodology (and in fact the previous sections - scope of applicability, required expertise, evaluation tools - don't require anything). The "methodology requirements" do really start in the next level-1 section ("Conformance Evaluation Procedure")."

Resolution: No change

Rationale: See ID 26

Remove the word "Optional" and deleting the third sentence ("While not required, it is recommended to employ review teams for conformance evaluation of websites.").

"I don't understand why this section is labelled as "optional"."

Resolution: No change

Rationale: See ID 26

Closed

Use the word "stage" for the high-level elements (1, 2, ..) and "steps" for the inner levels (1a, 1b,...).

"One issue with terminology: I think there is lack of consistence while using the words "stages" and "steps"."

Resolution: Change to step.

Rationale: The idea was indeed to use "stage" for the high level elements (1, 2, 3,..etc.) and "steps" for the inner levels (1.a, 1.b, etc.). During the 11 July telco the group decided that steps is more consistent.

"The diagram depicts each of the five steps defined in this section with arrows back and forth between each of two consecutive steps: 1. Define the Evaluation Scope; 2. Explore the Target Website; 3. Select a Representative Sample; 4. Audit the Selected Sample and 5. Report the Evaluation Findings."

The description has to be changed to: "The diagram depicts each of the five steps defined in this section with arrows forth between each of two consecutive steps: [...]"

"The description of the diagram (paragraph below the diagram) seems to be incomplete. It seems to me that there are back arrows from each step to any previous step: that is, for instance, from step 5 to steps 4, 3, 2 and 1. Am I right interpreting the diagram? There are also arrows back from each step to any of its previous steps."

Resolution: Description improved.

Rationale: Yes, this is indeed true. The description is missing the description of some arrows. The above description should be better.

"In a usability report one usually
starts by saying which method was used. Here you are assuming that there are more than 1 method, but you really do not list/present them."

Resolution: Clarifications made.

Rationale: The entire document describes a singe methodology, though different routes can be taken from start to end. Documentation of the route/steps happens throughout.

See ID 30

"The paragraph describing the flow diagram has visual dependencies. Change to: "The workflow diagram depicts each of the steps defined in this section. Work flow starts with the first step and proceeds to the last step in order. In addition, the diagram permits evaluators to return to any preceding step in the process whenever the evaluation reveals a need to rework a sequence of steps.""

"Provide functional description, instead of simply visual." "The new description is a functional description of the picture. "

Resolution: See ID 30

Rationale: See ID 30

This scope definition may not contradict the terms established in section Scope of Applicability.

Replace the word "may".

"The wording of this sentence is strange. I understand that WCAG-EM cannot use normative language (shall, should), but the use of "may" doesn't work for me."

Resolution: The term "may" has been removed from this section and throughout as much as possible.

Rationale: Better clarity.

This scope definition may not contradict the terms established in section Scope of Applicability.

Replace the word "may".

"EOWG sees the need for the sentence. since some found it confusing, please try to make it more clear. Replacing "may" with "should" makes this clearer."

Resolution: See ID 33

Rationale: See ID 33

"This looks to me useless. Why is a list of page titles and their URI’s not sufficient?"

Resolution: No change.

Rationale: A list of URIs is not always possible or enough to describe the pages or states of pages that are within the scope of the evaluation.

"The terminology is confusing. The name of the step is "define the goal of the evaluation", whereas the content talks about "types of evaluation". These two should be aligned. Either the title of the step is changed into "Define the type of evaluation" or the content of the step is changed to describe different goals instead of evaluation types."

Note: section has changed to 1.d

Resolution: Section removed.

Rationale: Proposed change to this section as discussed on mailing list and in telco 18 July to be more flexible and open for additional requirements from the evaluation commissioner(s). This section now presumes that the minimum requirements for reporting are set in Step 5.

"Certainly relevant would be:

A. Legacy site evaluation (old site known not to conform)
B. Development support evaluation (new accessible site under development)
C. Conformance evaluation (site that has been designed to conform or where a conformance claim has been made by the site owner)"

Resolution: See ID36

Rationale: See ID36

Relocate Step 1d to Step 2e.

"I strongly believe that this step does not belong to this stage (1. Define evaluation scope) but to the next one (2. Explore the target website). More precisely, I think it should be renumbered as step 2e. The reason is that, to me, one can only define techniques and failures once the functionality, the types of web pages and the technologies relied upon are known."

"If this step is moved to become 2e, then the last paragraph should be revisited and changed accordingly. For instance, the second sentence ("This definition...") refers to the exploration stage."

Note: section has changed to 1.c

Resolution: Clarifications made.

Rationale: The different steps related to this (Step 1, Step 2, and Step 4) are better clarified. Also the fact that this is usually iteratively established.

"However, it is not necessary to use these particular techniques. In fact, in some situations, such as in a closed network, it may be necessary to use techniques that are specifically developed to the particular needs of the users of that network. Individuals and organizations developing techniques have to employ methods for establishing the technique's ability to meet the WCAG 2.0 Success Criteria."

"Wouldn't it be useful to provide one or two examples of this? So it makes more sense to those reading this document?"

Note: section has changed to 1.c

Resolution: Section has been rewritten.

Rationale: Understanding WCAG 2.0 provides clarification that is now refered to.

"In addition to providing examples, this should not be an optional part of the methodology. It seems critical for the reviewer to define what techniques are being used to measure conformance or nonconformance to the SC. The section could make note of the fact that reviewers are not bound by previously established W3C Techniques and Failures, but it should still require that some documentation is needed of just what Techniques are being used. Could also encourage the addition of those new Techniques to the existing library."

"Without techniques being defined, the review is in danger of becoming undocumented opinion rather than measurement of conformance to a standard."

Note: section has changed to 1.c

Resolution: No change.

Rationale: Techniques and failures in the context of WCAG 2.0 are informative and not required for determining conformance with WCAG 2.0. This is why this step is optional. Also see earlier discussion about optionality in the DoC for the september 2012 Working Draft.

Resolution: Thank you for your feedback.

Rationale: Confirmation of approach.

Avoid the use of "relied upon" and use the term "technologies used" instead.

"There is a problem with the use of "relied upon" in this step. According to the definition of technologies that are "relied upon" what it means is that the content would not conform WCAG if those technologies were not supported. But the description of step 2.d talks about "technologies relied upon to provide the website", which is a different concept. My suggestion is either to avoid the use of "relied upon" in this step or to use it related to WCAG conformance and not to the overall website. My preferred option is the first one: to avoid the use of "relied upon" and use the term "technologies used" instead.If the second option is selected then this step would require two "substeps": (1) to identify all the technologies that are used to provide the website and (2) to identify which of those technologies are relied upon for accessibility purposes."

Resolution: No change

Rationale: In this step the evaluators assess which technologies are being used to provide a website and that are relied upon for WCAG conformance. Other technologies are not directly relevant for conformance evaluation (except for cases of potential intereference which is checked separately).

Only the web technologies that are relied upon to provide the website need to be identified for evaluation.

"Consider adding "functionality" after the word "website" or find another way to make it clear."

Resolution: No change

Rationale: Adding "functionality" seems to limit the scope of the current sentence, as static information is not typically understood to be functionality.

Identify the technologies relied upon to provide the website.

"Consider changing "provide" to "render", "produce", or something like that. Or changing it to "relied upon to provide the website functionality"."

Resolution: No change

Rationale: See comment 43 on adding the word "functionality". The word "render" only addresses the display aspect and not necessarily the delivery, while the word "produce" addresses the production aspect and not necessarily the delivery and display aspect. We think that "provide" is more comprehensive in coverage.

"During an evaluation we use an in-house spider tool to assist with Step 2: Explore the Target Website, that in turn informs Step 3: Select a Representative Sample. The tool is not used to identify if something passes or fails, but to drill down into the taxonomy to identify different content types, dynamic content, components and technologies used. We often move into evaluating pages not in the original sample by way of following a process or link/button on the page."

"When undertaking the audit against WCAG-EM, following the strict sampling steps and reporting a comprehensive list of pages included in the sample set added additional time to the evaluation process. This time would have to be added to the full cost of the evaluation that the client will need to billed for. We would question the cost/benefit value of providing a full list of pages evaluated over providing the corresponding URL(s) for each example(s) of an issue and the scope from which the sample set was derived (domain and subdomains included or excluded)."

Resolution: No change; "Review Note" added to gather more feedback.

Rationale: The methodology tries to find the balance between practicability and transparency/replicability. Review notes have been added in several sections, in particular to get additional feedback on documentation requirements. We expect refinements in the next version.

"I have one general comment on the "sampling" stage. I have not seen any guidance related to how to measure the confidence level based on parameters of the sample. But this confidence level is essential when declaring the results of the evaluation of the website as a whole. Some guidance should be included in the steps below."

Resolution: Added more background in the introduction for this step and added random sample selection as indicator for confidence.

Rationale: There is no single widely accepted alogrithm to calculate confidence but there are some indicators like comparing the results to that of a random sample.

The note in this section

"Consider if this is needed here and later. Right now it's too much text and repetitive. Can it be only in one place? Can you edit it significantly?"

Resolution: Note removed.

Rationale: The text of the note is repeated else where.

"Instead of creating pressure on orgs to consider assessment of ALL pages, why not focus on trying to get them to think about selecting the best possible samples within reasonable scope, rather than have them perceive this selection as a failure because they can't afford to cover all pages?"

"I know we mention it's usually not possible to evaluate all pages, but if we can recognize that it's rarely possible, then why say ideally it should be all pages? That creates unrealistic expectations and a pressure most people do not need to deal with. I feel this is wishful thinking - we need to be more realistic about this. Most accessibility evaluation do not cover all pages and no organization can ever afford to run a complete evaluation of all their pages, except for very small sites (and even then, if they have a small site, their accessibility budgets will tend to be proportionally small)"

Resolution: Clarifications made.

Rationale: Several edits have been made, in particular addition of a section to explain the relationship to WCAG 2.0 conformance.

.."Discuss whether these states should basicaly form additional pages (i.e. base URL plus documented action) or whether we propose a format where pages in the sample can be given a number of enumerated states

The same applies to sequential processes starting on pages (say a search might be described by
- base URL
- text input for searcvh followed by submission
- evaluation of generated page

Generated pages often cannot be specified as separate URL, they often need to be speified by describing the process strarting fropm some base URL"

Resolution: Changes made.

Rationale: Extensive description, in particular in step 3.d.

"As promised, I have tried to come up with draft (probably within step 3.d) for including complete processes."

Resolution: Accepted with changes.

Rationale: Changes made as described in comment 50.

Resolution: No change.

Rationale: A random sample is important as an confidence indicator; see comment 53.

"The minimal amount of pages depends on the size of the web site. It may be more relevant to talk about a percentage instead of a set number of pages. But some sites have such a big amount of pages that it would not be realistic to select a sample of 5%. In order to make regular audits, it may be helpful to select a sample with a restricted number of pages (not more than 20 pages) and make automated evaluations of randomly selected web pages, or on the whole web site (as some tools allow to do this). However, for manual checks, it is not feasible to give a percentage. This could also be given to the reviewer's discretion according to the context of the web site."

"Several of us work more and more with automated tools and they consider that if they would not work with tools they would probably miss many accessibility errors that cannot be detected while choosing manually the pages of the sample. they feel that WACG-EM shoud give more importance to the help an automated tool can provide as today's web sites can be more complex and some of them are huge."

Resolution: No change; clarifications made.

Rationale: The 5% is based on size of the structured sample rather than on the size of the entire website.

"I agree that a random sample can act as a verification indicator of the results found in the structured sample, but I don't really agree that a random sample can be used to increase the confidence in the results of the evaluation of the structured sample. The reason is that the structured sample and the random sample are two different samples and my understanding from my (low) knowledge on statistics is that these two different samples cannot be combined into a single sample to derive a global result for the website with an increased confidence level. So I think that this step should be focused on using the random sample as a "control sample" to see whether the results of the structured sample are replicated in the random sample. Another possible use of the random sample would be to increase the probability of finding new accessibility errors that were not present in the structured sample. In addition I would like to know the reason for the numbers used. Why 5%? Why a minimum of 5 pages? What if the website only contained 10 pages and 6 of them were already part of the structured sample? It would be great to have some reasoning explaining how the probability of finding new errors could increase by increasing the percentage of pages selected in the random sample."

Resolution: No change; clarifications made.

Rationale: The random sample is a "control sample" to verify that you have not missed to evaluate certain types of content. This increases confidence in the evaluation results. The text is now clearer that the random sample and structured sample do not have the same instances of web pages. The number 5% (now 10%) is thought to be sufficient as an indicator.

Random Sample

"I would suggest around 10-15% (where 50 pages would include another 5-7 random pages to complete the sample)"

"5% seems a little low to me, even when the sample contains over 30 to 50 pages total (WCAG-EM implies it should always be more than this)."

Resolution: Increased to 10%.

Rationale: After further discussion it was agreed that 10% is a better indicator.

"why 5? why 5%? Where does it come from? How does one select a random sample?

Literally that would mean to first list all (ALL!) the pages and then using a random number generator, select the sample.

A variation could be a random stratified sample, where pages are first grouped into categories (like product description pages) and a random sample is selected from each group.

Another way is a sample generated through random walks in the site (i.e. randomly selecting one link among the available and not yet visited ones) Notice that even a pure random sample of pages has to cope with “complete process”. And hence has to be extended.

The size of the sample depends on the kind of results that are needed.

For statistical significance, the size depends on the spread/variance of the measured variable. Hence size cannot be decided a priori. Other auditing goals might call for other criteria, like a coverage criterion that aims at capturing all the possible types of problems."

Resolution: No change; clarifications made.

Rationale: The methodology does not prescribe an algorithm for random sampling. The size of the random sample will vary depending on the size of the structured sample, which again will vary depending on the properties of the website. The random sample is intended to be a control group for (simple) verification of the results generated by the structured sample. See also related comment 53 and comment 54.

This step should be mandatory, not optional.

"I don't agree with this step being optional. I cannot think of any good reason to maintain duplicated pages in the sample. That would decrease the relevance of the sample and the confidence level of the result, wouldn't it? I think that this step should be mandatory.."

Resolution: Section removed.

Rationale: Avoiding duplicates is now considered throughout the sections.

"some of the content of this step seems to me to be out of scope. WCAG-EM is about testing conformance to WCAG 2.0, that is, to determine whether a web site conforms to the technical requirements of WCAG 2.0, which are the 61 success criteria and the 5 conformance requirements. WCAG-EM should not deal with other types of accessibility evaluations, such as user testing or the concept of "accessibility-in-use" as defined in current research. So the first substep, as 4.a (check for use cases), seems to me out of scope."

"What I expected as content of step 4 is guidance about:

• how to evaluate each success criterion for each element of a web page of the sample, including guidance about the best approach: inspection, testing by experts, testing by users...
• how to combine the results obtained for each element in order to get a result for the page,
• how to evaluate the 5 conformance requirements once the result of each SC is known for the page,
• how to combine the results obtained for each page to get the global result for the site (that then will be reported in step 5)."

Resolution: Section rewritten.

Rationale: Provides more guidance on how to evaluate content using WCAG 2.0 and its accompanying document.

Note 1

Proposal for that second sentence: "An evaluator does not need to repeatedly identify successes and failures in meeting the conformance target for these repetitive elements on every web page. The success and failures for such repetitive elements can be identified once and then be reused in all the pages those elements appear."

"The first note is confusing. If there are repetitive elements on every web page it makes sense to evaluate them only once, and then record and reuse the evaluation results. It is clear that an evaluator should not valuate several times the same repetitive elements but it is also important to note that the result assigned after the first evaluation has to be used when evaluating each page where the repetitive elements appear! My understanding of the second sentence of this note is that it implies that the repetitive elements are only evaluated once and their result is only used once... I'm probably misreading it, but I think that the text would benefit from some rewrite. In addition I don't see the point of referring back to step 1.b in this note. What would be the difference on dealing with repetitive elements depending of the type of expected report?"

Resolution: Edited.

Rationale: The note has been edited and worked into the main text.

Is this "in reality what [..] people do?"

Question: "in reality what do people do? For example the Germans that follow the BITV, what methodology do they employ?
Or the Knowbility.org people that run the accessibility rallies?
Is this methodology connected to those? Because there’s no bibliography in this document I think it is not. And it should not be such."

Resolution: No change.

Rationale: Representatives from both mentioned organizations (and more) are actively involved in the development of this methodology. This edior note was to gather additional input.

Note 1

"We should suggest isolating template findings from the rest of the findings related to main content of a page or pages, so findings/issues related to templates can be referenced globally for all related pages, rather than repeating the findings every time."

"By doing this, once the template findings are covered, all subsequent pages could only be covering the main content issues, making the whole process simpler and more organized."

Resolution: Note removed.

Rationale: Templates are addressed in the exploration and sampling steps instead, to make sure that templates that are actively used are being evaluated (rather than templates in absence of content).

Note 2

Proposal: "According to WCAG 2.0, Success Criteria to which there is no matching content are satisfied. In such cases, an evaluator may use text such as "Not applicable" as a description associated to the outcome "satisfied", to denote the particular situation where a success criterion was satisfied because no relevant content was applicable."

"I think that the wording of this note is confusing. It should clearly explain that, according to WCAG 2.0, "no matching content for SC" is really "SC is satisfied" with no question. It should also explain that the "not applicable" could be used as a "description" of the result but not as a result.

Resolution: Edited.

Rationale: Text edited to make clearer.

"There was little direction related to web applications related to any documentation that may have been provided with the web application, or published by the company selling the web app. For example, the Oracle whitepaper Testing Oracle Products for Accessibility , mentions many things related to the accessibility of web applications, including:

• making sure to read documentation to understand how the product is meant to be used, such as keystrokes
• documentation that might point out workarounds, including alternative paths through a website
• enablement of applicable accessibility modes, if any
• if using AT, ensuring that a defect found is not in the AT itself, and that the user is proficient
• if using automated test tools, ensuring that it is accurately representing the SC and sufficient/failure techniques that it claims to meet"

Resolution: No change.

Rationale: The proper assessment of Success Criteria is assumed to be part of the skills of an evaluator.

The second note in this section

"Make the first point stronger and more clear. Make the second more clear that it's optional."

Example: "If there is no content related to a success criteria (for example, no video), then the success criteria is "satisfied" or "met", according to WCAG 2.0. Optionally, an evaluation report can specifically indicate success criteria for which there is no relevant content, for example, with 'Not Applicable'."

Resolution: Change made.

Rationale: Proposal accepted with edits.

Closed

"It is strongly recommended to also involve real users during this process.... Whilst we completely agree with this and conduct 'technical' testing with end users during WCAG evaluations, it is no substitute for formal usability testing. 'Practical' accessibility issues are just as important as 'technical' accessibility issues but may be disregarded when assessed in the context of a technical evaluation as they do not always directly map to WCAG 2.0."

"We all interact with websites using different browsers, different ATs and different approaches. Determining the true nature of an accessibility issue when involving users is not as straightforward as merely asking them what they believe is the issue, or reporting on general observations as with mainstream users. To undertake comprehensive formal usability testing for accessibility we believe there is a real requirement to understand user experience, technical accessibility and how ATs are used and their capabilities; it is rare you find people with all of these skills and experience."

Resolution: Section rewritten.

Rationale: Provides more guidance on how to evaluate content using WCAG 2.0 and its accompanying document.

Change the title in: "Step 4.a: Check the conformance requirements".

"The title of misleading, because the requirement 4.a is about checking the conformance requirements, not about use cases."

Resolution: Section rewritten.

Rationale: Provides more guidance on how to evaluate content using WCAG 2.0 and its accompanying document.

"I would like to have guidance outlining a process such as:

• a) Look for failures (as published)
• b) If no failures are found, check if any SC fails
• c) If no SC fails and a "in-depth analysis" has been selected in step 1.b, then determines which techniques have been used to conform to each SC"

"I find that this step has gaps in the content I expected it to have. In my opinion checking for the 5 conformance requirements is not a straightforward task and more guidance is needed."

" My proposed process is based on the definition of "satisfies a SC" in WCAG: "the success criterion does not evaluate to 'false'", Given that definition, it is not really needed to find/determine the techniques. It should be enough to be sure that no SC fails."

"All the text about scenarios, use cases, personas and user involvement is, to me, out of scope when dealing about testing for conformance with a standard. These type of things are essential during the development process when following a user-centred approach, but I believe that they don't provide added value in the context of checking the conformance of WCAG 2.0. If this text about scenarios and users is to be kept, I'd suggest it being transformed into a note."

Resolution: Section rewritten.

Rationale: Provides more guidance on how to evaluate content using WCAG 2.0 and its accompanying document.

"The second paragraph, starting with the text "To carry out an evaluation effectively, it is often useful to construct and apply personas..." over-emphasizes a specific approach, particularly with language like the second sentence "It is critical to consider...". It should be possible to evaluate against WCAG 2.0 on a purely technical, objective basis. As such, this material could be presented as an advisory statement. As written now it goes beyond the scope of WCAG 2.0 conformance."

Resolution: Section rewritten.

Rationale: Provides more guidance on how to evaluate content using WCAG 2.0 and its accompanying document.

Place this step first

"If one enumerates the use cases then the sample of pages may change, as per “complete process” requirement. Hence this step should come first."

Resolution: Section rewritten.

Rationale: Provides more guidance on how to evaluate content using WCAG 2.0 and its accompanying document.

Resolution: Section removed; edits worked into Step 4.a and 1.c.

Rationale: More guidance on determining a baseline for accessibility support.

"This should be linked to step 2.d (identify web technologies relied upon)"

"The first "cut" for features that have accessibility support is to look for the ones using technologies identified"

Resolution: Section removed; edits worked into Step 4.a and 1.c.

Rationale: Edits have been made though emphasis remained that accessibility support is determined by feature rather than by technology.

"According to my comment for step 4.a, I think this could be optional as it is only needed when the evaluation report will contain detailed information about how each SC is met (see my outlined process, step c)."

Resolution: Section removed; edits worked into Step 4.a and 1.c.

Rationale: More guidance on determining a baseline for accessibility support.

"In an Australian Government context this is not optional."

Resolution: Section removed; edits worked into Step 4.a and 1.c.

Rationale: More guidance on the use of techniques in the context of WCAG 2.0; enforcing the use of (informative) techniques is an additional requirement to WCAG 2.0 and is beyond the scope of this methodology.

"I propose dividing this into two: 4.c.1 Use failures (mandatory) and 4.c.2 Use techniques (optional). (If this is accepted then the text of these two steps should be rewritten to fit the new structure.)"

"I agree that the use of techniques is optional, but I fundamentally disagree in the use of failures. Even if failures are part of an informative document it is 100% sure that if one of those failures is found, then the corresponding SC fails for the page being evaluated. I cannot imagine any exception to that idea. As I already explained above, given the definition of "satisfying a SC", the concept of "failures" is an essential tool for evaluators and should not be optional"

Resolution: Section removed; edits worked into Step 4.a and 1.c.

Rationale: More guidance on the use of techniques in the context of WCAG 2.0.

"This section appropriately describes the proper use of techniques (that they are informative and not mandatory). However, the document would be improved if this were particularly emphasized. We are seeing entities that are requiring the use of WCAG 2.0 also requiring the use of these techniques as THE way of demonstrating compliance."

Resolution: Section removed; edits worked into Step 4.a and 1.c.

Rationale: More guidance on the use of techniques in the context of WCAG 2.0; WCAG WG provided updated guidance that is referenced from the respective sections.

This step needs some clarification.

"I think that it would be good to have an explanation of the reasons to make this step optional. My first idea when reading the title of the step was to ask for it to be mandatory, as I thought that any evaluator should really archive the pages being evaluated (at least the URI and data required to be able to re-evaluate the page if needed). But then I read the details of this step and I tend to agree that all this information is optional. This is why I think that some clarification is needed."

Resolution: Section removed; edits worked into Step 5.b.

Rationale: More guidance on recording the evaluation specifics (optional) as opposed to documenting the evaluation outcomes (required).

This step should not be optional.

"I think that this step should not be optional. I agree that it is not always necessary to report on this information but evaluators should always record the information so it can be accessed if needed."

Resolution: Section removed; edits worked into Step 5.b.

Rationale: More guidance on recording the evaluation specifics (optional) as opposed to documenting the evaluation outcomes (required).

"This part is really interesting. It may be useful to add a more teaching part for reports dedicated to decision-makers, in particular, a section explaining the consequences of the success criteria for users. This would help to raise awareness on accessibility by all actors implied in the web site development. Moreover, a basic report with a list of errors is really useful, if it is dedicated to people who already know about web accessibility and who just need a checklist because they have no time to review the web site themselves. An example of report would be helpful to better illustrate what is meant here."

Resolution: Suggested report types have been removed.

Rationale: Section has been rewritten to better explain the different recordings that can be made. Providing explanation on the impact of Success Criteria is explained in Understanding WCAG 2.0 and is not directly part of evaluation. WAI provides a Template for Accessibility Evaluation Reports that can be used in combination with this methodology.

"In the list of information documented in the report there is one reference missing: step 3.f (optional), that should be listed with all the other sub-steps of step 3."

Resolution: Section removed.

Rationale: Avoiding duplicates is now considered throughout the sections.

Proposal for replacing the last sentence: "Where failures in meeting WCAG 2.0 Success Criteria on a web page are identified, at least one examples is provided for each identified failire in the page."

"I think that the approach for documented failures in the detailed report could be similar to the one in the basic report, but on a page by page basis. That is, if a success criterion fails, then at least one example is provided in the page."

Resolution: Related text removed.

Rationale: See comment 77.

Book by Rubin gives description of possible different goals for reports

"Rubin, in this book on usability testing, gives a neat description of different goals that usability investigations might have. Here something similar could be done."

Resolution: Related text removed.

Rationale: See comment 77.

"While the title uses the phrase "statement", within the body it returns to using the word and concept of "conformance", which, as defined by WCAG, only exists with perfection. Also in this section, the note appears to ignore the reality that many websites and web applications are in "continuous development", and are never finished (cf. the long lived "beta" applications from a number of well-known companies)."

Resolution: Changes made.

Rationale: Clarified text and intention, and linked to the newly added section on relationship to conformance claims.

Conformance Evaluation Statement (Optional)

Change: "As per the requirements for WCAG 2.0 conformance claims, also accessibility evaluation statements have to include the following information:" to: "As per the requirements for WCAG 2.0 conformance claims, accessibility evaluation statements also have to include the following information:".

Resolution: Changes made.

Rationale: Clarified text and intention, and linked to the newly added section on relationship to conformance claims.

The note in this section

"Delete the negative note at the end and add a positive statement at the beginning of the section saying taht conformance claims are for completed websites. Also, considef making "WCAG 2.0 conformance claims can only be made for the web pages that have been actually evaluated and identified to conform with its conformance requirements." more positive. "

Resolution: Changes made.

Rationale: Clarified text and intention, and linked to the newly added section on relationship to conformance claims.

Make clear what you mean by performance score and who/what it is for

"Who’s performance? In which sense? Are you talking of accessibility metrics, or expected performance by users belonging to certain categories and trying to achieve certain goals?"

Resolution: Changed to aggregation score.

Rationale: Edited to clarify that the score is calculated by aggregation of outcomes.

"The document highlights some of the pros and cons of the three suggested approaches and we can see the value for an organisation which internally wishes to benchmark their accessibility over time."

"Our concerns for the wider use of performance scores presented to the public are detailed below (in no particular order). We raise this as we have been asked to provide a score, percentage or similar ranking measure on numerous occasions. However, we do not believe the intention behind these requests on the whole are beneficial to the needs of a person with a disability or age related impairment:

• All three approaches fail to include practical accessibility issues not captured, or intended to be captured by a WCAG assessment. The final score may not be an accurate reflection of the lived experience for a person with a disability using the website
• Website owners may remove resources to rectify outstanding WCAG issues once they achieve a score they deem to be adequate
• May lead to a reduced incentive to undertake formal usability testing by website owners once they achieve a score they deem to be adequate
• From what we can establish there will be a variance in the final score of the three approaches if applied to the findings of the same audit, so there will be some discrepancy around how accessible a website is
• Website owners may wish consultants to use the most favorable of the three approaches if the score is to be made public
• None of the three approaches appear to reflect the impact of an issue on an end-user (level A issue over level AAA issue), so are somewhat arbitrary
• To make the performance scoring meaningful it involves quite a lot of additional effort from the evaluator; only Per Website would realistically be achievable for a commercial engagement, and is the value add really there."

Resolution: No change; "Review Note" added to gather more feedback.

Rationale: While there are valid concerns with using metrics, they can be an indicator of progress to drive implementation. Review notes have been added in several sections, in particular to get additional feedback on scoring. We expect refinements in the next version.

"Per and 'as per' is a pseudo-latin construction. Opinions appear to divide between clarity, efficiency, and improper use. There are however alternatives which are more natural."

"The only examples that fit well are: taking medicine 'per day', or sharing pencils 'one per child' and 'per your instructions. The use of 'Per website' and 'per web page' seem particularly puzzling and drove me to Google to find out why."

An interesting discussions on the use of per and as per

Resolution: Change made.

Rationale: Better English.

"The only valuable scoring methodology is one which takes into account the impact of the issues discovered on a page and/or site. Issues that will stop any disability group from fulfilling the use case for which the page and or site is designed should be marked as "Critical" or "Blocker". Any page with an issue of this nature should receive a score of 0."

"Other issue severities could be formulated each with its own score and the page and a site overall could be scored using this methodology. This score would then more closely reflect the actual utility of the site to the overall community of users with disabilities."

"The scoring methodology "Step 5.c" is recommending a performance score that has very little connection with the actual accessibility of a site for any score that is less than 100%.

The reason for this is simple: It is possible for a web site login page to pass every guideline except guideline 2.1.1 under your proposed scoring mechanism, this page would obtain a score of 91.6%, however it would be totally unusable by a keyboard only user.

On another page, there might be dozens of missing alt attributes on images that are purely decorational, plus some structural markup problems in the footer, color contrast issues on UI components not central to use of the functionality etc. and the score would be lower, but most users with disabilities would have no problem using the site."

Resolution: No change.

Rationale: See comment 84.

Resolution: No change.

Rationale: See comment 84.

"I appreciate the effort spent in providing some guidance in this difficult area of web accessibility reporting. The three options are well explained with details on their sensitiveness. This area of accessibility metrics is still under research and I think that WCAG-EM should at least acknowledge the W3C report (http://www.w3.org/TR/accessibility-metrics-report/) that was produced after a symposium organized in December 2011 (http://www.w3.org/WAI/RD/2011/metrics/)."

Resolution: Reference to (draft) Research Report is made.

Rationale: See comment 84.

"This is an invitation to consumers of WCAG-EM reports to set a "yes/no" bar that is at some specific point below 100% perfection (e.g. a score of 95 or higher is required). Such a move would be of little additional improvement to the present situation. We do not believe this should be even a suggested part of WCAG-EM."

Resolution: No change.

Rationale: See comment 84.

Resolution: Section removed.

Rationale: Section has been removed. WAI provides a Template for Accessibility Evaluation Reports that can be used in combination with this methodology.

Open links in a new tab

"It would be great to have links open in a new tab, so, for example, it isn't necessary to toggle back and forth between the survey, Step 1, and the test website. Would save time cutting and pasting."

Resolution: No change.

Rationale: User agents provide this support depending on the particular preference of the reader.

"Change headings to something like: "Contents Summary" and "Detailed Contents"."

Resolution: Change made.

Rationale: Better wording.

"Include the Appendices in the Detailed Contents list, rather than under a separate heading."

Resolution: No change

Rationale: Appendices are not part of the main content and easier to skim when separated out.

Step headings

• "Define the Scope of the Website" -> "Define the Scope"
• "Define the Goal of the Evaluation" -> "Define the Goal"
• "Define the Conformance Target" -> "Define the Conformance"
• "Define the Techniques and Failures to be Used (Optional)" -> "Define the Techniques and Failures (optional)"

"Differentiate the Step headings from the statement that follows. Maybe make the headings even more terse."

Resolution: Changes made.

Rationale: Wording of methodology requirements have been improved to make more clear and avoid repetition.

Methodology Requirement

"Delete "Methodology Requirement" and the colon so it's like: <h4>Step 1.a. Define the Scope</h4> <p class="req">1.a. Define the scope of the website..</p>"

Resolution: No change.

Rationale: Need to clarify that they are "requirements".

Statement under Steps

"Remove the links to the sub-steps"

"They are long, redundant, and the links are distracting."

Resolution: No change.

Rationale: Important to point out the hierarchy in the requirements.

Definition lists

"Change the definition lists in some places where it would be better to use different markup."

"For specific places and rationale, see e-mail"

Resolution: Changes made.

Rationale: Changed some of the definition lists to simple lists where applicable.

Summary sheet

"Provide a one-page summary in multiple formats, e.g., HTML, RTF, spreadsheet, pretty PDF for printing."

"EOWG can help with this."

Resolution: Accepted; no change.

Rationale: We welcome a proposal from EOWG.

Optional sections

"Explain up front why some sections are optional."

"Explain up front why some sections are optional, and that in most cases the optional sections are strongly recommended. If warranted, reiterate in specific sections that the step is strongly suggested and note why it's best practice."

Resolution: No change.

Rationale: Some indication already exists in most cases.

Basic Report, Detailed Report, In-Depth Analysis Report

"Please come up with better terms for each and better explanations. Also, the main explanation should be in one place; it's confusing to have it in two places, as it currently is. EOWG started talking about these levels in in 5 April telecon and comparing the two explanations, but didn't finish. Perhaps we might have a joint discussion? "

Resolution: Text removed.

Rationale: The concept of "Basic", "Detailed", and "In-Depth Analysis" reports has been removed from the document.

Review Teams and immediately following section, Involving Users both begin with the same sentence. "The methodology defined by this document can be carried out by an individual evaluator with the skills described in section Required Expertise." Reconsider if this sentence is really necessay in both places. If so, consider that it will look like an error to some people, so see if there's aother way to cover the information.

Resolution: Change made.

Rationale: Improved wording to avoid repetition.

"The document in general would benefit from a note mentioning some of the potential legal ramifications of using WCAG-EM."

"For example, if a company creates such a WCAG-EM audit even with the intent of correcting the items found, it could be used against them in a court of law (such documents can be found in discovery and would likely be looked for, even if not published). At a minimum, the document should advise anyone performing this type of work that it would be wise to consult legal counsel before starting (this is touched on in the Note at the end of 5.a, but really should be called out at the top in more detail)."

Resolution: No change.

Rationale: Legal ramifications are beyond the scope of this documet (and of WCAG too).

"Overall Comment: This is thorough. That is a good thing. Tom and I agreed that this would have been great for us when we started our giant accessibility project at the California State University. We had to discover so much of this on our own. The WCAG-EM group managed to pull together a lot of hard earned experience into a truly useful guideline for someone who needs to bring a site into complete Level 2 conformance. Overall, it reads well. EO has caught most of the bumps. Combined with the Easy Check from EO (for work-a-day) review. This is a powerful package. "

Resolution: No change

Rationale: Thank you

"The link text of sections (steps) that are referred to are not always the same as the sections titles. For example, at the end of section step 5.A, there is a link to "Step 5.b: Provide and Accessibility Statement)". First there need to be corrected in "an accessibility Statement". Second, the title of the heading is: "Step 5.b: Provide a Conformance Evaluation Statement (Optional)", whereas the Methodology Requirement 5.B is entitled: "Provide an accessibility conformance evaluation statement (Optional)."."

Resolution: Changes made.

Rationale: Editorial errors.

"Would it be useful here to suggest personas? To think about the different uses that people will have for the large website and to define those pathways? For example, a government agency site may have people coming who are looking for general information, or they may be seeking specific services. There could be a robust series of pages that do not require log-ins as well as those that do. Interest/interaction may be regional and there is likely to be a different series of pages for agency staff."

"I have found that if the commissioner of the review will define the stakeholder/constiuent groups who most frequently use the site and what they do there, it can help define critical pathways for various tasks."

Resolution: No change.

Rationale: Providing guidance on using personas is outside the scope of this document.

"I would like to suggest a proportion of 60%/40% between portion of web pages that were in the original sample and additional new web pages that were not in the original sample."

"To get significant distinction between old pages and new ones and measure how well they compare once remediation has been applied to webpages."

Resolution: No change.

Rationale: We propose to leave this decision to the evaluator and the evaluation commissioner. The evaluator or the evaluation commissioner could want higher or lower numbers here. This would be visible in the sample.

In first sentence of second paragraph of Abstract, should be comma after "for example"

Resolution: Change made.

Rationale: Editorial

sufficient technqiues

Change "technqiues" in "techniques".

May be several times in the document.

Resolution: Change made.

Rationale: Editorial

sufficient technqiues

Change "Sucess criteria" to "Success criteria".

May be several times in the document.

Resolution: Change made.

Rationale: Editorial