W3C

Results of Questionnaire Approval for publication of WCAG-EM 1.0 as a W3C Working Group Note

The results of this questionnaire are available to anybody. In addition, answers are sent to the following email addresses: public-wcag-em-comments@w3.org,shadi@w3.org,e.velleman@accessibility.nl

This questionnaire was open from 2014-06-23 to 2014-06-30.

16 answers have been received.

Jump to results for question:

  1. Abstract
  2. Introduction
  3. Using This Methodology
  4. Scope of Applicability
  5. Step 1: Define the Evaluation Scope
  6. Step 2: Explore the Target Website
  7. Step 3: Select a Representative Sample
  8. Step 4: Audit the Selected Sample
  9. Step 5: Report the Evaluation Findings
  10. Remaing Comments

1. Abstract

Summary

ChoiceAll responders
Results
accept this section 15
accept this section with the following suggestions 1
I do not accept this section for the following reasons
I abstain (not vote)

Details

Responder Abstract
Yod Samuel Martin accept this section
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section Very clear. No changes required.
Kerstin Probiesch accept this section
Kathleen Wahlbin accept this section
Frederick Boland accept this section
David MacDonald accept this section
Detlev Fischer accept this section
Michael Elledge accept this section with the following suggestions Priority: medium. Location: first paragraph. Current wording: "This document is one of a series of informative W3C/WAI resources about Evaluating Websites for Accessibility that complements the WCAG 2.0 Documents." Suggested revision: "complements" with "complement." "It does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way." Replace "supersede it" with "supersede them." Rationale: Improve grammar.
Ramón Corominas accept this section
Howard Kramer accept this section

2. Introduction

Summary

ChoiceAll responders
Results
accept this section 13
accept this section with the following suggestions 3
I do not accept this section for the following reasons
I abstain (not vote)

Details

Responder Introduction
Yod Samuel Martin accept this section with the following suggestions priority: mild (editorial)
location: definition of "Template"
current wording: "From ATAG 2.0 definition for "templates"
suggested revision: add ATAG 2.0 as well to the reference section
rationale: its definitions are used in the text (can it be referenced despite being still a CR?)
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section 1. The list of "required" reading is somewhat daunting. Reading the Techniques end to end is counter recommendation, but that appears to be the suggestion.
2. The glossary is spectacular. That collection of definitions alone is worth reading the document!
Kerstin Probiesch accept this section
Kathleen Wahlbin accept this section
Frederick Boland accept this section
David MacDonald accept this section
Detlev Fischer accept this section
Michael Elledge accept this section with the following suggestions Priority: medium. Location: Introduction section. Current wording: "how much knowledge the evaluators have of how the website was or is being developed". Suggested revision: "how much knowledge evaluators have of the website's development" and "feature by feature" with "feature-by-feature". Location: Purposes section. Current wording: "Web accessibility monitoring activities used to benchmark or compare the accessibility conformance over time." Suggested revision: "Web accessibility monitoring activities used to benchmark or compare accessibility conformance over time." Rationale: Improve grammar.
Ramón Corominas accept this section
Howard Kramer accept this section with the following suggestions priority: medium
location: last line of second paragraph of Introduction
current wording: This methodology supports common approaches and understanding for evaluating the extent of conformance of websites to WCAG 2.0, though in the majority of use cases it does not directly result in conformance claims.
suggested revision: I don't have a specific revision but it was perhaps the only sentence I found confusing due to the wording, in particular the phrase "understanding for" it did not quite make sense. Also, the whole issue of when this evaluation could be used to make conformance claims was no 100% clear to me. Maybe an example here could help.
rationale: lack of clarity

3. Using This Methodology

Summary

ChoiceAll responders
Results
accept this section 13
accept this section with the following suggestions
I do not accept this section for the following reasons 2
I abstain (not vote)

Details

Responder Using This Methodology
Yod Samuel Martin accept this section
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section No change.
Kerstin Probiesch I do not accept this section for the following reasons Same opinion like David. I also think that the section "combined expertise" gives advantages to organizations. I would agree with David's suggestion - except the last sentences and suggest:

"Combined Expertise (Optional)

"This methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise). Using the combined expertise of different evaluators may provide an effective way to evaluate content when some of the required expertise is missing from one team member but is possessed by another on the team."
Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald I do not accept this section for the following reasons Combined Expertise (Optional)
"... using the combined expertise of different evaluators may provide broader coverage of the required skills and help identify accessibility barriers more effectively..."

===========
The referenced document "Using Combined Expertise to Evaluate Web Accessibility" is 12 years old (2002)

I appreciate the first sentence disclaimer and "optional" status, but the message seems clear...more evaluators on content is better. We had some discussion of this section at CSUN and at TPAC. Several veteran evaluators who have worked in large organizations, also felt that this is not the reality of how things work. Accessibility evaluation companies that I know of use teams of evaluators, but they split up a site and each evaluator takes a separate section. They are not combining expertise on the same content.

I think the current language unnecessarily gives advantages to organizations with teams of evaluators. There may be a mistaken impression to procurement departments that they better meet this recommendation than smaller firms, even though in reality they have 1:1 ratio of one evaluator to any chunk of content.

This section is not about the important recommendation of including users with disabilities, that is a separate section of this document ... this section is about teams of evaluators looking at the same content.

How about a friendly amendment that might better address the issue?
===============
Combined Expertise (Optional)

"This methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise). Using the combined expertise of different evaluators may provide an effective way to evaluate content when some of the required expertise is missing from one team member but is possessed by another on the team. While not required for using this methodology, the use of review teams may sometimes be necessary and/or beneficial. Using Combined Expertise to Evaluate Web Accessibility provides further guidance on using the combined expertise of review teams, which is beyond the scope of this document."

======
Also I would put the paragraphs "Involving Users (optional)" with disabilities and "Evaluation tools (optional)" above this paragraph in the section because they are more important.

Detlev Fischer accept this section
Michael Elledge accept this section
Ramón Corominas accept this section
Howard Kramer accept this section Good examples.

4. Scope of Applicability

Summary

ChoiceAll responders
Results
accept this section 14
accept this section with the following suggestions 1
I do not accept this section for the following reasons
I abstain (not vote)

Details

Responder Scope of Applicability
Yod Samuel Martin accept this section with the following suggestions Priority: mild (editorial)
Location: Re-Running Website Evaluation, last paragraph
Current wording: "is typically ~50%"
suggested revision: replace the tilde with a complete word (e.g. "approximately 50%", "around 50%").
rationale: this sign is mathematical jargon, not suitable to be used within running text, and not always screen-reader friendly.
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section The Principle of Website Enclosure is Especially Clear.
The entire section gives a clear framework for determining scope.

Question: Does the referenced example site exist?
Kerstin Probiesch accept this section
Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald accept this section "...amount of replaced web pages in a fresh sample is typically ~50% though this could be increased when web pages on a website mostly conform to WCAG 2.0."

Wondering where 50% came from? I don't object particularly, but wondering.
Detlev Fischer accept this section
Michael Elledge accept this section
Ramón Corominas accept this section
Howard Kramer accept this section

5. Step 1: Define the Evaluation Scope

Summary

ChoiceAll responders
Results
accept this section 13
accept this section with the following suggestions 1
I do not accept this section for the following reasons
I abstain (not vote)

Details

Responder Step 1: Define the Evaluation Scope
Yod Samuel Martin accept this section
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section Good and Clear
Kerstin Probiesch accept this section
Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald accept this section
Detlev Fischer
Michael Elledge accept this section with the following suggestions Priority: Medium. Location: Description; 2nd paragraph, Step 1.a. Current wording: "in such a way that for each web page it is unambiguous whether it is within scope or not". Suggested revision: "so that it is unambiguous that a web page is within scope." Rationale: Clarify.
Priority: Medium. Location: 1st paragraph. Current wording: "define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate to." Suggested revision: "define the target WCAG 2.0 conformance level ("A", "AA", or "AAA")." Alternatively: "define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") for evaluation." Rationale: Grammar.
Priority: Medium. Location: 2nd paragraph, Step 1.c. Current wording: "website is expected to work with, and that is in-line with." "common expectations on." Suggested Revisions: "website is expected to work with, and that are in-line with." "common expectations for." Rationale: Grammar.
Priority: Medium. Location: 3rd paragraph, Step 1c. Current wording: "extended with the tools that were used in addition." Suggested revision: "extended with the additional tools." Rationale: Grammar.
Priority: Medium. Location: Definition, Step 1d. Current wording: "agreed on between." Suggested revision: "agreed by." Rationale: Grammar.
Priority: medium. Location: 4th bullet point, Step 1.d. Current wording: "Explanation of possible solutions to the encountered issues beyond what would be in the scope of website evaluation." Suggested revision: "Description of possible solutions to the issues encountered beyond the scope of the evaluation." Rationale: Clarity.
Ramón Corominas accept this section
Howard Kramer accept this section

6. Step 2: Explore the Target Website

Summary

ChoiceAll responders
Results
accept this section 12
accept this section with the following suggestions 1
I do not accept this section for the following reasons
I abstain (not vote)

Details

Responder Step 2: Explore the Target Website
Yod Samuel Martin accept this section
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick More examples of essential functionality may be needed like "application for services from an agency."
Explicit reference of Dojo or JQuery may be vendor biased. Perhaps use categories like polyfills or Widget Libraries.
Kerstin Probiesch accept this section
Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald accept this section with the following suggestions "Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, jQuery" ...

perhaps add something like "if a CMS is used it will be helpful to identify it, and it's version, along with a list library components added to it's core framework."
Detlev Fischer accept this section
Michael Elledge
Ramón Corominas accept this section
Howard Kramer accept this section

7. Step 3: Select a Representative Sample

Summary

ChoiceAll responders
Results
accept this section 9
accept this section with the following suggestions 4
I do not accept this section for the following reasons
I abstain (not vote) 1

Details

Responder Step 3: Select a Representative Sample
Yod Samuel Martin accept this section with the following suggestions priority: medium
location: Step 3.a: Include a Structured Sample and Step 3.b: Include a Randomly Selected Sample
current wording: "The number of required instances of web pages and web page states depends on the particular aspects of the website explained in the previous section factors influencing the sample size" regarding the structured sample and "The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5" in the randomly selected sample.
suggested revision: Clarify the relation between the selection criteria and the sizes of both samples.
rationale: It seems there is some inconsistency here. Some aspects explained in the step introduction do not influence step 3.b, but the step takes them into consideration. Conversely, some of the steps do influence step 3.c but they are kept out.
For instance "Adherence to development processes" should not affect the size of the structured sample, as it doesn't add more of any of these: common web pages, essential functionalities, types of web page states, web technologies relied upon, or other relevant web pages. However, it should yet influence the size of the random sample, as the introduction states these sites show more (or less) variation in code quality and accessibility.
Likewise, a higher required level of confidence may not require a larger structured sample: if I have already considered one page of each combination, adding more does not cover any new essential functionalities, etc. Yet the level of confidence should require a larger random sample.

priority: mild (editorial)
location: Step 3.d: Include Complete Processes, 1st list item
current wording: "For any web page and web page state selected (...), locate the starting point (...) for the process and include them in the selected sample"
suggested revision: "For *each* web page and web page state selected (...), locate the starting point (...) for the process and include *it* in the selected sample"
rationale: grammar agreement and concord


priority: medium
location: Step 3.b: Include a Randomly Selected Sample
current wording: random
suggested revision: Add a sentence explaining that "random sampling does not imply following a uniform sampling distribution". For instance, following one of the suggested sampling methods, if web pages are extracted from access logs at random, most visited pages would be overrepresented, but this is not necessarily wrong.
rationale: Clarify that random does not mean "with equal probability".
Bruce Bailey accept this section
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section with the following suggestions Very clear.
Kerstin Probiesch accept this section
Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald accept this section I like the 10% idea... its practical and easy to calculate... this entire section really makes sense to me now.
Detlev Fischer accept this section with the following suggestions Location: Step 3.b: Include a Randomly Selected Sample

Priority: strong

Current wording: "The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5 instances of web pages and web page states."

Suggested revisions:

A. Amend - possibly in a note - the requirement of a minimum of 5 randomly selected instances of web pages and web page states to account for simple sites that will only require a very small sample.

B. Clarify what selecting a random sample of states means when creating a sample for web apps or remove the requirement for an additional random sample for web apps.

Rationale:
A. For some sites, the page sample can be very small: Think of a blog that may just have the default start page view and a bog entry page, with simple text-only blog entries. A minimum of 5 randomly selected pages / states does not make sense in such a case. Another edge case are 'one-pagers' which provide a minimal presentation for a company or product. There is sometimes literally just one page to test.

B. For web apps, it is not clear how the evaluator should select states randomly. I can think of no procedure as there is usually not a list of URLs, search hits or similar to work from.
Michael Elledge
Ramón Corominas I abstain (not vote)
Howard Kramer accept this section with the following suggestions priority: medium
location: first paragraph above "Step 3.a: Include a Structured Sample"
current wording: This selection in this step relies initially on the exploration carried out in Step 2: Explore the Target Website. The selection is also continually refined during the following Step 4: Audit the Selected Sample, as the evaluator learns more about the particular implementation aspects of the target website.
suggested revision: depends on meaning of "This selection in this step"
rationale: All the other statements that being "During this step" occur right under an overall methodology item (usually highlighted with a box border that it is clear what "step" is being referred to. Here, I wasn't sure if "this selection in this step" referred to 3a alone or the whole of methodology 3.

8. Step 4: Audit the Selected Sample

Summary

ChoiceAll responders
Results
accept this section 8
accept this section with the following suggestions 3
I do not accept this section for the following reasons 1
I abstain (not vote) 1

Details

Responder Step 4: Audit the Selected Sample
Yod Samuel Martin accept this section
Bruce Bailey accept this section with the following suggestions For Step 4.a: Check All Initial Web Pages, I recommend copying the text from the two cited FAQs rather just having the links. The cited material is not that long, and the FAQ is not as formal and stable as this document. Yes, that means the are more likely to get out of sync, but I think it is okay that this document preserves a snapshot of our best thinking at this moment at time. Working Group Notes are more authoritative and subject to more public review than the FAQ. Plus, one less click, and it is good content, well worth having in multiple places.
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section with the following suggestions This is the most difficult to read. It is the most detailed, legalistic section. So, rough reading may just be part of the landscape.
Kerstin Probiesch I do not accept this section for the following reasons I like this section, especially because of the mentioned Conformance Requirements. I'm missing explict guidance on criteria for passing and failing an evaluation. Please add explicitly "pass/fail" as scheme for the auditing the samples.
Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald "For example, evaluators may utilize specific testing instructions and protocols that may be publicly documented or only available to the evaluators."

May need to add a sentence that they would need to be able demonstrate that techniques they chose have actually met the SC.

============

"Optionally, an evaluation report can specifically indicate Success Criteria for which there is no relevant content, for example, with "not present".

May want to check in with Gregg on this... I personally don't have a problem with it, but it was a pretty hot topic at one point. "Not present" is better than N/A, and may be ok with him.

Detlev Fischer accept this section
Michael Elledge
Ramón Corominas I abstain (not vote)
Howard Kramer accept this section with the following suggestions priority: mild
location: all the sub-headings under 4a. such as "Conforming Alternate Versions"
current wording: these sub-headings are styled in italics with an underline. I think this is the only instance where this type of formatting is used for headings or sub-headings.
suggested revision: follow the convention for sub-headings used elsewhere in document.
rationale: First, of all, these items don't stand out as headings because their font weight and size is same as underlying paragraphs. Second, they look like links - another source of confusion.

9. Step 5: Report the Evaluation Findings

Summary

ChoiceAll responders
Results
accept this section 8
accept this section with the following suggestions 3
I do not accept this section for the following reasons
I abstain (not vote) 2

Details

Responder Step 5: Report the Evaluation Findings
Yod Samuel Martin accept this section with the following suggestions priority: medium
location: Methodology Requirement 5.d: Provide an Aggregated score (Optional).
current wording: this sentence has been removed since the previous WD "It is also recommended that the scoring approach is documented and made available to the evaluation commissioner along with the report to facilitate transparency and repeatability"
suggested revision: reintroduce that sentence or something similar
rationale: The document has now removed all the details about scoring systems. However, this should not prevent the evaluators from documenting whatever scoring system they are using, if any.
Bruce Bailey accept this section with the following suggestions I agree with all of Step 5, but I want to comment that the revised Step 5.d: Provide an Aggregated Score (Optional) represents a pretty significant editorial change from the previous draft. Please consider soliciting another round of comments or extending this feedback period. It's a good change IMHO, just a bit of a surprise.
Martijn Houtepen accept this section
Sarah J Swierenga accept this section
aurélien levy accept this section
Vivienne Conway accept this section
Bim Egan accept this section
Wayne Dick accept this section This covers everything I can imagine.
Kerstin Probiesch accept this section with the following suggestions 5.d: Provide an Aggregated score (Optional).

This section is much better now. Thanks for the improvement.

Prio: strong

Please change "there is currently no single widely recognized metric that reflects the required reliability, accuracy, and practicality." into "there is currently no single metric that reflects the required reliability, accuracy, and practicality."

I feel that "no widely recognized metric that reflects the required reliability, accuracy, and practicality." indicates that there are systems which are reflecting reliaibility and so on and that this metrix is just not "widely recognized". I believe it's a bit contradictory, because if there would be a metric like this, it would be widely recognized.

Prio mild:

I would still prefer to see the whole score-section as statement in the appendices section and not even as optional, as long as there is no metric which reflects goodness criteria for tests.

Kathleen Wahlbin accept this section
Frederick Boland
David MacDonald Methodology Requirement 5.c:

Conformance level <keep-bold>evaluated</keep-bold>
Accessibility support <keep-bold>baseline</keep-bold>

Perhaps add something like
"If an automated test has been conducted provide either have a list of urls, or the number of pages crawled."


Perhaps a sentence should be added such as
"All pages sampled in this evaluation pass WCAG 2"
this could help distinguish this from a WCAG conformance statement.

Detlev Fischer accept this section
Michael Elledge
Ramón Corominas I abstain (not vote)
Howard Kramer I abstain (not vote)

10. Remaing Comments

Provide any remaining comments that you may have.

Details

Responder Comments
Yod Samuel Martin priority: mild (editorial)
location: throughout
current wording: "web pages and web page states", "web page or web page state", versus "web pages" (alone), "web page" (alone), "web pages and states of web pages".
suggested revision: revise each instance of the use of "web page" (alone) and "web pages" (alone), and check whether they should appear together with "or web page state", "and web page states". Both terms are not always needed, but sometimes they are. In addition, replace the single appearance of "states of web pages" with "web page states".
rationale: consistency in the usage of "web pages and web page states" throughout the document.
Bruce Bailey I would categorize my comments as low priority, suggestion only.
Martijn Houtepen
Sarah J Swierenga
aurélien levy
Vivienne Conway
Bim Egan This is a phenomenal piece of collaborative work, it is a huge step toward harmonized and comparable web accessibility evaluations.
Wayne Dick A complete but focused guide to website evaluation, Really a necessary contribution. It needs advertising.
Kerstin Probiesch Good work. I especially like the sampling sections.
Kathleen Wahlbin
Frederick Boland
David MacDonald It's come a long way and I think it is just about ready. It's hard to get a bunch of evaluators to agree, and this for the most part has been accomplished. Congrats all around.
Detlev Fischer
Michael Elledge
Ramón Corominas
Howard Kramer Great work. I know this isn't supposed to show how to apply WCAG 2.0 specifically but the step from "audit" to actually evaluating samples seemed like it needed some clarification or an example. In other words, how would you decide which WCAG 2.0 rules to evaluate. You have the The "five WCAG 2.0 conformance requirements" but I thought some type of example of how you would then apply WCAG 2.0 to conduct the audit would be helpful.

More details on responses

  • Yod Samuel Martin: last responded on 24, June 2014 at 15:43 (UTC)
  • Bruce Bailey: last responded on 24, June 2014 at 18:48 (UTC)
  • Martijn Houtepen: last responded on 25, June 2014 at 12:33 (UTC)
  • Sarah J Swierenga: last responded on 25, June 2014 at 13:35 (UTC)
  • aurélien levy: last responded on 25, June 2014 at 17:41 (UTC)
  • Vivienne Conway: last responded on 26, June 2014 at 10:25 (UTC)
  • Bim Egan: last responded on 27, June 2014 at 12:16 (UTC)
  • Wayne Dick: last responded on 27, June 2014 at 15:54 (UTC)
  • Kerstin Probiesch: last responded on 30, June 2014 at 08:47 (UTC)
  • Kathleen Wahlbin: last responded on 30, June 2014 at 11:09 (UTC)
  • Frederick Boland: last responded on 30, June 2014 at 11:14 (UTC)
  • David MacDonald: last responded on 30, June 2014 at 12:54 (UTC)
  • Detlev Fischer: last responded on 30, June 2014 at 17:49 (UTC)
  • Michael Elledge: last responded on 30, June 2014 at 21:17 (UTC)
  • Ramón Corominas: last responded on 30, June 2014 at 22:08 (UTC)
  • Howard Kramer: last responded on 30, June 2014 at 23:44 (UTC)

Compact view of the results / list of email addresses of the responders

WBS home / Questionnaires / WG questionnaires / Answer this questionnaire