This is a disposition of comments received on the Website Accessibility Conformance Evaluation Methodology (WCAG-EM), Editor Draft of 23 June 2014. This page is intended for internal discussion by the WCAG 2.0 Evaluation Methodology Task Force (Eval TF).
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 41 | Open | Yod Samuel Martin | throughout | "web pages and web page states", "web page or web page state", versus "web pages" (alone), "web page" (alone), "web pages and states of web pages". | revise each instance of the use of "web page" (alone) and "web pages" (alone), and check whether they should appear together with "or web page state", "and web page states". Both terms are not always needed, but sometimes they are. In addition, replace the single appearance of "states of web pages" with "web page states". | consistency in the usage of "web pages and web page states" throughout the document. | After another review, all critical instances seem to correctly use the "web page and web page states" formulation. In some cases, such as in the titles, it has been left out for more clarity and readability. Changing throughout would add significant complexity without evident benefit. |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 1 | Closed | Mike Elledge | first paragraph | This document is one of a series of informative W3C/WAI resources about Evaluating Websites for Accessibility that complements the WCAG 2.0 Documents. | Replace "complements" with "complement" | Improve grammar | Fixed |
| 2 | Closed | Mike Elledge | first paragraph | It does not define additional WCAG 2.0 requirements nor does it replace or supersede it in any way. | Replace "supersede it" with "supersede them" | Improve grammar | Fixed |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 5 | Closed | Mike Elledge | Introduction section | how much knowledge the evaluators have of how the website was or is being developed | how much knowledge evaluators have of the website's development | Improve grammar | Changed to "how much knowledge the evaluators have about the process used to design and develop the website" |
| 6 | Closed | Mike Elledge | Introduction section | feature by feature | feature-by-feature | Improve grammar | No change - we previously decided to change to "feature by feature" |
| 8 | Closed | Howard Kramer | last line of second paragraph of Introduction | This methodology supports common approaches and understanding for evaluating the extent of conformance of websites to WCAG 2.0, though in the majority of use cases it does not directly result in conformance claims | I don't have a specific revision but it was perhaps the only sentence I found confusing due to the wording, in particular the phrase "understanding for" it did not quite make sense. Also, the whole issue of when this evaluation could be used to make conformance claims was no 100% clear to me. Maybe an example here could help | lack of clarity | Reworded to add clarity |
| 7 | Closed | Mike Elledge | Purposes section | Web accessibility monitoring activities used to benchmark or compare the accessibility conformance over time | Web accessibility monitoring activities used to benchmark or compare accessibility conformance over time | Improve grammar | Changed to "Web accessibility monitoring activities used to benchmark and to compare accessibility conformance over time" |
| 4 | Closed | Wayne Dick | Background Reading | The list of "required" reading is somewhat daunting | Reading the Techniques end to end is counter recommendation, but that appears to be the suggestion | The text says "Evaluators using this methodology are expected to be deeply familiar with all the listed resources", which does not necessarily mean reading end to end. We think evaluators need to master all the listed resources for proper conformance evaluation. | |
| 3 | Closed | Yod Samuel Martin | definition of "Template" | From ATAG 2.0 definition for "templates" | add ATAG 2.0 as well to the reference section | its definitions are used in the text | Added |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 9 | Closed | David MacDonald and Kerstin Probiesch | Combined Expertise (Optional) | Though this methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise), using the combined expertise of different evaluators may provide broader coverage of the required skills and help identify accessibility barriers more effectively. While not required for using this methodology, the use of review teams may sometimes be necessary. Using Combined Expertise to Evaluate Web Accessibility provides further guidance on using the combined expertise of review teams, which is beyond the scope of this document. | This methodology can be carried out by an individual evaluator with the skills described in the previous section (Required Expertise). Using the combined expertise of different evaluators may provide an effective way to evaluate content when some of the required expertise is missing from one team member but is possessed by another on the team. While not required for using this methodology, the use of review teams may sometimes be necessary and/or beneficial. Using Combined Expertise to Evaluate Web Accessibility provides further guidance on using the combined expertise of review teams, which is beyond the scope of this document. | the message seems clear...more evaluators on content is better [...] veteran evaluators who have worked in large organizations, also felt that this is not the reality of how things work [...] the current language unnecessarily gives advantages to organizations with teams of evaluators | Section reworded taking the suggested wording into consideration |
| 10 | Closed | Kerstin Probiesch | Combined Expertise (Optional) | Suggested change in comment 9 | Remove "While not required for using this methodology, the use of review teams may sometimes be necessary and/or beneficial. Using Combined Expertise to Evaluate Web Accessibility provides further guidance on using the combined expertise of review teams, which is beyond the scope of this document." | Section reworded | |
| 11 | Closed | David MacDonald | Combined Expertise (Optional) | put the paragraphs "Involving Users (optional)" with disabilities and "Evaluation tools (optional)" above this paragraph in the section | they are more important | It was moved to this location to help better differentiate "combined expertise" from "involving users" |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 12 | Closed | Yod Samuel Martin | Re-Running Website Evaluation, last paragraph | is typically ~50% | replace the tilde with a complete word (e.g. "approximately 50%", "around 50%") | this sign is mathematical jargon, not suitable to be used within running text, and not always screen-reader friendly | Changed to "about half" |
| 14 | Closed | David MacDonald | Re-Running Website Evaluation, last paragraph | ...amount of replaced web pages in a fresh sample is typically ~50% though this could be increased when web pages on a website mostly conform to WCAG 2.0 | Wondering where 50% came from? | Changed to "about half" to reduce emphasis on a particular threshold. These are experiencial values. | |
| 13 | Closed | Wayne Dick | Principle of Website Enclosure | Does the referenced example site exist? | "example.org" is generally used for examples. It is registered as such. |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 15 | Closed | Mike Elledge | 2nd paragraph, Step 1.a | in such a way that for each web page it is unambiguous whether it is within scope or not | so that it is unambiguous that a web page is within scope | Clarity | Changed accordingly |
| 16 | Closed | Mike Elledge | 1st paragraph | define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") to evaluate to | define the target WCAG 2.0 conformance level ("A", "AA", or "AAA") for evaluation | Improve grammar | Changed accordingly |
| 17 | Closed | Mike Elledge | 2nd paragraph, Step 1.c | website is expected to work with, and that is in-line with | website is expected to work with, and that are in-line with | Improve grammar | No change - the "is" refers to the "minimum set of combinations" (singular) |
| 42 | Closed | Mike Elledge | 2nd paragraph, Step 1.c | common expectations on | common expectations for | Improve grammar | Changed accordingly |
| 18 | Closed | Mike Elledge | 3rd paragraph, Step 1.c | extended with the tools that were used in addition | extended with the additional tools | Improve grammar | Changed accordingly |
| 19 | Closed | Mike Elledge | Definition, Step 1.d | agreed on between | agreed by | Improve grammar | Changed accordingly |
| 20 | Closed | Mike Elledge | 4th bullet point, Step 1.d | Explanation of possible solutions to the encountered issues beyond what would be in the scope of website evaluation | Description of possible solutions to the issues encountered beyond the scope of the evaluation | Clarity | Changed accordingly |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 21 | Closed | Wayne Dick | Step 2.b: Identify Essential Functionality of the Website | More examples of essential functionality may be needed like "application for services from an agency." | No change - the suggested example seems too general and vague | ||
| 22 | Closed | Wayne Dick | Step 2.d: Identify Web Technologies Relied Upon | Explicit reference of Dojo or JQuery may be vendor biased. Perhaps use categories like polyfills or Widget Libraries. | No change - text says "libraries and components [...] such as" to list these as mere example. Removing them would make the text too vague and difficult to understand. | ||
| 23 | Closed | David MacDonald | Step 2.d: Identify Web Technologies Relied Upon | "Note: Where possible, it is often also useful to identify the libraries and components used to create the website, such as Dojo, jQuery" ... | perhaps add something like "if a CMS is used it will be helpful to identify it, and it's version, along with a list library components added to it's core framework." | Edited text to address the suggestion |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 30 | Closed | Howard Kramer | first paragraph above "Step 3.a: Include a Structured Sample" | This selection in this step relies initially on the exploration carried out in Step 2: Explore the Target Website. The selection is also continually refined during the following Step 4: Audit the Selected Sample, as the evaluator learns more about the particular implementation aspects of the target website. | depends on meaning of "This selection in this step" | All the other statements that being "During this step" occur right under an overall methodology item (usually highlighted with a box border that it is clear what "step" is being referred to. Here, I wasn't sure if "this selection in this step" referred to 3a alone or the whole of methodology 3. | Changed to "The selection carried out during this step" |
| 24 | Closed | Yod Samuel Martin | Step 3.a: Include a Structured Sample and Step 3.b: Include a Randomly Selected Sample | "The number of required instances of web pages and web page states depends on the particular aspects of the website explained in the previous section factors influencing the sample size" regarding the structured sample and "The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5" in the randomly selected sample. | Clarify the relation between the selection criteria and the sizes of both samples. | It seems there is some inconsistency here. Some aspects explained in the step introduction do not influence step 3.b, but the step takes them into consideration. Conversely, some of the steps do influence step 3.c but they are kept out. For instance "Adherence to development processes" should not affect the size of the structured sample, as it doesn't add more of any of these: common web pages, essential functionalities, types of web page states, web technologies relied upon, or other relevant web pages. However, it should yet influence the size of the random sample, as the introduction states these sites show more (or less) variation in code quality and accessibility. Likewise, a higher required level of confidence may not require a larger structured sample: if I have already considered one page of each combination, adding more does not cover any new essential functionalities, etc. Yet the level of confidence should require a larger random sample. | No change - The factors listed in the introduction influence the size of the structured sample, which is the basis for the size of the random sample. Thus the factors influence the entire sample size. The factors are qualified using "may" or "often" and are merely examples to illustrate how the sample size may be influenced. For example, the text reads "higher confidence in the evaluation results often requires evaluation of a larger sample" to take into account particular situations where adding pages does not increase the confidence. This section is already complex enough so that we are concerned about adding more complexity as proposed. |
| 26 | Closed | Yod Samuel Martin | Step 3.b: Include a Randomly Selected Sample | random | Add a sentence explaining that "random sampling does not imply following a uniform sampling distribution". For instance, following one of the suggested sampling methods, if web pages are extracted from access logs at random, most visited pages would be overrepresented, but this is not necessarily wrong. | Clarify that random does not mean "with equal probability" | No change - we removed references to probability to reduce complexity of the section. The note says "While the random sample need not be selected according to strictly scientific criteria, the scope of the selection needs to span the entire scope of the website (any web page and web page state on the website may be selected), and the selection of individual web page and web page states does not follow a predictable pattern". |
| 27 | Closed | Detlev Fischer | Step 3.b: Include a Randomly Selected Sample | The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5 instances of web pages and web page states. | Amend - possibly in a note - the requirement of a minimum of 5 randomly selected instances of web pages and web page states to account for simple sites that will only require a very small sample. | For some sites, the page sample can be very small: Think of a blog that may just have the default start page view and a bog entry page, with simple text-only blog entries. A minimum of 5 randomly selected pages / states does not make sense in such a case. Another edge case are 'one-pagers' which provide a minimal presentation for a company or product. There is sometimes literally just one page to test. | The requirement of a minimum of 5 web pages has been removed. |
| 28 | Closed | Detlev Fischer | Step 3.b: Include a Randomly Selected Sample | The number of web pages and web page states to randomly select is 10% of the structured sample selected through the previous steps, with a minimum of 5 instances of web pages and web page states. | Clarify what selecting a random sample of states means when creating a sample for web apps or remove the requirement for an additional random sample for web apps. | For web apps, it is not clear how the evaluator should select states randomly. I can think of no procedure as there is usually not a list of URLs, search hits or similar to work from. | No change - the text at the beginning of the section reads "In cases where it is feasible to evaluate all web pages, which is highly recommended, this sampling procedure can be skipped and the "selected sample" in the remaining steps of this evaluation process is the entire website" |
| 25 | Closed | Yod Samuel Martin | Step 3.d: Include Complete Processes, 1st list item | For any web page and web page state selected (...), locate the starting point (...) for the process and include them in the selected sample | For *each* web page and web page state selected (...), locate the starting point (...) for the process and include *it* in the selected sample | grammar agreement and concord | Changed as suggested |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 31 | Closed | Bruce Bailey | Step 4.a: Check All Initial Web Pages | I recommend copying the text from the two cited FAQs rather just having the links | The cited material is not that long, and the FAQ is not as formal and stable as this document. Yes, that means the are more likely to get out of sync, but I think it is okay that this document preserves a snapshot of our best thinking at this moment at time. Working Group Notes are more authoritative and subject to more public review than the FAQ. Plus, one less click, and it is good content, well worth having in multiple places. | Removed references to the FAQ and added a reference to the corresponding section of the Understanding document, which provides the same guidance as the FAQ. | |
| 32 | Closed | Wayne Dick | This is the most difficult to read. It is the most detailed, legalistic section. So, rough reading may just be part of the landscape. | We tried to simplify the section as much as possible | |||
| 33 | Closed | Kerstin Probiesch | Please add explicitly "pass/fail" as scheme for the auditing the samples | I'm missing explict guidance on criteria for passing and failing an evaluation | Clear guidance on this point seems missing from WCAG 2.0 documents in general. We added "WCAG 2.0 Success Criteria are each formulated as a "testable statement that will be either true or false when applied to specific web content"" to the note, which is the clearest text that we could find. We also linked the Understanding Conformance section, in case this is updated with more clarity in the future. | ||
| 34 | Closed | David MacDonald | For example, evaluators may utilize specific testing instructions and protocols that may be publicly documented or only available to the evaluators. | May need to add a sentence that they would need to be able demonstrate that techniques they chose have actually met the SC | Added "that meet the requirements for sufficient techniques" and linked it to the corresponding section in the Understanding document | ||
| 35 | Closed | David MacDonald | Optionally, an evaluation report can specifically indicate Success Criteria for which there is no relevant content, for example, with "not present" | May want to check in with Gregg on this... I personally don't have a problem with it, but it was a pretty hot topic at one point. "Not present" is better than N/A, and may be ok with him | This has been done before publication of the previous Working Draft in January | ||
| 36 | Closed | Howard Kramer | all the sub-headings under 4a. such as "Conforming Alternate Versions" | these sub-headings are styled in italics with an underline. I think this is the only instance where this type of formatting is used for headings or sub-headings. | follow the convention for sub-headings used elsewhere in document. | First, of all, these items don't stand out as headings because their font weight and size is same as underlying paragraphs. Second, they look like links - another source of confusion. | Improved the style to make the h5(!) headings more distinguishable |
| ID | Status | Commenter | Location | Current text | Suggested Change | Rationale | Resolution |
|---|---|---|---|---|---|---|---|
| 37 | Closed | Yod Samuel Martin | Provide an Aggregated score (Optional) | this sentence has been removed since the previous WD "It is also recommended that the scoring approach is documented and made available to the evaluation commissioner along with the report to facilitate transparency and repeatability" | reintroduce that sentence or something similar | The document has now removed all the details about scoring systems. However, this should not prevent the evaluators from documenting whatever scoring system they are using, if any | Added "Whenever a score is provided, it is essential that the scoring approach is documented and made available to the evaluation commissioner along with the report, to facilitate transparency and repeatability" |
| 38 | Closed | Bruce Bailey | Provide an Aggregated score (Optional) | Please consider soliciting another round of comments or extending this feedback period. It's a good change IMHO, just a bit of a surprise. | Revision represents a pretty significant editorial change from the previous draft | This has been discussed during the face-to-face meeting in March. We strongly encourage not to hold up publication for this one issue. | |
| 39 | Closed | Kerstin Probiesch | Provide an Aggregated score (Optional) | there is currently no single widely recognized metric that reflects the required reliability, accuracy, and practicality | there is currently no single metric that reflects the required reliability, accuracy, and practicality | I feel that "no widely recognized metric that reflects the required reliability, accuracy, and practicality." indicates that there are systems which are reflecting reliaibility and so on and that this metrix is just not "widely recognized". I believe it's a bit contradictory, because if there would be a metric like this, it would be widely recognized. | Changed to "there is currently no single metric that is known to address the required reliability, accuracy, and practicality" |
| 40 | Closed | Kerstin Probiesch | Provide an Aggregated score (Optional) | I would still prefer to see the whole score-section as statement in the appendices section and not even as optional, as long as there is no metric which reflects goodness criteria for tests. | No change - it seems better to provide clear guidance on this frequent practice than to remove the guidance. | ||
| 43 | Closed | David MacDonald | Methodology Requirement 5.c | Conformance level <keep-bold>evaluated</keep-bold> | Changed accordingly | ||
| 44 | Closed | David MacDonald | Methodology Requirement 5.c | Accessibility support <keep-bold>baseline</keep-bold> | Changed accordingly | ||
| 45 | Closed | David MacDonald | Methodology Requirement 5.c | Add "If an automated test has been conducted provide either have a list of urls, or the number of pages crawled" | No change - the focus on automated test deviates from the rest of the document | ||
| 46 | Closed | David MacDonald | Methodology Requirement 5.c | Add "All pages sampled in this evaluation pass WCAG 2" | this could help distinguish this from a WCAG conformance statement | No change - disclosing this information publicly is not typical for evaluation statements |