w3c/wbs-design or
by mail to sysreq.
W3C
Site Navigation
Nearby
Results of Questionnaire Approval for draft publication of WCAG-EM
The results of this questionnaire are available to anybody. In addition, answers are sent to the following email addresses: public-wcag-em-comments@w3.org,shadi@w3.org,e.velleman@accessibility.nl
This questionnaire was open from 2013-12-03 to 2013-12-17.
22 answers have been received.
Jump to results for question:
- Abstract
- Introduction
- Using This Methodology
- Scope of Applicability
- Step 1: Define the Evaluation Scope
- Step 2: Explore the Target Website
- Step 3: Select a Representative Sample
- Step 4: Audit the Selected Sample
- Step 5: Record the Evaluation Findings
1. Abstract
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 13 |
| accept this section as draft with the following suggestions | 9 |
| I do not accept this section as draft | |
| I abstain (not vote) |
Details
| Responder | Abstract | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft | |
| Howard Kramer | accept this section as draft | |
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft with the following suggestions | Priority: Mild suggestion; Location: Second paragraph; both first and second sentences Current wording: contains the adjective "primarily" in both sentences. Suggest: one of the instances could use the word "mainly". Rationale: avoids repetition. |
| David MacDonald | accept this section as draft with the following suggestions | |
| Alan Smith | accept this section as draft | |
| Richard Warren | accept this section as draft | |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft with the following suggestions | cf Michael. I think this would relieve my (mild) discomfort with the first sentence. While the rest of the paragraph tries to make clear the scope of the document, "provides guidance on evaluating how well websites conform" is still easily read as "the way" rather than "a way". |
| Alistair Garrison | accept this section as draft with the following suggestions | priority [High] Make the following sentence bold "It is primarily designed for evaluating existing websites, for example to learn about them and to monitor their level of accessibility." Rationale: It is the only place in the whole document that the documents actual purpose is defined - if you miss this sentence, you will not understand the purpose of the document just from the text. Priority [Medium] Remove ", which is addressed by the WCAG 2.0 techniques layer" from the second sentence. You do not need to use or follow WCAG 2.0 techniques. |
| Detlev Fischer | accept this section as draft with the following suggestions | priority: mild I agree to reinstate the section Michael refers to, with a minor change. Current wording (actually wording of previous version quoted by Michael): "...on evaluating the conformance of these selected web pages to the target level of WCAG 2.0 conformance" We have conformance twice here. Suggested revision: "...on evaluating the conformance of these selected web pages to the targeted level of WCAG 2.0" Actually, there is practically no guidance regarding the evaluation of content bar a blanket referral to the Techniques, so perhaps this part should be deleted? |
| Kathleen Wahlbin | accept this section as draft | |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | accept this section as draft | |
| Gregg Vanderheiden | accept this section as draft with the following suggestions | This document has come a long way. Congratulations. Some minor and a couple larger concerns -- but overall a very nice job. =========================== In the ABSTRACT -- one bug " It does not provide instructions for evaluating web content feature by feature, which is addressed by the WCAG 2.0 techniques layer. " is not correct. The techniques are not designed to be used for evaluation. they are neither necessary nor sufficient -- so they cannot be used to evaluate. suggest that the sentence be revised to read: "It does not provide instructions for evaluating web content feature by feature, which is addressed by WCAG 2.0 success criteria." (otherwise very nice) |
| Kerstin Probiesch | accept this section as draft with the following suggestions | Priority: strong I think the term "optional" could be misleading and there is no definition of this in the "Terms and Definitions" section. Suggestion: change "optional" to "not required" or "not mandatory" or "not facultativ" or add a definition of "optional" in the "Terms and Definitions" section, which clearly says that all sections which are marked as optional are not facultativ. |
| Yod Samuel Martin | accept this section as draft with the following suggestions | Detailed comments have been posted to the WCAG-EM comments mailing list. They are archived at <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html> Specifically, comment #1 <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.jlift3cap8ye> applies to the Abstract with strong priority (to be addressed before publication). That comment is aligned to comments already provided by other respondants. |
| Michael Cooper | accept this section as draft with the following suggestions | P: med I miss the text that was removed from the previous version: "It provides guidance on defining parameters for the evaluation scope; on exploring and understanding websites including their key features and functionality; on sampling representative web pages from websites where it is not feasible to evaluate all web pages of a website; on evaluating the conformance of these selected web pages to the target level of WCAG 2.0 conformance; and on documenting and reporting the findings from such evaluation." While that was detailed and could perhaps be shortened, the replacement seems to gloss a little too much. P: strong Putting here since there isn't a place for overall comments. I think the headings should be numbered, e.g., 1.3.1 etc. I found myself getting lost in all the headings, hard to tell from minor size differences what are sub sections and what are new sections, etc. |
2. Introduction
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 12 |
| accept this section as draft with the following suggestions | 7 |
| I do not accept this section as draft | 2 |
| I abstain (not vote) |
Details
| Responder | Introduction | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft with the following suggestions | I suggest that we replace at least some instances of the repetitive phrase "needs to" with "must," "should consider," or "will want to." I understand that there are constraints around some of the words, but the "need to" is so repetitive it becomes distracting. |
| Howard Kramer | accept this section as draft | |
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft with the following suggestions | Priority: Strong suggestion: Location: second paragraph, first sentence. Current wording: "... and highlights considerations that evaluators to apply these steps in the context of a particular website." Suggestion: rephrase "that evaluators to apply ", Rationale: as is, this sentence seems incomplete, (sorry no suggested alternative as I'm not sure what's meant) . |
| David MacDonald | accept this section as draft with the following suggestions | "It also defines how optional conformance claims can be made to cover individual web pages, <add>a</add>series of web pages such as a multi-page form, and multiple related web pages such as a website." Easy Checks - A First Review of Web Accessibility <add>Involving web accessibility experts</add> Involving Users in Web Accessibility Evaluation Selecting Web Accessibility Evaluation Tools Using Combined Expertise to Evaluate Web Accessibility |
| Alan Smith | accept this section as draft | |
| Richard Warren | accept this section as draft | |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | Editor: * cf Bim * second paragraph: "though in the majority of use cases it does not directly result into conformance claims." -> "though in the majority of use cases it does not directly result in conformance claims." * Add commas to first sentence of Background Reading: "The information below, related to web accessibility essentials, evaluation, and WCAG 2.0, is important for using this methodology:" Suggestion to address Kerstin's comment: "The methodology relies on WCAG 2.0 techniques such as the Techniques for WCAG 2.0 documented by W3C/WAI, but is not limited to this set of techniques." -> "The methodolody relies on evaluating against techniques for meeting WCAG 2.0 success criteria, such as the Techniques for WCAG 2.0 documented by W3C/WAI, but does not require this or any other specific set of techniques." | |
| Alistair Garrison | accept this section as draft with the following suggestions | Priority [High] "Web Page States" - should be "Web Page DOM States" - it is important to tell the reader that it is the DOM which is changing, after the page has loaded, not the web page (which could be construed as web page source code). Priority [Low] Sentence "considerations that evaluators to apply", should be "considerations that evaluators should take onboard when applying". Sentence "result into conformance claims" should be "result in conformance claims". The phase is "result in", not "result into" - this needs a global replace. |
| Detlev Fischer | accept this section as draft with the following suggestions | Cf. Loretta's wording suggestion: "The methodolody relies on evaluating against techniques for meeting WCAG 2.0 success criteria, such as the Techniques for WCAG 2.0 documented by W3C/WAI, but does not require this or any other specific set of techniques." Would we want to require that techniques used for evaluation need to be documented somewhere in the public domain? If so, I would suggest: "The methodolody relies on evaluating against publicly documented techniques for meeting WCAG 2.0 success criteria, such as the Techniques for WCAG 2.0 documented by W3C/WAI, but does not require this or any other specific set of techniques." -------------- Suggestion by Kerstin Probiesch to remove section "Review teams": This section is optional and clearly states that evaluations can be carried out by individuals. Even experienced evaluators can miss things, so a second pair of eyes is often beneficial. I see no need to remove the section. Perhaps change wording instead of deletion? Current wording: "While not required for using this methodology, the use of review teams is highly recommended when performing an evaluation of a website." Suggested wording: While not required for using this methodology, the use of review teams is often beneficial when performing an evaluation of a website. -------------- Priority: high Location: Terms and Definitions, Entry Web Page States Current wording: "Some web page states are ancilliary or treated similarly to individual web pages in the context of this methodology." Suggested revision: "In the context of this methodology, web page states can be treated as ancilliary to pages (i.e., recorded as additional state of a page in a page sample) or as individual web pages." Rationale: Current wording is rather fuzzy. I admit that the content in brackets may go beyond what you would expect in the definition of a term but "ancilliary to pages" alone does not indicate that states should be recorded with their base page during exploration and sampling. |
| Kathleen Wahlbin | accept this section as draft | |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | accept this section as draft with the following suggestions | I am wondering if the white-box, black-box, grey-box testing mention is relevant. Second sentence is long and weird: "This methodology, the Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0, describes the steps that are common to website conformance evaluation processes and highlights considerations that evaluators to apply these steps in the context of a particular website." I would change for the following: "The Website Accessibility Conformance Evaluation Methodology (WCAG-EM) 1.0 describes the steps that are common to website conformance evaluation processes. It highlights considerations that evaluators need to take into account to apply these steps in the context of a particular website." Remove the "to" in the third sentence: "Following this methodology will help evaluators apply good practice, avoid commonly made mistakes, and achieve more comparable results." |
| Gregg Vanderheiden | I do not accept this section as draft | A few smaller things and one major 1) Sentence 1 of paragraph 2 ends with "..... and highlights considerations that evaluators to apply these steps in the context of a particular website. " which does not grok. It seems to be missing a word or something. 2) Last sentence of PP 2 ends: "...though in the majority of use cases it does not directly result into conformance claims." suggest either "result in confomance claim language" or "resolve into conformance claims" (though that is a bit ambiguous) or something. But it currently is unclear what it is trying to say. 3) Remove "ALSO" from third paragraph. 4) I think removing the first WCAG 2.0 from this sentence will make it clearer that the techniques you are referring to are more than the WCAG 2.0 WG defined techniques. ( "WCAG 2.0 techniques" can very easily be misread "WCAG 2.0 WG techniques". "The methodology relies on WCAG 2.0 techniques such as the Techniques for WCAG 2.0 ..." ================================ UNDER "Relation to WCAG 2.0 Conformance Claims =============================== Sentence 1 reads "WCAG 2.0 defines conformance requirements for individual web pages that are known to satisfy each conformance requirement, rather than for entire websites. It also defines how .... " doesn’t read well. Not clear what it is saying. I THINK you might mean "WCAG 2.0 defines conformance requirements for individual web pages (and in some cases, sets of web pages), but does not describe how to evaluate entire websites. WCAG 2.0 also defines how... " ==================== UNDER TERMS AND DEFINITIONS ==================== I understand the purpose of the term "core functionality" but its use bothers me very much, since it has been abused so completely in every other domain of accessibility. For the W3C to define or endose the term is extremely troubling. I would advise talking about "High Frequency pages" -- and "Pages needed to complete processes". And woud really speak against the use of the term CORE. It is not needed, and it is extraodinarily dangerous - both for web page evaluation and dangerous to accessibilty overall. <this is the only show stopper problem with this section. > < SEE " STEP 2" question below for a resolution to this issue - involving the use of a different term. For example DEPENDENT COMPONENTS. The definition here would be DEPENDENT COMPONENTS COMPONENTS of a website that, if removed, fundamentally changes the use, purpose, OR FUNCTIONALITY of the website for users. This includes information that users of a website refer to and tasks that they carry out to perform this functionality. Note: Examples of functionality include "selecting and purchasing a product from the shop area of the website", "filling and submitting the form provided on the website", and "registering for an account on the website". Note: Other PARTS OF THE WEBSITE ARE not excluded from the scope of evaluation. The term "DEPENDENT COMPONENTS" is intended to help identify critical web pages and include them among others in an evaluation. (one COULD use DEPENDENT FUNCTIONALITY if "components" causes problems but DEPENDENT COMPONENTS is better for a number of reasons). But CORE isn't really correct and is quite dangerous. |
| Kerstin Probiesch | I do not accept this section as draft | # Priority: strong Location: "The methodology relies on WCAG 2.0 techniques such as the Techniques for WCAG 2.0 documented by W3C/WAI, but is not limited to this set of techniques." Suggestion: Delete this sentence. This is in contradiction with the idea and the concept of the Techniques Document. Not if a technique is used is important but wether an SC is met or not. Therefore a methodology for evaluating can't rely on techniques. What the WG has published here: http://www.w3.org/WAI/WCAG20/wcag2faq.html#techsnot is also relevant for evaluators. Even with "but is not limited to this set of techniques." the paragraph places too much emphasis on the techniques. |
| Yod Samuel Martin | accept this section as draft | Detailed comments to the Introduction are available at <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.cwp3pfb1yttc> , they are all minor comments that can be addressed after the publication of the Working Draft. |
| Michael Cooper | accept this section as draft with the following suggestions | P: high I miss the para that was removed from Scope: "This methodology does not replace the need for quality assurance measures that are implemented throughout the design, development, and maintenance of websites to ensure their accessibility conformance. Following this methodology may not identify every possible occurrence of support for or violation of WCAG 2.0 conformance. It is one possible approach for evaluating the conformance of existing websites to WCAG 2.0 in different evaluation contexts, including self-assessment and third-party evaluation. It is primarily intended for use by experienced accessibility evaluators." As best I can follow the paper trail, this was removed in response to a comment requesting simplifying the wording. However, I seem to recall the disclaimer this paragraph provides was very important to the WCAG WG. I don't think it should have been simplified out, unless I mis-remember the WCAG view. I think the para needs to be restored. I'm not asking for the Scope section to be restored, just stick the para somewhere. It could use some simplification, just not all the way to nil. P: mild I think "Website" should be spelled "web site". I think it's usually two words, and Web is only capitalized when used as a noun. |
3. Using This Methodology
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 12 |
| accept this section as draft with the following suggestions | 6 |
| I do not accept this section as draft | 1 |
| I abstain (not vote) |
Details
| Responder | Using This Methodology | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft | |
| Howard Kramer | accept this section as draft with the following suggestions | Priority: medium Location: 6th bullet item Current wording: : Web accessibility monitoring activities who want to benchmark or compare the accessibility conformance of websites. suggested wording: (would combine with previous bullet item to read as follows): People and services that want to monitor, benchmark, and compare the accessibility conformance of websites over time or between different websites. (I would also add greater line spacing between all bullet items. |
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | ||
| Alan Smith | accept this section as draft | |
| Richard Warren | accept this section as draft with the following suggestions | I would like to re-arrange the text in the section "Evaluation Tools" to make it more clear that automated testing will never be sufficient on its own. I suggest changing the second sentence to read "Many checks are not automatable, however, web accessibility evaluation tools can significantly assist evaluators during the evaluation process and contribute to more effective evaluation" Sentence 4 would not then be required. |
| Kathleen Anderson | accept this section as draft with the following suggestions | Priority: medium Location: 5th paragraph Current Heading: Involving Users {Optional) Suggest: Please consider changing to required. Rationale: Testing a website for accessibility should be given the same importance as testing a website for all other functionality, and no website should go live without user testing. |
| Loretta Guarino Reid | accept this section as draft | |
| Alistair Garrison | accept this section as draft with the following suggestions | Priority [Low] First sentence "evaluation of websites with WCAG 2.0" should be "evaluation of websites against WCAG 2.0". Priority [Low] Change "Required Expertise" section to read: It is assumed that users of this methodology have a deep working knowledge of all topics covered in the Background Reading section, especially: - WCAG 2.0; - accessible web design; - assistive technologies and how people with different disabilities use the Web; - WCAG 2.0 evaluation techniques; and - tools which can help identify potential barriers for people with disabilities. Reason - It makes it easier for people to read. |
| Detlev Fischer | ||
| Kathleen Wahlbin | accept this section as draft with the following suggestions | priority: medium location: Particular Types of Websites > Website in Multiple Versions current wording: Note: Websites using responsive design techniques (i.e. adapting the presentation according to user hardware, software, and preferences) as opposed to redirecting the user to a different location are not considered to be independent website versions. suggested revision: Note: Websites using responsive design techniques (i.e. adapting the presentation according to user hardware, software, and preferences) as opposed to redirecting the user to a different location are not considered to be independent website versions unless the site at the different breakpoints utilizes different code. In that case, the website at the different breakpoints could be considered as individual websites each for evaluation. rationale: clarify what is needed for evaluating responsive websites |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | accept this section as draft | |
| Kerstin Probiesch | I do not accept this section as draft | # Priority: strong suggestion Location: Required expertise Current wording: "Users of this methodology are assumed to be knowledgeable of WCAG 2.0" Suggested revision: "Users of this methodology are assumed to be knowledgeable of WCAG 2.0 which includes the Success Criteria, the role of techniques http://www.w3.org/TR/WCAG20-TECHS/intro.html) in general and the overall concept of WCAG 2.0". # Priority: strong Location: Review Teams (Optional) Current Wording: Whole section Suggested revision: Delete the section "Review Teams" Rationale: That Review Teams are helping to identify barriers "more effectively" is at a first glance a logical statement, but because there are no systematic data which provide evidence it is just a thesis or an opinion. Two examples: a. is the evaluation of a Review Team with some months experience "more effectively" than the evaluation carried out by an individual evaluator with 5, 6, 7 years or more experience? b. what if a Review Team are testing their own interpretation: are the results of this Review Team better than the results of an individual evaluator who follows for examples decisions of the WCAG Working Group? As this is an opinion which is not substantiated by systematical data it can't be "highly recommended". When we speak about evaluation we of course speak about accessibility, but we also speak about business models and about a lot of money. Evaluators are in competition with one another Individual Evalators are in competition with Testing Organizations. WCAG-EM is for example explicitly mentioned in the "European Accessibility Requirements for Public Procurement of Products and Services in the ICT Domain" especially in this document: http://www.etsi.org/deliver/etsi_en/301500_301599/301549/01.00.00_20/en_301549v010000c.pdf (page 90, note 2). It is likely that the preference of Review Teams over Individual Evaluators will have a negative impact on the free competition. We should give guidance on how to evaluate web pages and should obstain from all (as long as they are not facts) what is likely to prefer one group of evaluators - even in a "mild" form. |
| Yod Samuel Martin | accept this section as draft with the following suggestions | Comment #7 archived online applies to this section with moderate priority <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.40et18ysw45p> (regarding the use of "aging-related impairments") |
| Michael Cooper | accept this section as draft |
4. Scope of Applicability
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 17 |
| accept this section as draft with the following suggestions | 2 |
| I do not accept this section as draft | |
| I abstain (not vote) |
Details
| Responder | Scope of Applicability | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft | |
| Howard Kramer | accept this section as draft | |
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | ||
| Alan Smith | accept this section as draft | |
| Richard Warren | accept this section as draft | |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft | |
| Alistair Garrison | accept this section as draft | |
| Detlev Fischer | ||
| Kathleen Wahlbin | accept this section as draft | |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | accept this section as draft | |
| Kerstin Probiesch | accept this section as draft with the following suggestions | # Medium Location: Principle of Website Enclosure. Current Wording: "When a target website is defined for evaluation, it is essential that all web pages, web page states, and functionality within the scope of this definition are considered for evaluation. Excluding such aspects of a website from the scope of evaluation would likely conflict with the WCAG 2.0 conformance requirements". Suggestion and Rationale: For clarification it should be made clear that also PDF and other formats have to be considered for evaluation and that not only HTML is meant. |
| Yod Samuel Martin | accept this section as draft with the following suggestions | Detailed comments applicable to this section are archived here <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.mwt6lvp69oth>, although some of them are prioritary, they are deemed to be addressed after the publication of the Working Draft, **except for these**, which should be addressed before publication, with strong priority. - Comment #14 (on black-box testing) <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.alic0avy866s> - Comment #15 (on conflicting use of third-party contents) <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.m3f6msx8kffh> |
| Michael Cooper | accept this section as draft | P: mild Think it would be useful to add examples of what is *not* in scope, e.g., single pages, aggregated sites, etc. This can be a comment for the next round, though, not something that needs to be addressed before publication of this round. |
5. Step 1: Define the Evaluation Scope
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 13 |
| accept this section as draft with the following suggestions | 4 |
| I do not accept this section as draft | |
| I abstain (not vote) | 1 |
Details
| Responder | Step 1: Define the Evaluation Scope | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft with the following suggestions | Suggest to change the wording of Step 1b to avoid the cognitive dissonance of suddenly being referred to a step that is two steps ahead of where we are. Perhaps an introduction saying something like "A clear definition of goals at this point will bring greater clarity to subsequent steps in the evaluation process. For example, clear evaluation goals are particularly relevant...etc" |
| Howard Kramer | ||
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | ||
| Alan Smith | accept this section as draft | |
| Richard Warren | I abstain (not vote) | Steps 1d and 1e. I understand, and approve of, the concept but I am not sure if the wording is right and cannot think of how to improve it at this stage. |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft | |
| Alistair Garrison | accept this section as draft | |
| Detlev Fischer | ||
| Kathleen Wahlbin | accept this section as draft | |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | accept this section as draft with the following suggestions | Step 1d says: "W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some evaluators might use other methods (inline with the requirements for custom techniques) to evaluate conformance to WCAG 2.0.W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria. " However, techniques are not designed or provided for evaluation. They are provided as example ways to meet SC. Suggest changing this to: "W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that provide one way to meet the WCAG 2.0 Success Criteria. W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that ONE WAY TO MEET THE WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some AUTHORS might use other methods (IN LINE with the requirements for custom techniques) to CREATE conformance to WCAG 2.0 AND EVALUATORS SHOULD ACCEPT VIABLE ALTERNATIVE TECHNIQUES AS WELL. ========================================= |
| Kerstin Probiesch | accept this section as draft with the following suggestions | Priority: strong Step 1d. I strongly support Gregg's suggestion for changing this step from "W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some evaluators might use other methods (inline with the requirements for custom techniques) to evaluate conformance to WCAG 2.0.W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that help evaluate conformance to WCAG 2.0 Success Criteria." to what Gregg suggested: "W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that provide one way to meet the WCAG 2.0 Success Criteria. W3C/WAI provides a set of publicly documented (non-normative) Techniques for WCAG 2.0 that ONE WAY TO MEET THE WCAG 2.0 Success Criteria. However, it is not necessary to use these particular techniques (see Understanding Techniques for WCAG Success Criteria). Some AUTHORS might use other methods (IN LINE with the requirements for custom techniques) to CREATE conformance to WCAG 2.0 AND EVALUATORS SHOULD ACCEPT VIABLE ALTERNATIVE TECHNIQUES AS WELL." |
| Yod Samuel Martin | accept this section as draft with the following suggestions | Detailed comments applicable to this section are archived here <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.be0hmyad5e5u>although some of them are prioritary, they are deemed to be addressed after the publication of the Working Draft, **except for this**, which should be addressed before the publication of the Working Draft: - Comment #20 <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.sjtc6q7p3chn> : the comment is indeed asking for public feedback on Step 1.c, so it can only be either honoured or dismissed before the draft is published. |
| Michael Cooper | accept this section as draft |
6. Step 2: Explore the Target Website
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 13 |
| accept this section as draft with the following suggestions | 5 |
| I do not accept this section as draft | |
| I abstain (not vote) |
Details
| Responder | Step 2: Explore the Target Website | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft with the following suggestions | Step 2b"common functionality" is particularly jarring here. Common to what? Perhaps the phrase can be replaced with "critical functionality," although I do not understand why "basic" is not acceptable. |
| Howard Kramer | ||
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | ||
| Alan Smith | accept this section as draft | |
| Richard Warren | accept this section as draft | |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft | |
| Alistair Garrison | accept this section as draft | |
| Detlev Fischer | ||
| Kathleen Wahlbin | accept this section as draft with the following suggestions | priority: mild location: Step 2a and Step 2c current wording: Methodology Requirement 2.a: Identify the common web pages, including web page states, of the target website. Explore the target website to identfy its common web pages, which may also be web page states in web applications. Typically these are linked directly from the main entry point (home page) of the target website, and often linked from the header, navigation, and footer sections of other web pages. The outcome of this step is a list of all common web pages and web page states. suggested revision: Methodology Requirement 2.a: Identify the common web pages of the target website. Explore the target website to identify its common web pages, which may also be web page states in web applications. Typically these are linked directly from the main entry point (home page) of the target website, and often linked from the header, navigation, and footer sections of other web pages. The outcome of this step is a list of all common web pages. rationale: Both of these two section talk about web page states. Remove this from step 2a since it is detailed in step 2c. This would simplify the steps. The word "identfy" is not spelled correctly. |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | accept this section as draft with the following suggestions | Again the concept of CORE appears. I think that what you seek can be accomplished by changing the terminology to "DEPENDENT COMPONENTS". This would achieve your goal and avoid the CORE FUNCTIONALITY landmine. Step 2.b: Identify DEPENDENT COMPONENTS of the Website Methodology Requirement 2.b: Identify an initial list of DEPENDENT COMPONENTS of the target website. Explore the target website to identify its DEPENDENT COMPONENTS. While some DEPENDENT COMPONENTS will be easy to identify, others will need more deliberate discovery. For example, an online shop is expected to have a payment function though it might be less easy to identify that it also has a currency conversion function that is ESSENTIAL to the particular context of the online shop - AND THAT THE FULL FUNCTIONING OF THE SHOP IS DEPENDENT ON IT. The outcome of this step is a list of DEPENDENT COMPONENTS that users MUST BE ABLE TO USE on the website, for example: Selecting and purchasing products from web shop; Filling and submitting the survey forms; Registering for an account on the website. Note: The purpose of this step is not to exhaustively identify all functionality of a website but to determine those COMPONENTS that the purpose and goal of the target website ARE DEPENDENT ON. This will inform later selection of web pages and their evaluation. Other functionality will also be included in the evaluation but through other selection mechanisms. |
| Kerstin Probiesch | accept this section as draft with the following suggestions | Priority: strong Location: Step 2c Current Wording: "Web pages using varying technologies such as HTML, CSS, JavaScript and WAI-ARIA;" Suggestion: "Web pages using varying technologies and formats such as HTML, CSS, JavaScript, WAI-ARIA, PDF etc." |
| Yod Samuel Martin | accept this section as draft with the following suggestions | Detailed comments to Step 2 are available at <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.9w3za3bx0cyg> , even though some comments are moderately prioritary, they can all be addressed after the publication of the Working Draft. |
| Michael Cooper | accept this section as draft |
7. Step 3: Select a Representative Sample
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 14 |
| accept this section as draft with the following suggestions | 4 |
| I do not accept this section as draft | 1 |
| I abstain (not vote) |
Details
| Responder | Step 3: Select a Representative Sample | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft | |
| Howard Kramer | ||
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | accept this section as draft with the following suggestions | typo distinctinstance Spelling Constistent Spelling 3e Nethods Representative Sample Step 3 There are no example baselines of the number of pages to sample. No ballpark and this could result in much variation across evaluators and jurisdictions. I think there are two ways to improve this and provide better guidance that will allow more consistent results across jurisdictions. 1) use the “size of website” criteria as baseline and provide a statistically relevant sample recommendations, such as those used by the Canadian Government in response to the Donna Jodhan Case. Size of the website — websites with more web pages typcially require a larger sample to evaluate <add> (for example a statistically relevant size with a the following sample sizes would give a 90% confidence level, +/- 10% error: If the website has web pages numbering: ≤60, then a sample of 32 <100 then a sample size of 47 <200 then a sample size of 56 <500 then a sample size of 60 <1000 then a sample size of 64 <5000 then a sample size of 67 >5000 then a sample size of 68 These are established international statistical sample sizes. Then with that baseline we can talk about increasing (or decreasing) the sample size based on the other factors of complexity, age, consistency etc... If we feel this is too prescriptive then simple providing 3 or 4 examples might make this question more tangible. === I think we should have an added section in the different types samples 3f templates. Choose a page using each type of template. There is some implicit mention early on about templates but they seem drop off in this important section where I think they should be included explicitly. |
| Alan Smith | accept this section as draft | |
| Richard Warren | I do not accept this section as draft | The implication is now that ALL websites require selecting a representative sample. This is not so. Ideally all pages should be evaluated. If, and only if, the site is too large for a full evaluation to be viable should we need to select a representative sample. Also I am not happy with the step 3e (select a random sample). There have been many discussions about this and no definitive conclusion. I suggest that we leave the method of selecting the random sample up to the evaluator - who MUST record his/her method in step 5 so that it can be replicated. |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft | |
| Alistair Garrison | accept this section as draft | |
| Detlev Fischer | Priority: medium Location: Step 3.a: Include Common Web Pages of the Website Methodology Requirement 3.a: Include all common web pages, including web page states, into the selected sample. Current wording: "Include all common web pages and web page states that were identified in Step 2.a: Identify Common Web Pages of the Website into the selected sample for evaluation." Suggested revision: "Include all common web pages and web page states that were identified in Step 2.a: Identify Common Web Pages of the Website into the selected sample for evaluation. Web page states and the method to call them up can be recorded together with the base page. Alternatively, web page states can constitute a separate page." Rationale: I think we should make it clear to evaluators that they will often need to call up additional states to be evaluated, and the idea of replicability requires that the way to call up states is documented. I am not happy with my own wording though, so maybe someone else has a better suggestion? | |
| Kathleen Wahlbin | accept this section as draft | |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | accept this section as draft with the following suggestions | TYPO: need a space between "distinctinstance" in requirement 3c ALSO: replace CORE FUNCTIONALITY with DEPENDENT COMPONENTS In step 3d you don't actually say anywhere that ALL processes should be included in the sample. Just that if you select one page in a process- you must include the whole process. Maybe do DON'T WANT to say that all processes are included. (some site may have an almost endless number of them). Perhaps you should put this step AFTER the random selection step and then say: "IF ANY OF THE ABOVE PROCESSES HAVE PUT A PAGE INTO YOUR EVALUATION SAMPLE, WHERE THAT PAGE IS PART OF A PROCESS, THEN ALL OF THE PAGES INVOLVED IN THAT PROCESS MUST ALSO BE ADDED TO THE EVALUATION SAMPLE." (but not in all caps of course) |
| Kerstin Probiesch | accept this section as draft with the following suggestions | I agree with David especially the following "1) use the “size of website” criteria" and the concrete examples for sample sizes. This will reduce possible variations and I believe will increase consistency, which is necessary for reliable and valide results. |
| Yod Samuel Martin | accept this section as draft with the following suggestions | Detailed comments to Step 3 are available at <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.8tw4knxsa4r2> , even though some comments are moderately prioritary, they can all be addressed after the publication of the Working Draft. |
| Michael Cooper | accept this section as draft |
8. Step 4: Audit the Selected Sample
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 13 |
| accept this section as draft with the following suggestions | 4 |
| I do not accept this section as draft | 2 |
| I abstain (not vote) |
Details
| Responder | Step 4: Audit the Selected Sample | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft with the following suggestions | Step 4 sentence revision: "Many web pages and web page states in the sample will have components that occur repeatedly, such as the header, navigation bars, search form, and others. Typically these components do not need to be re-evaluated for each occurrence unless they appear or behave differently,..." "...According to WCAG 2.0, Success Criteria for which..." Step 4.b sentence revision: "...only the content that changes within the process..." "...and other interactions are part..." Step 4.e sentence revision: "...does not contain types of content and outcomes..." "...outcomes from evaluating the randomly selected sample should not present new findings to those already identified..." |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft | |
| Howard Kramer | ||
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | I do not accept this section as draft | I think there is some ambiguity between baseline WCAG conformance and good usability/ best practices. Although I almost always include people with disabilities in evaluations, and they often identify things that can be improved on a web site's accessibility/usability, it rarely results in identifying strict WCAG failures that were not found in the "expert review". I think this sentence could be improved to correct the ambiguity. <snip>"Involving people with disabilities and people with aging-related impairments helps identify additional accessibility barriers that are not easily discovered by the evaluators alone."</snip> Let's leave evaluators out of this sentence. "Involving people with disabilities and people with aging-related impairments provides a clearer picture of how the site actually works for people with disabilities. It can result in a more rounded and useful assessment, and therefore better usability and overall accessibility of the site." === <snip> Note:... In such cases, an evaluator may use an identifier such as "not applicable" to denote the particular situations where Success Criteria are satisfied because no matching content is presented.</snip> We may want to check with Gregg about this, I think he felt pretty strongly about not having “N/A” on conformance claims, although I don’t personally have particular issue about it. I think we should listen to his rational. |
| Alan Smith | accept this section as draft | |
| Richard Warren | accept this section as draft | |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft with the following suggestions | Agree with David that we should recommend a different term than "not applicable". "No matching content"? "satisfied by omission"? "automatically satisfied"? |
| Alistair Garrison | accept this section as draft | |
| Detlev Fischer | ||
| Kathleen Wahlbin | accept this section as draft with the following suggestions | priority: strong location: none/missing current wording: n/a suggested revision: Add a new step to include information on how to use assistive technology in testing. This step could be marked as optional. rationale: In section 1.c, we ask people to define the web browser, assistive technologies and other user agents for which features provided on the website are to be accessibility supported but then we do not have language about this in the auditing of the selected sample. I feel that this should be included. |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | accept this section as draft with the following suggestions | ================ STEP 4 ================ Please do not use the term "Not Applicable" (or N/A). It is not necessary and it is very dangerous to introduce the term. Every time I have see it introduced, it was used far beyond wha tit was defined for -- and people started marking things NA because it wasn’t met (really), because they didn’t think it was imporatant, because the site wasn’t designed for the group, because they didn’t know of a way to do it, because the techniques defined by WCAG wouldn’t work on this page, because they were using a different technology, and others. The working group went to great lengths to keep that term away from the evaluation process. ALL SC apply to a website. If the site does not have MEDIA then all of the pages meet the MEDIA SC. The SC are not NA. Again, once evaluators start labelling things NA -- all sorts of other reasons are used for the designation. Suggest changing in 4 - NOTE: In such cases, an evaluator may use an identifier such as "not applicable" to denote the particular situations where Success Criteria are satisfied because no matching content is presented. to In such cases, an evaluator may use an identifier such as "not present" to denote the particular situations where Success Criteria are satisfied because no matching content is presented. ============================= REQUIREMENT 4e ============================= I do not understand this one at all. "The evaluation outcomes of the structured and random sample correlate when they are sufficiently large and representative. While the individual occurences of WCAG 2.0 Success Criteria will vary between the samples, the randomly selected sample should not show new types of content (as identified in Step 2: Explore the Target Website), and the outcomes from evaluating these randomly selected sample should not show new findings to those that were already determined in the structured sample. When the correlation fails then evaluators need to select additional web pages and web page states (as per Step 3: Select a Representative Sample), to reflect the newly identified types of content and outcomes. The outcomes of Step 2: Explore the Target Website might need to be adjusted accordingly as well. This step is repeated until the structured sample is adequately representative." If the two correlate, then there would be no reason to do them both. The whole reason for doing both structured and random is because they will provide different samples. Also, the level of correlation is not specified. Nothing correlates 100% except a sample and itself. And 50% is chance. Two unrelated things will correlate at .5 or so (on average). (and how many people on the WCAG WG know how to calculate a correlation of this type?) ALSO - if the structured sample has a particularly troublesome type of content, the random sample evaluation results will always be different. And if there is no more of that type in the site, you could random sample forever and never get the random sample eval results to be the same as the structured sample (it would always be better). Finally, the two eval results will never be the same ever unless they were the same sample or they were both perfect, (or it was one of those 1 in 10,000 chances). If the purpose of this is to just determine if the structured sample includes all of the page and content types -- then I would avoid the word correlation and just say: Methodology Requirement 4.e: Check that each web page and web page state in the randomly selected sample does not show types of content and outcomes that are not represented in the structured sample. The purpose of this step is to ensure that the overall sample includes all of the page types. This is done by comparing the structured and random sample to see there are new types of content (as identified in Step 2: Explore the Target Website) found in the random sample than in the structured sample. If there are then the structured sample should be expanded and a new random sample taken until the random sample produces no new types of content. At this point it is a fair assumption that the sample is representative of the website (absent any other knowledg to the contrary). DOES THIS DO WHAT YOU ARE TRYING TO ACHIEVE? |
| Kerstin Probiesch | I do not accept this section as draft | I agree with David's comments. |
| Yod Samuel Martin | accept this section as draft | |
| Michael Cooper | accept this section as draft |
9. Step 5: Record the Evaluation Findings
Summary
| Choice | All responders |
|---|---|
| Results | |
| accept this section as draft | 13 |
| accept this section as draft with the following suggestions | 3 |
| I do not accept this section as draft | 4 |
| I abstain (not vote) |
Details
| Responder | Step 5: Record the Evaluation Findings | |
|---|---|---|
| Roberto Scano | accept this section as draft | |
| Michael Elledge | accept this section as draft | |
| Martijn Houtepen | accept this section as draft | |
| Sarah J Swierenga | accept this section as draft | |
| Sharron Rush | accept this section as draft | |
| Howard Kramer | ||
| Paul Schantz | accept this section as draft | |
| Bim Egan | accept this section as draft | |
| David MacDonald | I do not accept this section as draft | Conformance level satisfied: Level A, AA or AAA as per Step 1.b. Define the Conformance Target; I don't think an organization can claim absolute WCAG conformance based on this methodology, as this phrase appears to indicate. At least not as it is defined currently in WCAG which requires EVERY page to conform. I think it might expose them to legal action. I think it should be reported like statistics are reported. "We report Conformance Level (Level A, AA, AAA) with a fair degree of confidence, based on the WCAG Evaluation Methodology Framework" with a link to the this document. The report should also include another bullet. -Contact information to report any accessibility issues on pages that may not have been evaluated. ===== Grammar Currently <add>comma</add> the following performance scoring approaches are provided by this methodology: |
| Alan Smith | accept this section as draft with the following suggestions | In the section "Records for evaluation specifics could include any of the following:"Specific test credentials (useids, etc.) required to replicate a unique data set or workflow." Often data sets are unigue to a particular userid and drive a web page design, especially in financial institution web pages/apps. Since we do have a bullet for Description of settings...Perhaps "test credentials" wording would be best served by putting it in that bullet? |
| Richard Warren | accept this section as draft | |
| Kathleen Anderson | accept this section as draft | |
| Loretta Guarino Reid | accept this section as draft | |
| Alistair Garrison | accept this section as draft with the following suggestions | Priority [High] Step 5.c: Provide an Evaluation Statement Currently there is no difference between the official Conformance claim text and what people put in the statement. We cannot let people put "Conformance level satisfied" - because in the majority of cases the methodology will not allow this. My proposal would be to change "Conformance level satisfied" to "Conformance level evaluated" - with some boiler plate text to cover the outcomes i.e. No issues where discovered, etc... |
| Detlev Fischer | accept this section as draft with the following suggestions | I add a reply to Gregg's suggestion to drop "Step 5.d: Provide a Performance Score (Optional)" altogether. I agree with Gregg that scoring may not adequately represent the overall practical accessibility of content, especially in edge case examples he provides. Scores fulfil another important role, however. In EVAL TF, several participants have emphasised that site owners / developers commissioning accessibility testing are keen to be able to record and track where problems (SC Failures) exist in order to manage the a11y part of their development, prioritise remediation, and check whether things have improved when re-testing. In *this* application context of WCAG-EM, the question whether the site as a whole will ultimately conform is irrelevant (we know that most likely it won't, ever). Still, it is important to engage in the real-world development process in a meaningful way. For cases such as Gregg's first example, the site-wide score 86 out of 87 would indeed poorly reflect the site's practical accessibility, but for a developer, it shows that there is but one particular issue that needs attention. A page based score in the second example with few minor issues distributed widely over SCs would arrive at a very good average SC values across the sample (say 0.99, where 1.0 would be conformance for all pages in the sample). Again, the developer would know that the site is basically in good shape, but lots af small issues still need to be addressed. Useful information indeed. For scores to adequately reflect the actual accessibility of the site tested, we would need to include the criticality of issues. Lacking that, scores will neverthelsss often be instructive from a site development point of view. In my view, the section should stay (it is optional, after all). Perhaps we need to point out what the resulting scores can be used for, and that they may not reflect the overall practical accessibility of the site tested. ========== Priority: high Location: Step 5.d: Provide a Performance Score (Optional) Suggested revision: This optional step suggests three different scoring approaches: per web site, per web page, and per instance. The first two are pretty uncontroversial. The third approach suggests a score calculation based on instances on the page "for which each WCAG 2.0 Success Criterion is applicable". In the last Eval TF Telco, there was a consensus decision (with one abstention) to remove this third type of scoring method: http://www.w3.org/2013/12/05-eval-minutes.html In a future version of WCAG-EM, we might express that there are different ways of calculating scores. If references are given, these might point to the approach that is currently being proposed (or anything that has emerged from work on UWEM 2 ?), the Accessibility Priority Tool used by Roger Hudson and others, or some other additional, informative graded scoring approach that can reflect the criticality of failed instances. |
| Kathleen Wahlbin | accept this section as draft | |
| Eric Velleman | accept this section as draft | |
| Denis Boudreau | ||
| Gregg Vanderheiden | I do not accept this section as draft | All is good except 5 d Scoring has always proven to be problemmatic and misleading when actually tried -- and unless and until you have data showing that it actaully works based on evaluation of diverse sites -- we should not have this in the document. THis is alway seductively attractive but ive never seen it work. Some counter examples. I have a site with 1000 pages. Each page has (the same) something that makes the page completely inaccessible to people who are blind. The site scores 86 out of 87. (I only violated one SC -- but it was a showstopper and it was on every page). I have a really good site (10,000 page) but there is something on one page or another that technically violates every SC. They are minor and very widely dispersed. And all are because of something key about that particular page and don't really affect the accesbility. The site is given awards for accessibility but scores an F (very low numerical score) because of the way the calculations are made. (90 little problems on 10,000 pages of otherwise perfect pages.) etc. I don't know what the purpose of the numberical score is. What will it be used for? I think it is more dangerous than useful and without widespread testing - unproven as having any validity (face or otherwise) ALSO NOTE That this score calculation section also has "applicable to the evaluation" in 8 places in it. see previous concerns on not-applicable) Suggestion? Drop 5d. RE Detlev -- I don't understand your response. You say "several participants have emphasised that site owners / developers commissioning accessibility testing are keen to be able to record and track where problems (SC Failures) exist in order to manage the a11y part of their development, prioritise remediation, and check whether things have improved when re-testing. " I'm not sure what this has to do with a sitewise score. This seems to point in the other direction. I think any measure that provides a matrix of scores (one line for each SC ) is great and lets you see problems at a glance But a composit number gives false positive and false negative numbers and should not be used. And I don't thin we should be promoting a scoring system that has not been experimentally validated ( or validated in any fashion) . It is too easy to create these in meetings -- but I have never seen one that was valid in practice. And the W3C endorsement of this approach -- without any data --- I don't think is helpful to the field. ================ Otherwise the rest of this section is good. PS Sorry I can't be there for the meeting but it is right on top of one of our weekly Cloud4all meetiings (a 25 member consortium in Europe -that I co-lead so can't skip out...) |
| Kerstin Probiesch | I do not accept this section as draft | Also on this David's statement reflects my own opinion. One additional point: # Priority: strong Location and current wording: "About the Evaluation" point 1: Name of the evaluator (responsible person, organization, or other entity)" Suggestion: Names of the evaluators (responsible person, members of the review teams, organization, or other entity) Reason: transparency (especially because WCAG-EM is part of "European Accessibility Requirements for Public Procurement of Products and Services in the ICT Domain") I also agree with Gregg's statement on 5d especially: "(...) unless and until you have data showing that it actually works based on evaluation of diverse sites -- we should not have this in the document." (Gregg) Unless the reliability of a scoring system is not proven it should not be part of this document. Especially because (and as written before) WCAG-EM is explicitly mentioned in the documents of "European Accessibility Requirements for Public Procurement of Products and Services in the ICT Domain" (M376). |
| Yod Samuel Martin | I do not accept this section as draft | Detailed comments to Step 3 are available at <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.q1cxdi3kjojx> Some of the comments can be addressed after the publication of the Working Draft. However, **these are deemed to be dealt with before** with strong priority: - Comment #37 (on requiring a mention to the scoring procedure if a score is provided) <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.wp2xh6l2zmlj> - Comment #39 (unclear wording for the procedure to compute per web page score) <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.fat2qngbwagt> - Comment #41 (on conflicting usage of the terms referring to applicability of Success Criteria <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.ipi41u4q10hp> - Comment #42 (on discouraging the use of scores for comparisons among websites this is an editorial change, anyway) <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.lglkt0yidzbc> - Comment #43 (on keeping the request for public feedback regarding scoring) <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.pcficqc6p0u> - Comment #44 also has strong priority, but due to its complexity, I understand it cannot be addressed before publication as a Working Draft, and it is subsumed under the general discussion regarding scoring. <http://lists.w3.org/Archives/Public/public-wcag-em-comments/2013Dec/att-0065/CommentstoWCAGEMEditorsDraft29November2013.html#h.1u3ruz2iffek> |
| Michael Cooper | accept this section as draft |
More details on responses
- Roberto Scano: last responded on 3, December 2013 at 15:32 (UTC)
- Michael Elledge: last responded on 4, December 2013 at 22:06 (UTC)
- Martijn Houtepen: last responded on 5, December 2013 at 14:08 (UTC)
- Sarah J Swierenga: last responded on 5, December 2013 at 14:53 (UTC)
- Sharron Rush: last responded on 6, December 2013 at 03:38 (UTC)
- Howard Kramer: last responded on 6, December 2013 at 14:57 (UTC)
- Paul Schantz: last responded on 6, December 2013 at 15:03 (UTC)
- Bim Egan: last responded on 6, December 2013 at 16:24 (UTC)
- David MacDonald: last responded on 7, December 2013 at 05:59 (UTC)
- Alan Smith: last responded on 9, December 2013 at 16:41 (UTC)
- Richard Warren: last responded on 9, December 2013 at 21:16 (UTC)
- Kathleen Anderson: last responded on 9, December 2013 at 22:09 (UTC)
- Loretta Guarino Reid: last responded on 10, December 2013 at 04:16 (UTC)
- Alistair Garrison: last responded on 12, December 2013 at 12:03 (UTC)
- Detlev Fischer: last responded on 12, December 2013 at 12:55 (UTC)
- Kathleen Wahlbin: last responded on 12, December 2013 at 14:01 (UTC)
- Eric Velleman: last responded on 12, December 2013 at 14:09 (UTC)
- Denis Boudreau: last responded on 12, December 2013 at 22:40 (UTC)
- Gregg Vanderheiden: last responded on 13, December 2013 at 21:33 (UTC)
- Kerstin Probiesch: last responded on 15, December 2013 at 09:29 (UTC)
- Yod Samuel Martin: last responded on 15, December 2013 at 21:56 (UTC)
- Michael Cooper: last responded on 17, December 2013 at 17:45 (UTC)
Compact view of the results / list of email addresses of the responders
WBS home / Questionnaires / WG questionnaires / Answer this questionnaire
Report issues
on GitHub project