W3C

Results of Questionnaire Eval TF Review: 30 July 2012 Editor Draft of Website Accessibility Conformance Evaluation Methodology 1.0

The results of this questionnaire are available to anybody. In addition, answers are sent to the following email addresses: shadi@w3.org,E.Velleman@bartimeus.nl

This questionnaire was open from 2012-08-07 to 2012-08-15.

10 answers have been received.

Jump to results for question:

  1. Support for publishing this draft
  2. Comments on the Draft
  3. Disposition of Comments

1. Support for publishing this draft

The current proposal is to publish this group-internal Editor Draft as a Public Working Draft (WD) for additional input from the public.

Summary

ChoiceAll responders
Results
I support publishing this as WD as is 4
I support publishing this as WD; however, I suggest the changes in the comments section below (for editors' discretion) 3
I support publishing this as WD only with the changes indicated as [important to be addressed before publication] in the comments section below 3
I do not support publishing this as WD because of the comments indicated as [important to be addressed before publication] in the comments section below
I abstain (not vote)

Details

Responder Support for publishing this draft
Eric Velleman I support publishing this as WD as is
Frederick Boland I support publishing this as WD as is
Elizabeth Fong I support publishing this as WD as is
Kerstin Probiesch I support publishing this as WD only with the changes indicated as [important to be addressed before publication] in the comments section below
Detlev Fischer I support publishing this as WD; however, I suggest the changes in the comments section below (for editors' discretion)
Martijn Houtepen I support publishing this as WD; however, I suggest the changes in the comments section below (for editors' discretion)
Sarah J Swierenga I support publishing this as WD as is
Aurélien Levy I support publishing this as WD only with the changes indicated as [important to be addressed before publication] in the comments section below
Kathleen Wahlbin I support publishing this as WD; however, I suggest the changes in the comments section below (for editors' discretion)
Richard Warren I support publishing this as WD only with the changes indicated as [important to be addressed before publication] in the comments section below

2. Comments on the Draft

(remember to include priority, location, suggested revision, and rationale for each comment.)

Details

Responder Comments on the Draft
Eric Velleman
Frederick Boland
Elizabeth Fong
Kerstin Probiesch Priority High for all issues:

In different sections of the WD we have the following: " However, the evaluator is responsible for an objective and thorough assessment."
Of course the evaluator is responsible but also the methodology itself is responsible for ensure objectivity. This point needs further discussion and the WD needs a section for this issue. In the WD also stands, that it is a "standardized methodology". As long as we don't have specific techniques for goodness criteria, I think we shouldn't speak about "standardized". Suggestion: Include goodness criteria like objectivity and reliability.

3.1.5 Step 1.e: Define the Techniques to be Used (Optional)
Agree with Richard.

3.2 Step 3 Select a Representative Sample
This step should be optional, because it depends on the size of the website wether a sampling is necessary or not.

But leave the sentence: "However, many web pages will have repetitive content, such as the header, navigation, and other common components that may not need to be re-evaluated on each occurrence. Guidance on evaluating the sample identified in this step is provided in 3.4 Step 4: Audit the Selected Sample."

3.4 Step 4: Audit the Selected Sample
Agree with Richard. Change into: "Audit the site or the selected sample" and change also every reference to this section in other sections.

3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible
I still don't agree with this and think that this should be optional.
"Requirement 4.b: Where possible, WCAG 2.0 techniques shall be used to demonstrate successes and failures in meeting the WCAG 2.0 Success Criteria relevant per 3.1.3 Step 1.c: Define the Conformance Target."

This sentence is contrary to what is said in the techniques-intro of WCAG 2.0: "Test procedures do not, however, imply success or failure beyond the particular technique. In particular, test procedures for individual techniques should not be taken as test procedures for the WCAG 2.0 success criteria overall. "

The sentence "In this case the evaluator must determine whether the WCAG 2.0 Success Criteria are met without the use of techniques. " is unclear. Reason: there is no way to build a web page without using techniques.

Rewrite the sentence and the section and make it optional.

I particular like the idea of not testing the same issues on every page again and again or give an pass/fail for issues already have been tested. I think time should be spended in evaluating and not in giving a pass or fail of already mentioned things on every page a website or a sample.
Detlev Fischer http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Aug/0006.html
Martijn Houtepen priority: mild
location: 1.4 Terms and definitions - definition of template
current wording: Content can be viewed when entered into the template.
suggested revision: Templates are often filled with content,
rationale: This can be confusing; are templates something visitors can enter content in? If the relationship between content and template is important: see suggested revision. Otherwise other clarification may be needed.

priority: editorial
location: 3.2 Step 2: Explore the Target Website
current wording: Carrying out initial cursory checks during this stage already helps identify web pages that are relevant for more detailed evaluation later on.
suggested revision: Carrying out initial cursory checks during this stage helps identify web pages that are relevant for more detailed evaluation later on.
rationale: the word already seems superfluous
Sarah J Swierenga
Aurélien Levy priority : low
location : 1.1 end of first paragraph.
suggested revision : add link to the Accessibility responsibility breakdown of wai-engage community group http://www.w3.org/community/wai-engage/wiki/Accessibility_Responsibility_Breakdown

priority : low
location : 3.4.1 Step 4.a + 3.5.3 Step 5.c
current wording : "Note: According to WCAG 2.0, Success Criteria that do not apply to the content are deemed to have been met."
suggested revision : add the link to the source of this note in the wcag2. It look strange for me regarding EARL witch clearly have a not applicable status

priority : high (important)
location: 3.4.2 Step 4.b
current wording : Use WCAG 2.0 Techniques Where Possible
suggested revision : Use WCAG 2.0 Techniques test procedure Where Possible (optional)
rationale: as 3.1.5 Step 1.e is optional this step must be optional too. Furthermore without making it optional some of current wcag evaluation methodology such as Accessiweb, RGAA, UWEM,etc will fail to conform to WCAG-EM

priority : high (important)
location : 3.5.2 Step 5.b second item of first list
current wording : The website owner commits to removing any valid issues known to them within 10 business days;
suggested revision : how and why 10 days ? in some situation it can take longer to get something fixed on some big corporate or ministerial website. I suggest to reword this part as : The website owner commits to give feedback on any valid issues known to them within 5 business days; This feedback should containt a detailed planning of remediation or an alternative way to get access to the information
Kathleen Wahlbin Priority: Low
Location: 2.1
Suggested Revision: Change .org to .com
Rationale: Public websites are mostly .com sites

Priority: Med
Location: 3.1.5 Step 1e
Suggested Revision: Add information about sufficient vs advisory techniques
Rationale: This is an area that is confusing for people. Advisory techniques may not be fully supported by AT and should be noted in this section. If they are used, we should note that evaluators should make sure that they work with the web browsers and AT selected in step 3.1.4 Step 1d

Priority: Med
Location: 3.2.1 Step 2a
Suggested Revision: Add page states
Rationale: Agree with Detlev's rationale here. Page states is just as important as common webpages and templates

Priority: Med
Location: 3.3 Step 3
Suggested Revision: Add note referring to 2.1.1 where it notes that small websites can skip this step
Rationale: For small sites there selected sample may be the full site

Priority: Med
Location: 3.3.2 Step 3b
Suggested Revision: Drop requirement to have two distinct pages
Rationale: See Detlev's comments. I agree with his suggestions here.
Richard Warren 3.1.5 Step 1.e: Define the Techniques to be Used (Optional)
This section requires completely rewriting to make it clear we are talking about the evaluation techniques not the web-design techniques. Also it should not be optional.
Suggested text for this section -

Describe the various tools and techniques that will be employed when conducting this evaluation. For example:
*use W3C validator to check validity of stylesheets
*use Juicystudio colour wheel to check adequacy of colour contrast
*use J Gunderson Accessibility toolbar to (a)list images and alternative text on individual pages. (b) list semantic structure,(c) view pages without stylesheets
*manually check that link text and section headings are relevant to their purpose.

By specifying the various tools and techniques it is possible for future evaluators to replicate/confirm your results. It also provides a level of transparency to aid quality assurance.


Reason: This heading suggests that I will find information about the techniques that the evaluator expects to use during his/her evaluation. (I.E –robots to evaluate HTML and CSS code, toolbar to evaluate semantic structure on a page-by-page basis, screen-reader to confirm that audio output matches visual output. Etc..). Instead I find reference to the recommended techniques that a developer should use to build a compliant site.. At this stage of the evaluation process we have not even explored the site – so there is no way of knowing which “recommended techniques” have been used. Also remove the link to WCAG 2.0 Techniques.

3.2 Step 2: Explore the Target Website
Suggestion: Change first paragraph to the following:

During this step the evaluator explores the target website to be evaluated, to develop a better understanding of the website purpose, use and functionality.

Carrying out initial cursory checks during this stage already helps identify web pages that are relevant for more detailed evaluation later on. For example, an evaluator may identify web pages that seem to be lacking color contrast, consistent navigation, or document structures (headings, links, etc.) with simple checks while studying the website, and note them down.


Reason: This first exploration of the website should also check/confirm the *purpose* of the site. This is the only time the evaluator will get a “first impression” so the word purpose (or similar) should be included in the first sentence of this introduction to step 2. The second and third sentence in the first paragraph of the introduction are helpful suggestions and not related directly to the first sentence or the main purpose of this step. Therefore they should form a new paragraph.

Also please do not refer to the sampling procedure at this stage in a way that implies *all* evaluations will use only a sample of web pages. We agreed that the default position is to evaluate every page of the website as this is the only way to be 100% sure of the reliability of our result. We did agree that checking every page on a large web-site such as www.whsmith.co.uk or www.amazon.com would be prohibitively expensive and therefore a sampling procedure might be required (and reflected in any conformance claim), but this is OPTIONAL not compulsory.

3.3 Step 3: Select a Representative Sample
Change: This section should be optional and requires an introductory paragraph to explain why sampling might be required.

Reason: We have discussed this at length and agreed that a full audit (every page) should be the default position. Sampling should only be used if the site is large and resources are limited. If sampling is used it must be stated in the conformance claim

3.4 Step 4: Audit the Selected Sample
Change: Title should read “Audit the site or the selected sample”
Reason: As above - we agreed that full audits were the default position


The penultimate sentence of the Note: for step 4
Change to add phrase "repetitive elements" and read as follows
evaluator may not need to continue to identify successes and failures in meeting the conformance target for these repetitive elements on every web page.

Reason: The note refers to repetitive content. At present this sentence reads “evaluator may not need to continue to identify successes and failures in meeting the conformance target for each web page.” This is a little ambiguous and could be taken to include none-repetitive content.

3.4.2 Step 4.b: Use WCAG 2.0 Techniques Where Possible Requirement 4.b:

Change to read
Requirement 4.b: Where the correct use of appropriate WCAG 2.0 techniques can be identified they can be used to demonstrate successes in meeting the WCAG 2.0 Success Criteria.

Reason:
It is still not clear that if the evaluator can identify the correct use of WCAG techniques then these can be used by the evaluator as evidence of compliance. I suggest that the wording be changed. This means that the evaluator can confirm compliance if the appropriate technique has been correctly applied without having to do any other test. For example: if the label element has been applied to enclose both the field instruction and the input field then there is no need to check manually if the form progresses properly when using the keyboard or can work in forms mode with a screen reader.

3.4.3 Step 4.c: Assess Accessibility Support for Technologies Requirement 4c
Suggestion: change word *create* to *present* in the requirement statement as follows
Requirement 4.c: Each use of the web technologies used to present the web page content shall be checked to be accessibility supported by the tools defined in Step 1.d:.

Reason: We are not evaluating the tools etc used to create the site (e.g Dreamweaver, Joomla, Websphere etc). We are checking the technologies used to deliver the content to the user. For example if scripts are used to display content such as warnings (and we have note specified in step 1d that all users will have scripting enabled) will the content still display if scripting is not enabled ?


4.4 Large-Scale Evaluation of Multiple Websites
Change second sentence to the following-
This methodology can not be undertaken using just automated evaluation, which can only evaluate a small portion of the accessibility requirements, it also requires manual evaluation by experts.

Reason: The current sentence is ambiguous.

3. Disposition of Comments

Please provide any comments you have on the Disposition of Comments. Make sure to indicate the ID of the comment that you are responding on, and please keep your comments brief.

Details

Responder Comments
Eric Velleman
Frederick Boland
Elizabeth Fong
Kerstin Probiesch
Detlev Fischer
Martijn Houtepen
Sarah J Swierenga
Aurélien Levy
Kathleen Wahlbin
Richard Warren

More details on responses

  • Eric Velleman: last responded on 7, August 2012 at 21:40 (UTC)
  • Frederick Boland: last responded on 8, August 2012 at 12:56 (UTC)
  • Elizabeth Fong: last responded on 8, August 2012 at 14:12 (UTC)
  • Kerstin Probiesch: last responded on 8, August 2012 at 17:33 (UTC)
  • Detlev Fischer: last responded on 9, August 2012 at 06:46 (UTC)
  • Martijn Houtepen: last responded on 9, August 2012 at 10:01 (UTC)
  • Sarah J Swierenga: last responded on 9, August 2012 at 16:47 (UTC)
  • Aurélien Levy: last responded on 13, August 2012 at 15:08 (UTC)
  • Kathleen Wahlbin: last responded on 15, August 2012 at 03:47 (UTC)
  • Richard Warren: last responded on 15, August 2012 at 19:28 (UTC)

Everybody has responded to this questionnaire.


Compact view of the results / list of email addresses of the responders

WBS home / Questionnaires / WG questionnaires / Answer this questionnaire