WCAG 2.0 Evaluation Methodology Task Force Teleconference

06 Dec 2012


See also: IRC log


Shadi, Eric, Martijn, Vivienne, Mike, Liz, Moe, Kathy, Katie, Sarah, Detlev, Peter, Tim


Updated Disposition of Comments

<ericvelleman> <http://www.w3.org/WAI/ER/conformance/comments-20120920>

<Detlev> looked at it but not its entirety

eric: Please look in document and look for Please Review in column.
... Second column is status
... Id 8 Please review telco discussion, commenter, location, terms and conditions. Proposed resolution. Rationale.

<Detlev> Eric, can you post link to the questionnaire?

Eric: Please review document let me know if you do not agree with any of them. We have a few more days to look at.

<ericvelleman> <http://www.w3.org/WAI/ER/conformance/comments-20120920>

Eric: This link is the updated disposition of comments

Peter: How do we determine the actual comment

<Detlev> This is the link to the W3C survey: https://www.w3.org/2002/09/wbs/48225/evaltfq6/

Eric: First column id, 3rd column commenter, 4th column link to their comment

Peter: This is a bit cumbersome while their link may go to several comments.

Eric: Places text of comment in the document and proposed changed.

Peter: Change column regarding Suggested Change and Rationale to Commenter's Suggested Change and Commenter's Rationale

Eric: Look for "Please review outcome" in column 2

<Vivienne> can we have a couple of days longer?

<Sarah_Swierenga> +1 monday deadline

Eric: Is Monday, Dec. 10 a good day for deadline to review?


Detlev: Please let us know what to do before Monday?

Eric: Just look for "Please review" in column 2. Review Proposed resolution and confirm agreement or disagreement.

<MartijnHoutepen> +1

<Vivienne> that sounds better

Detlev: Would like a later deadline.

Eric: Deadline, Wednesday, 12/12
... Submit feedback to mailing list.

<Detlev> fine

Eric: Let's return to agenda
... Let's finish survey by Wednesday 12/12/12 as well and we will discuss next Telco
... Agenda item 3: #3. Extending the Scope to include websites during development Please join into the discussion we started about this on the list: <http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Dec/0001.html>

Extending the Scope to include websites during development

Peter: While there a 2 separate concepts, one leads to the other.
... Maybe the case that we should talk about exclusion in two or more cases. Not only in shipping the product. Maybe more comfortable with exclusion in one or other.

Vivienne: Both extend scope. Still like to keep separate. Some people will agree with one and not the other.

<Detlev> agree

Eric: #1 Extending scope to include websites during development

Peter: I think this is very important. We want to encourage use of our methodolgy during development as much as possible. Need to make a distinction about making a final report but see no reason why we should not invite developers to use methodology.

<Detlev> fully agree (as you know I would)

Peter: Only need to address situations where one would need to make a public statement.
... We have no control over how developers use public entities but would be useful to encourage it.
... We could add text in the introduction that would invite people to not wait until application or site is done before using this methodology. Would not be appropriate to publish final report during development while not ready for that.

Vivienne: Difference is intent. Development, test as you go. Cannot make a statement of compliance. Can be used during development to test as you go however a full test is needed at the end.
... Gradual inclusion at the beginning. WCAG supports incremental testing, life cycle from the beginning.
... Try to not include something that is not satisfactory. Knowing your website is not compliant. Intent is not to get "out of something" but rather to perform testing on an incremental basis.

Eric: I agree. More or less what was said by Detlev during last Telco. Made difference between the exclusions. A full compliance statement is not possible during development.

Shadi: I feel we are repeating quite a bit that we discussed at face to face. Fully agree with Vivienne and Peter. The keyword is purpose. Looking at work statement and abstract of the document, the main purpose is to evaluate a website that exists.
... Not only make a statement but also deliver a report.
... Could be a website that is in development or a website that is completed. Think of purpose rather than target audience.
... Encourage evaluation throughout development otherwise one will not achieve accessibility. Primary purpose is still to evaluate existing website but does not exclude one from testing during development.

<korn> +1

<Sarah_Swierenga> +1 agree with "purpose" emphasis, rather than target audience

<Vivienne> +1


<Detlev> fine

<MartijnHoutepen> +1 to shadi

<Kathy> +1

<Liz> +1

Shadi: The outcome of the f2f discussion, please review the minutes. We did not have quorum.

Detlev: Can use the same process for both phases to be ensure that you have a proper outcome.

Correction: Detlev made previous comment.

Shadi: If pages are known to be inaccessible, this is a method for sampling pages of an existing website. If you know website in development, you may know that the outcome will ... Don't want to exclude someone who wants to use this in development but do not want to expand scope of document.
... Not sure that would need to be a change in the body.

Eric: Let's propose a redraft.
... Want to give a bit more time for folks to review the comments.

Eric: #4. Extending the scope to make exclusions possible Please join into the discussion started about this on the list: <http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Dec/0002.html>

Peter: Makes sense if someone performing the evaluation, especially during development, to make a statement that they will limit the scope of their evaluation.
... If we know a certain class of failures will happen, it may make sense at the outset to exclude things while it is known that they will fail. It is recognized that people will do it.
... Real quesiton: If you do this, are you able to make a conformance claim?
... Doing exclusions is a matter of practicality. Very useful and normal thing to do.

Detlev: Agree with Peter. The real issue involves conformace claim.
... Minimum: on a page level, everything has to comply or conform.
... Are cases in limiting certain things, like PDF document. Would need to document that the inaccessible PDF is available. Clearly flagged as inaccessible.
... Need to clearly draw the line.
... Should not raise the impression that everything is fine and conforms.

Shadi: May come to completely different results and this raises confusion. We are providing a standardized procedure for doing that.
... Don't think we should claim that we follow the procedure unless it is truly followed. Reminder: This document is informative and not normative. Can use the guidance but to make a statement about conformance need to use all parts.

Vivienne: I agree with Shadi. Uncomfortable to making this guidance messy as to what it can be used for.
... The purpose of this methodology is to evaluate entire websites. While one can use this methodolgy in any way they choose, it is designed to audit a complete website once established.
... Don't want to encourage folks to exclude parts of the websites if they do not comply. Ok to add a statement in the intro that you could use methodology as a guide but cannot make a statement of conformance or compliance.

Peter: I think real crux of the matter is compliance statement. The world is messy already. When look at single url web applications that extremely complex which has accessibility bugs that are not hit during review pass.
... Take slight exception of purpose is to audit. The purpose is to evaluate.

<Vivienne> sorry Peter, poor choice of words with 'audit' meant 'evaluation'

Peter: There might be other statements that could be made just short of a compliance statement. Is there other output that one might make other than 100% compliance statement?
... Don't have a problem if you are making a compliance claim that you should not exclude anything
... Purpose of report should have more than just a compliance claim.

Kathy: Majority of accessibility reviews are not on complete products. We review as we go through. Would be good to have methodology apply to development as well. And important to indicate that this cannot be a compliance claim.
... Majority of accessibility evaluations that happen are not on full products or full websites, especially when evaluating applications.

Eric: Agree if one wants to make a compliance statement there should not be a possibility of excluding things.

Shadi: I am interested by Kathy's statement. Maybe there is something that we should consider. Could use methodology in different ways but the question is around conformance statement.
... When evaluating products during development, how important is it to follow a methodology such as Eval TF? As compared to after website dev is completed?

Kathy: The methodology is a good way of saying this is how you go about testing a website or application. Here is how to select sample. This procedure is valid whether looking at subset or entire product.
... Challenge is that you have multiple development teams working on different parts and finish at different times. Ususally you have a contained set of functions per review.
... But still follow the same methodology.

<Detlev> Agree

Kathy: Report on all reviews the same way as if doing a full review. We combine results into a much larger review for conformance claim.

Eric: Biggest item is conformance claim and whether you can or cannot exclude things. Let's take discussion to the list. When would we want exclusions? What would the compliance statement depend upon?
... Focus on this during the coming week.

Detlev: Eric you state, "only point to make is that it is clear for final conformance statement we should not have exclusions". Feel this is still an open question. Still will have inaccessible PDFs etc.

Eric: I understand the legacy part also. Maybe we can point to WCAG statement on legacy parts.

Eric: #5. Other issues
... If not, wish everyone a good week and encourage discussion around these topics.

Eric: Plan to launch an edited draft next Wednesday.

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.133 (CVS log)
$Date: 2012-12-11 06:41:11 $