WCAG 2.0 Evaluation Methodology Task Force Teleconference

Specific discussion on section 5.4 Barrier Recognition

Eric resumes the discussion so far in the last weeks about 5.4

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120119.html

<shadi> [[ reminder that the latest Editor Draft is always available from http://www.w3.org/WAI/ER/methodology/ ]]

Eric will send a mail with a list of changes later

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120119#barrierrecognition

Eric: not a lot of discussion about barrier recognition, more questions

Detlev: many of the barriers might not be obvious, so it's necessary to evaluate the whole page to be sure
... looking for differences is important, not only looking for the same

Detlev, this was very fast

<Detlev> agree

agarrison: barrier recognition needs better definition

eric: will try to describe barrier recognition in a better way

<agarrison> I would actually remove the term barrier recognition

Shadi: Difference between stop condition and barrier recognition

Detlev: I meant probably something else, distinction between severe problems, systematic erros and not severe problems
... question which errors could be tolerated?

<agarrison> Surely this is the difference between A, AA and AAA

Eric: change of terminology for 5.5

<shadi> [[ wonder if "incidental" is better than "accidental" ]]

Kathy: Discussing error margin, no good definition until now

Eric: issue will come back in the reporting

<Detlev> My understanding is that you pick a sample and run every page against *all* SC

<Kathy> mute me

Eric: first setup for when to stop evaluation this week

<shadi> [[ suggestions from Mike "types of errors" or "error reporting" ]]

Mike: documenting different type of errors

Specific discussion on section 5.5 Error Margin


<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120119#errormargin

Eric: Do we accept error margin? example: one missing alt or Do we don't accept?

Vivienne: testing against SCs, give the client possibility to fix them, first evaluation a lot pages will fail, after fixing there is a chance for conformance
... if errors will not be fixed -> o compliance

<Elle> I agree with Vivienne, if errors are identified, they must be accounted in the overall conformance (or lack thereof)

Kathy: we can use a lot of tools, check a lot of website, it wll never be complete. Suggestion: Conformance Claim just for pages we had checked

<lwatson> +1 to Kathy

Detlev: one way dealing with might be seperating strict conformance and graded accessibility results
... full conformance will never be happened, need a realistic approach

Sarah: agree with Kathy
... suspects that people will claim conformance, and the public will point out problems, purpose of methodology is to identify what is needed for conformance

<Detlev> agree

sorry, I don't understand alistair

agarrison: importance of same results
... comparison is important

Eric: in the netherlands we check minimum 20 pages, error margin is 5 %
... to get the logo 10% error margin is allowed, but not for severe barriers, but not always measering in percentages

Leonie: realistic amount of pages in sample is important
... acceptable is when out of 20 pages 2 fail

Detlev: error margin is important
... critical barriers like keybaord traps should not pass
... what are acceptable errors

<agarrison> Who decides what are acceptable errors - would this lead to a replicable evaluation?

Detlev: in our appproach we have 95%, then is full conformance. more realistic are grades

Kathy: minor things should pass. How is error margin calculated in the NL

Eric: error margin depends on the elements
... if one of 10 has a problem (eg. videos) it will pass

<Elle> apologies, the call signal was bad and I'm only able to stay within IRC (thanks for the scribing!)

Kathy: we also have to talk about frequency

<vivienne> sounds good to me

<Detlev> not quite

Eric: Suggestion: there could be minor errors and if so, then we don't give a full conformance

Detlev: if there is no error margin possible its from a pratical view not usefull

<Mike_Elledge> +1

agarrison: points out the replicability of the test
... difficult to say, that if just 2 pages it pass
... when tested 20 pages, it's important to go further

sorry, haven't understood

<Zakim> shadi, you wanted to propose dual approach

Leonie and agarrison: discussing the issue that probably interpretations of SCs are not the same. which will be a problem concerning replicacability

Hope this scribe was correctly

Shadi: suggestion: you only claim conform if you conform, possibility. aggregation and documenting what passes and what fails, documenting as process (percentages) for motivating
... aggregation way could mean different levels, until now not found a really good aggregation method
... full conformance as primary measure, aggregation as addo

addon

+1

Eric: Starting soon with evaluation, important discussions about reliability, validity - in the next call