WCAG 2.0 Evaluation Methodology Task Force Teleconference

<shadi> http://www.w3.org/WAI/ER/conformance/comments-20140130

Eric: Some discussion last week
... A link was sent for the minutes of the previous meeting
... We needed more discussion on the scope last week
... we need to go through major points within Disposition of comments

<shadi> http://www.w3.org/2014/03/27-eval-minutes

Eric: We could put points from discussions in surveys which could be circulated
... We talked about version from title, scoring (needs a bit more)
... As many comments as possible will be put in the survey so we can focus our discussions

<Detlev> fine

Eric: This will also allow others to look at the work

Shadi: Last week we debriefed from face-to-face
... the survey idea for comments re disposition of comments is a good idea

<shadi> http://www.w3.org/TR/WCAG-EM/#step1d

<shadi> Step 1.d: Define Evaluation Methods to be Used (Optional)

Shadi: Steps requiring further thought 1d
... 1d needs to be better defined, with more explicit reference to techniques and tools
... Email, or a survey or thinking about the questions in the survey would be a good place to start this discussion

<shadi> http://www.w3.org/TR/WCAG-EM/#step4a

Shadi: step 4a

<shadi> Step 4.a: Check for Each Success Criterion

Shadi: we need to check back to the minutes from the face to face - IBM comments?
... There were some gaps issues

<shadi> http://www.w3.org/TR/WCAG-EM/#step5d

Shadi: last scoring

<shadi> Step 5.d: Provide an Aggregated Score (Optional)

Shadi: this needs to be discussed further with several groups, including WAI coordination group
... We need to make sure that all things we do are in line with the overall goals of WAI
... Are there other topics which need more discussion?

<ericvelleman> http://www.w3.org/TR/WCAG-EM/#step1c

Eric: Step 1c
... We had a bit of trouble defining this
... Difficult to describe baseline
... need a bit more guidance
... Defining the baselines becomes very broad
... There are a number of sections to be addressed - total of 4
... should we go through the 4 sections

Shadi: Its fine that we raise issues ourselves
... define the baseline is a candidate for discussion with the WCAG group on accessibility support - understanding doc

Kathy: Discussion in WCAG working group has had some discussions

Shadi: Kathy to be coordination point between WCAG group and EVAL re baseline
... We need to think about the questions which go with each section, which might go into surveys

Eric: Leave 1c and start with 1d

Shadi: The questions would be what do we want to do and what do we need to move forward

Eric: Back to 1c - what can we clarify to make this more workable

Detlev: Term baseline is ambigous - is it the same as lowest common denominator
... From the wording it is difficult to know what is required, and what should be defined

Eric, baseline is a list of several things at the moment

Detlev: We could have positive - browser x, assistive tech y
... Or a lower version with lots of tech which needs to be supported

Kathy: Under WCAG accessibility support there are a number of considerations which we could take and make into a diagram

<ericvelleman> http://www.w3.org/TR/WCAG20/#accessibility-supporteddef

Kathy: would could also have a list of things people have to think about when choosing a list of tech - what should you be testing on
... There are further implications with regard to different techs

<ericvelleman> http://www.w3.org/TR/UNDERSTANDING-WCAG20/conformance#uc-accessibility-support-head

Kathy: we could put the information into a diagram
... you cannot say test in this, this and this

Eric: we could copy from the document, but we have tried not to to date

<Detlev> aim for *simple* explanaton of options for defining baseline

Kathy: if you read the considerations documents there is a list of things

Eric: Like the idea of flow diagram

<shadi> Alistair: baseline is something you'd expect the website owner to define

<Detlev> XCan't undersatand Alistair!

<shadi> ...WCAG document is primarily designed for developers rather than evaluators

<shadi> ...really up to the Evaluation Commissioner in this situation

Detlev: Disagree

<shadi> [[This step ... is ideally carried out in consultation with the evaluation commissioner (who may or may not be the website owner) to ensure common expectations about the scope of the evaluation.]]

Detlev: We evaluate against BIVT, which fixes what we look at

<shadi> Alistair: but essentially up to the website owner what they define as their baseline

<shadi> ...putting a definition constrains them

Detlev: It is what you are looking at - but to simply take what the website owner states as sufficient is missing the mark

Eric: the question is who decides the baseline - the commissioner or the evaluator
... it would be useful to discuss this further

Alistair: the commissioner sets the context of where they think their website will be used

Eric: We should leave this for further discussion

Kathy: as far as support, regardless of who decides the baseline we still have a number of questions which would need to be asked
... many people don't know, and many will need extra help from experts
... The baseline will differ from person to person - depending on who is using the documentation
... We probably understand better that the commissioner as to accessibility support
... It is going to be different in different contexts - the goal is to ensure that it will work for the end user, in their different regions

Shadi: Agree with everyone - minimal requirements are set by WCAG
... bar is set, below that you are not meeting WCAG
... This is a grey zone, where you can technically meet WCAG but in reality this does not work broadly
... How to meet WCAG are decisions for the web developers, we can of course provide a set of warnings
... We agreed from the beginning that this would not be adding additional requirements
... We cannot be more prescriptive that WCAG, and we would hope to work with the website owner to determine what baseline they were considering
... Warnings as opposed to what is technically required by WCAG
... WCAG sets the requirements

Eric: Look again at WCAG and see if we can clarify things a little.

<shadi> [[+1 to providing clarity *from the perspective of an evaluator*]]

<shadi> Alistair: issue is that setting the baseline by the evaluator might lead to differences

<shadi> ...can't get to an agreed baseline

<shadi> ...involvement of the website owner is necessary

Eric: 1c clarify steps or questions, and see if we can add ownership
... can we atleast give people the questions - think about this when setting baseline

Detlev: Evaluator should not set the baseline, it should be more set by a group - disabilities groups- etc...
... The audience should create a consensus
... Some people like the latest things, but from a user perspective this might not be the best thing

Eric: I'll look at the section and see what could be done - maybe a number of questions.

Shadi: Another issue or two: consistency questions - does the baseline need to be consistent in a page, over a website
... and that the baseline should be over all widgets, rather than one for one and another for another
... We need to differentiate between WCAG failures and other issues

Eric: Summary, I will propose a list of questions
... Surveys will be started for a number of comments
... Then everyone can have a look at the surveys
... next time 1d
... and scoring

Detlev: EIII project - evaluating websites - is it possible to get input
... information on scoring would be great

Eric: We haven't talked about scoring yet

<shadi> http://www.eiao.net/publications/

Eric: The same discussions are being held everywhere, highlighting the difficulties with scoring

Tim: We're working on something

Shadi: Scoring is a very open ended discussion
... No rubber stamped solutions
... Other three things are easier to move forward with

Eric: Next telco next week

Eric: Survey will come in advance