WCAG 2.0 Evaluation Methodology Task Force Teleconference Meeting

Welcome

http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Aug/0011.html

Minutes of previous meeting

http://www.w3.org/2011/08/18-eval-minutes.html

EV: please let me know if you disagree with the minutes

<sds> nothing to change on my side

[no objections]

Recap of the proposed work

http://www.w3.org/WAI/ER/2011/eval/eval-tf

EV: shadi sent around some questions, i just sent around some too
... also found previous requirements

http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Aug/0041.html

EV: any questions on the Eval TF page?

Liz: no questions

<Ryladog_> Ryladog is Katie Haritos-Shea

SAZ: will work on a group page with the minutes and stuff

TB: will be using some form of issues tracking?

SAZ: an instance will be provided, will explain in one of the upcoming calls

EV: one of the objectives is collecting information
... maybe we can use a wiki?

SAZ: need clearer structure for a wiki or it gets difficult to track
... let's talk more offline

<sds> throwing best pratices to mailing list = something like http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Aug/0013.html ? or do we prefer URI to reference documents?

EV: will collect information through the mailing list for a start

Kostas: different languages ok?

EV: probably ok if some of us can understand it

Liz: would be good to provide a small paragraph explanation

Kerstin: can all the tests be best-practices?

EV: might be good to have survey, studies, and such

Kerstin: was just referring to the term "best practices"

SAZ: concerned about quality and getting disoriented by tracking external stuff
... better to start by our own requirements and see what existing experiences we can use

<vivienne> that makes sense

SAZ: feel uncomfortable starting out with someone elses requirements
... need a better common understanding of what we are looking for

EV: work on requirements

Discussion on the requirements

EV: some discussion on the list already

<vivienne> yes

http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Aug/0041.html

[[Is the "evaluation methodology" expected to be carried out by one person or by a group of more than one persons?]]

EV: number of people probably dependent on their level of expertise

Detlev: maybe have one tester, than aggregate more than that
... or usability testing or other improvements

<sinarmaya> I think it depends on the scope of the review and the stage where the web is. Stage design, development, publishing, updating. Scope: Self-evaluation, external evaluation, or audit.

Vivienne: I usually use a group of users, doesn't matter if by one person or a group of people

Samuel: we do an audit with only 1 or 2 people

<sam> http://alpha.gcwwwtemplates.tbs-sct.ircan.gc.ca/theme-clf2-nsi2/accessRespBreakdown-eng.html

Samuel: we do define "roles" though

<sinarmaya> I think that we need define "audit" ;-) For me an audit can't be doing but one person only ...

Kathy: often have multiple people involved in the review, depending on their roles
... role-based is good
... but also important to have one-person mode

Vincent: need to support people with less experience to be able to do some tests

<vivienne> I agree, we can train people to do some of the tests. We don't have to have experts doing everything.

Katie: not disagreeing but needs to be a test methodology, regarding who is doing it
... has to work for everybody

<sam> we have to make sure to eliminate the negative issues of the "hero" factor i guess. everybody can have a little part of the responsability

Katie: important to have comparability between evaluations

<vivienne> agreed

<kerstin> agreed

Tim: agree with everyone else
... the more perspectives, the better
... assistive technology use, etc

EV: so, at least one person should be able to do it but can be supported by others
... that person would need some level of expertise

[[What is the expected level of expertise (in accessibility, in web technologies etc) of persons carrying out an evaluation?]]

EV: anyone from the street can carry out an evaluation or do we need some level of requirements?

Vivienne: people have to be trained, and have to know what they are looking for

Detlev: often quite good to conduct test be two testers and have them talk about it
... often helps to make better decisions
... to get consensus on results

Detlev: agree need some level of expertise

<kostas> maybe basic technical expertise of the person could be important in order to understand the results

Detlev: but with technical knowledge can get up to speed quite quickly
... HTML, CSS, and scripting knoweldge
... scripting gets quite tricky

<sinarmaya> I believe that anyone can perform certain tests, but a person who assumes full assessment must meet the specifications of the languages ??or technologies used in the web, know the WCAG, understand the most commonly used technical aids, learn strategies adaptation of the users.

Samuel: need some level of technical knowledge
... but need basic understanding of the reality
... so know basic use of browsers, assistive technologies, etc

Alistair: often find that experts have different understanding of the requirements

<Detlev> Different understandings: quite normal, unavoidable

Alistair: too much room for the expert than that would influence the results

EV: what would be the solution to reduce room for interpretation?

Alistair: maybe some form of a training

Kostas: some level of understanding of WCAG is important
... additional instructions important too

Katie: don't want to divide things up between different types of disabilities
... but need knoweldge about the impact of barriers

<vivienne> I agree, some of the guidelines are more specifically helpful to different groups of people

EV: is this about prioritization?

Katie: no, would not want to prioritize based on disability
... want to build for everyone, but want to optimize for specific users too

Detlev: would not need to be part of the methodology itself
... could map the results to how it impacts specific groups of people
... in each of every test there can be different assessments
... depending on the context, particularly when aggregating the tests

EV: aggregation is one of the points that we need to discuss more in-depth

<vivienne> that's rather where we are now

Alistair: if people's expectation is that the methodology leads to 100% certainty about conformance but indeed it is not, then you have strong ramifications
... methodology should not provide information about how to fix issues but just the outcome

Vivienne: often assessments include suggestions for improvement
... something that is helpful

Detlev: only theoretical to have 100% compliance, so there has to be some degree of difference

<vivienne> aagreed

Tim: should facilitate education and opportunities to learn about accessibility

<vivienne> I agreed that we should have recommendations, sorry

Alistair: goal of WCAG2 is to reduce uncertainty

EV: will need to address that in the methodology
... how to increase inter-rater reliability

[[Is the involvement of people with disabilities a necessary part of carrying out an evaluation versus an improvement of the quality?]]

Vivienne: don't think it is essential but very helpful
... not everybody will have access to a group of people to test

Katie: it is a recommendation rather than a requirement

Kathy: agree with everyone else
... testing with users with disabilities certainly helps a lot to find additional issues but maybe not a requirements
... but maybe use of assistive technologies is a necessity

Kerstin: depends on the complexity of the subject
... may need to involve people with disabilities

EV: would need to define "complexity" of websites

Samuel: the way actual people use assistive technology is essential to understand how people use the web
... was important learning exercise
... need to keep up-to-date too

EV: needs to be in the methodology or a requirement for the evaluator?

Samuel: recommendation but not necessarily an obligation

Kostas: involvement of people with disabilities in the evaluation phase may not be helpful
... maybe better to use people with disabilities to validate the methodology
... results categorized by people with disabilities quite important

Alistair: the more people you ask, the more opinions you will get
... like the section508 model where you run down the tests then do functional testing too

Detlev: good for the usability of the methodology to keep it simple
... but agree that some complex sites may require involvement of people with disabilities

EV: out of time
... please continue discussion on the mailing list

EV: will draft first set of requirements