WCAG 2.0 Evaluation Methodology Task Force Teleconference

Continue discussion about new section ..Context of WCAG-EM

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#conformanceintro

<ericvelleman> list discussion: http://lists.w3.org/Archives/Public/public-wai-evaltf/2013Sep/0004.html

Eric: had discussion in the last meeting and the list. Should we change the title of the section "Context of WCAG-EM" or should we move this section?
... We should see how far in the discussion during the telco

Martijn: Agree with Shadi in putting it in the scope section, and Mike section on the edits for Section 5

<shadi> http://www.w3.org/WAI/ER/methodology/#scope

Mike: seems to me that part of the question is what is conformance. We should define what has conform. Listing what pages are included in the conformance.

<Detlev> Shadi, weren't you suggesting to leave out the term 'conformance' altogether?

Shadi: I like the word scope. Agree with Peter that we need to clear about conformance claim. My preference is to avoid the word conformance. It is a reserved term. We should try to avoid overlaps within the document

Vivienne: Confused about the whole thing since it has gone back and forth. If we evaluate every page and it meets WCAG 2.0 then they could say that it conforms to WCAG 2.0 for that particular date and time. We need to make it clear that conformance is for every page.
... For a website that we did not review all pages then we could list the pages for conformance

Eric: Who is ok with including it in the scope?

<Liz> Put is scope.

<MartijnHoutepen> +1

<Vivienne> I need the two points explained more

Shadi: We can't just put it in the scope to remove the overlap.

<Detlev> Might be better to take up again when Peter is present?

Eric: It is in multiple sections. We can see if we want to take the word conformance out or move it to 5b

Mike: it could be in a separate section then we should change the name to be specfic to WCAG 2.0 conformance claim

Shadi: We should try to avoid the word conformance but we can use conformance. We cannot redefine WCAG 2.0 conformance within this document

Miike: Agree with Shadi

Detlev: Accessibility statement is good. Conformance claims can only be made for single pages.

Eric: We imply in the document that if you look at a sample, then you have looked at the entire website. We don't have conformance claim unless we look at all pages

Detlev: Good to separate the evaluation statement from conformance claim

Eric: We could claim conformance for the sample and explain what we mean for the rest of the site

<shadi> Kathy: throughout the document we will have people wondering about how this relates to conformance claims

<shadi> ...so agree with Peter that need to tackle head-on

<MartijnHoutepen> good title

Shadi: How about the title "how this output relates to WCAG 2.0 conformance claims"? That is clear and will explain how they relate

Eric: we will take a look at this

<shadi> "How WCAG-EM Accessibility Statements relate to WCAG 2.0 Conformance Claims"

Discussion of listitems in Step 3: Select a Representative Sample

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#step3

Eric: There is a list that has been added.
... Added factors that are related to the likelyhood of a sample being representative for a website. We should include in this section also text about the notion of "reasonable confidence". This notion is also used in section 5.b called likelyhood of a conformance claim being valid for a complete website (within the scope of the evaluation as described in section 1.a).
... What do you think of the list?

Detlev: List is true but from a practical level it may not be clear as to what to do. Because there is so many different factors it will be hard to give more information

Eric: If you start describing it, they you have a long explanation about all of them

Vivienne: Ideas are good and things that people can think about. If I was novice person but I would not know how big the sample needs to be
... We need to go a bit further then here are some examples. An illustration or typical scenarios based on common situations

<shadi> Kathy: when recommending a sample size to the clients, we work with a questionnaire

<shadi> ...maybe different scenarios of how these factors come into play

<shadi> ...for example to spell out parameters like website categories

<shadi> ...because not all factors will always come into play

Detlev - one thing is a bit odd; "Experience level of the developers". If you increase the sample size you may not get better results

Detlev - importance of the page should be included in the list. Practical advice on how to pick pages to reduce the number of pages needed for the review.

Eric: This is in some places but not here.

<Detlev> Kathy - I see - I misread that!

<shadi> Kathy: point refered to level of experience with accessibility of a developer

<shadi> ...so might catch something in one page but not another

<Detlev> I thought it was talking about experience level of evaluators!

<Vivienne> I think it should stay there

Eric: Should we remove the list of factors?

<Vivienne> I want it to stay Eric

Eric: We need a more pratical approach. People will need to know what this means

<Vivienne> I like the idea of some examples

Eric: How could we get to a more practical approach?

<Detlev> Someone should draft something whichwe can then discuss...

Moe: Raise one concern; type of technology and templates used. The evaluator may not know this information. Concrete example will be helpful

<shadi> Kathy: if evaluators have less experience in selecting pages then may need more pages

<shadi> ...might need to work with other peers to get the right pages

<shadi> ...multiple people's experience often helps in the start

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#expertise

Vivienne: in the beginning we have a section about who should use this but that there is a base level of knowledge required.

<shadi> "Required Expertise"

<shadi> also "Review Teams (Optional)" http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#teams

Vivienne: we have stated that expertise is required to use the methodology so perhaps we can refer to that

Shadi: we have a later section on review teams. Maybe we can have this clarified earlier. We say encourage review teams but we could add something about being new or less experience then review teams are useful

Eric: we need to work on this but we will need help. Examples will be clear. But if we say it will require a larger sample

Vivienne: The evaluator we state is responsible for the page selection. This removes the bias. Our concerns may be misplaced
... to answer Eric's question, we could put together 2 examples for a typcial website

<shadi> Kathy: yes, evaluator is ultimately responsible but can get the commissioner/owner involved in the selection process

<shadi> ...to figure out the site and what to consider

<shadi> ...think could provide examples

<shadi> ...but also wondering about a chart

<shadi> ...not sure what that would look like, just as an idea

<shadi> ...might help people better understand the complexities

<Vivienne> I really like the idea of a chart

Eric: I like that. We could try that.

<Vivienne> I was going to say that I could help

Eric: Who can help write the examples?
... If you have time, please work on examples

Prescriptiveness

Eric: We should do the discussion on the list. Should we be more prescriptive in the document?
... and where should we?

Discussion start about Step 3.e: Include a Randomly Selected Sample

Eric: How much more should we add there?

<ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130903#step3e

Eric: We have received many articles about this. Do we want to say how to select a random sample? Is this necessary? I think we should leave it out

Vivienne: We have a random sampler application that students are working on

Eric: all of the papers are describing a certain method and many different ways depending on the site

<shadi> Kathy: if we stay away from it, then stay away all together

<shadi> ...or leave it open for people to decide

<shadi> ...if we add example it should not be exclusive

Mike: agreed with Kathy. What do you mean by leave out of the document? In the appendix, we could include this information as it will be useful. We should recommend an approach

Eric: How could we get this done? It will be hard to say go to this website and refer to it

Shadi: possible but the content needs to be develop.

Eric: investigating not including it in this document and linking to another document

Shadi: the other document would need to exists. If it causes a depency then we will need it

<shadi> Kathy: is there something that we can summarize in a chart or something?

<shadi> ...might support an overview

Eric: we will open a list item
... Participate in the discussion on the list.