WCAG 2.0 Evaluation Methodology Task Force Teleconference

Possible Face-to-Face Meeting

<shadi> https://www.w3.org/2002/09/wbs/48225/EvalTF_CSUN2014/results

Shadi: 17 people responded
... WCAG working group is looking at the 18th
... more turn out on Tuesday
... A few people may have a conflict with WCAG working group if we meet on Tuesday
... will confirm in the next week

Eric: We could consider later on Monday afternoon or evening

Shadi: We could try that

Vivienne: Could people dial in?

Shadi: we will need to look at timezones
... some overlap for parts

Eric: we could do a summary and discussion at particular times

Vivienne: 4pm summary would work for Australia

Shadi: We should have a flexible agenda
... We have people coming and going

Allistair: What are the other opportunities

Shadi: Paris in July

Alistair: When will the work wrap up?

Shadi: This could be the last draft. Timeline to publish end of January, 4 weeks review period, 2 months to wrap and finalize the draft (April/May). By July we should be done

WCAG working group meeting

<shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129

<shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129#comment18

Shadi: WCAG working group has been busy with the publications. We did get through some things. Raised some of the open comments (#18 - term change and 27)
... No opposition, we can go ahead
... Changed it from open to review
... couple of other comments. Look at the minutes of the WCAG working group

<shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129#comment43

Shadi: #43 Scoring, same concerns were raised - misleading, dangerous. At the same time there is a need and has some use. Maybe the turn scoring is misleading. We are doing a tally or sum
... Changing the name may be good
... One of the comments is that there is no data to show that this works. If there is nothing for anyone to use then we will not have data
... some people think it is good and others are opposed
... look at the options; misuse vs. benefits

Vivienne: heard the term aggregation rather than score

Shadi: that is what I am thinking about, expectation would change with term change and would clear up the concerns

Eric: do we want to change it before we publish?

Shadi: Change it

Kathy: Detlev may have data

Shadi: it is weighted scoring; tim is also presenting on scoring. There will be studies that say things in both directions
... we are doing something similiar than scoring

Eric: Should we change to aggregation rather than score?

<Vivienne> +1

+1

<ericvelleman> +1

Alistair: It has come down to a simplistic calculation. Do we need to tell people to do this?

<Sarah_Swierenga> +1

Shadi: We could drop the section
... we could keep it and then drop it later
... it would help those who don't have a scoring mechanism

Alistair: Could be a separate document

Shadi: This could be applied to many of the sections. Usability testing refers out to a document. We don't have the resource yet

Eric: there are a lot of people who would like a score

Shadi: benefit to having it in the methodology, cannot compare scores if there are more ways of doing the score

Eric: we could change to aggregation for this version and it could take out in the last version if needed

Vivienne: would like to see it stay in. It helps justifies the costs and how much they have improved

Eric: they don't really need the methodology to do a score

<shadi> http://www.w3.org/WAI/ER/conformance/comments-20131129#comment28

Shadi: comment 28 on providing sample size
... David suggested what the government of Canada recommends
... we don't know how those numbers were calculated and it is hard to know how many pages are in the website. Suggestion from David was to add in typical sizes

<Vivienne> But what is a typical website?

<Vivienne> We offer a variety of sample sizes depending upon budget -10,25,50,75,100

<Vivienne> depends largely upon the client's budget

Shadi: what does the group think?

Vivienne: Australia says 10% of the website; we recommend that people choose the largest size the budget will allow 10-100; most common is 25 pages
... most likely will not have to go over 30

<ericvelleman> Our research shows that for level A you do not need more than 23 pages to find 98 percent of the problems

Vivienne: benchmark a site, then could review 10 pages

<ericvelleman> We could include this question in the test-run

<ericvelleman> The minimum of the current WCAG-EM is 49 pages + 5 random pages = 54 pages minimum sample

Shadi: We agreed that the methodology is independent of budget. It may not be feasible but the sample size should be appropriate for the website. Need a high degree of confidence

<ericvelleman> That is a lot

Shadi: In order to achieve high level of confidence and analysis there may be typical sizes

Sarah: agree with Vivienne, we don't want to base on budget it is reality and the typical client pages is 10 or 20. We have not gone above 30. Methodology should be able to be used regardless of number of pages.

Shadi: we are not suggesting a percentage or a fix number of pages
... is there a ball park number of pages

<ericvelleman> If we want WCAG-EM to be used on 10 pages, then we should provide some guidance. Maybe in an appendix? Now the number required is closer to 54 if you take a seperate page for each element

<Vivienne> what about a range e.g 10 - 50?

<ericvelleman> The methodology gives you the possibility to add more pages

Shadi: what do you do if the site does need more pages for proper analysis but there is no more budget

Sarah: cannot give the assurance

Vivienne: we give recommended sample size, if they don't do that then there is a disclaimer on the report

Kathy: Range would be ok, but I would not want hard numbers because there is alot of variables to consider

Alistair: Eric, how did you come up with 54?

Eric: Took the pages in the methodology it comes to 54

Alistair: it could be a problem since 54 is more than most reviews
... sample size should be about 25 pages or more

Eric: it depends on how you read the sampling section
... can combine items in the page. So then you could get 24 pages for the minimum including the random size. Depends on interpretation of the sampling section

Sarah: in the methodology we may want to set the minimim number
... could be 10 pages; clients look at this like site optimization; the minimum number of pages could be useful in guiding clients

Shadi: running out of time; it wasn't as clear that you can combine pages to reduce the number of pages required for the audit.
... maybe set a range for the number of pages and clarifying the sampling methology

<Sarah_Swierenga> 'combining pages' must be here. it would not have occurred to me to propose a new page for every item....

Shadi: Should we put an editor note or clarify before publication? We will take this up in the next meeting
... the other piece is to review the comments and how we are addressing them. If you have questions or comments, use the mailing list

Vivienne: sometimes the site is small and may not meet the minimum