WCAG 2.0 Evaluation Methodology Task Force Teleconference

22 May 2014


See also: IRC log


Mike, Vivienne, Martijn, Shadi, Alistair, Sarah, Kathy
Detlev, Liz, Tim, Eric


<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140514

MH: Eric is unable to make today's meeting so Martiyn is filling in for him.
... please review editor's draft next week and the following week we'll discuss. Next week is a public holiday in Europe so no meeting, so please review draft

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/comments-20140130#comment77

MH: first today: comment #77
... regarding evaluating third party content

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140514#considerations

MH: is this re-write of that section okay?
... explained the re-write, please +1 if you agree


<martijnhoutepen> Agree to rewrite?

<martijnhoutepen> +1

SA: don't understand how the re-write addresses the comments

MH: original text contained ambiguous terms such as 'however'. Re-write to make it clearer.

SA: the use is in the use of the word 'third party'

<shadi> http://services.w3.org/htmldiff?doc1=http%3A%2F%2Fwww.w3.org%2FTR%2FWCAG-EM%2F&doc2=http%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Fconformance%2FED-methodology-20140514#considerations

SA: differential view doesn't really show this
... problem is using the word 'third party' of conformance and then content which refers to WCAG
... in this we don't mean third party as defined by WCAG

MH: we need to find out which one this is supposed to be - maybe 'external'

SA: disambiguate from third-party content

VC: comment 77 in the disposition of comments is evaluating third-party content - which is WCAG definition

MH: should put in the WCAG definition

ME: in reading the re-written paragraph addresses the issue of extrapolating from looking at a number of pages to the website as a whole in respect to evaluating conformance. Seems to have disappeared from the re-write.

SA: re-reading the comment - it's about content, not about the conformance statement

<shadi> http://services.w3.org/htmldiff?doc1=http%3A%2F%2Fwww.w3.org%2FTR%2FWCAG-EM%2F&doc2=http%3A%2F%2Fwww.w3.org%2FWAI%2FER%2Fconformance%2FED-methodology-20140514#thirdparty

Sa: we don't talk about evaluating - maybe add a sentence or 2 to the current section to say something like "for third party content you need to check..."
... could just rephrase "for third party content that is not under the control of the website the evaluator needs to determine whether this continuously monitored and repaired, meets WCAG 2 and that its clearly identified for any non-conforming sections"
... check if errors in the third-party content can affect the other content
... this clarification needs to be added

MH: add to the definition so that it is more clear to the evaluator

<shadi> "Editors to add some guidance like "ask website owner if monintoring is implemented" and "check if errors in third party content can affect other content" etc (see minutes for more discussion)"

<shadi> (from Comment 77)

<shadi> +1

<martijnhoutepen> +1

MH: is everbody in agrement to adding the sentence?


<Mike_Elledge> +1

<Sarah_Swierenga> +1

MH: seems to be agreement

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/comments-20140130#comment82

next agenda comment - #82

MH: EOWG thnks too much link to definitions which makes the document confusing. Maybe we can check the links and make sure that we don't unnecessarily link to definitions that are not essential in that context.

<shadi> +1


<martijnhoutepen> +1

<Sarah_Swierenga> +1

SA: editors need to do that - Shadi & Eric

MH: this is important for the more difficult parts of the content

<Mike_Elledge> +1 certainly not link to EOWG :^)

SA: could be an older comment as there were more links in an earlier version

next comment: ID 90

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/comments-20140130#comment90

MH: put links at the end of sentences - not in the middle of the sentence
... just relocating the links to the end of sentences. Editors can do this.

SA: this is a bit more substantial, but agree with comment

MH: has already improved since previous version

<shadi> [[Some of the content might already be reflected in the web pages and web page states selected in _Step 3.a: Include Common Web Pages of the Website_ and _Step 3.c: Include Other Relevant Web Pages_.]]

SA: agree with the comment, but not too sure what to do. Discuss with Eric.

MH: Have we decided what random is? Agendum 4

<martijnhoutepen> http://www.w3.org/2014/05/15-eval-minutes.html

MH: last week we ended on a discussion about randomness and random sampling

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/comments-20140130#comment23

MH: should we skew the sample toward frequently visited pages or frequently used pages. The consensus was that the random must be random.

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140514#step3e

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140514#step2e

MH: relates to step 3e. We need to keep the random, random. It is to validate our structured sampling. This was decided previously. However we can put in top search results and maybe changing step 2e where we say at 'identify other relevant web pages'
... top visited pages are relevant to people with and without disability. This could be added there.

AG: last week we discussed how people are meant to be selecting random pages. If this is left to the evaluator, they may not be looking totally randomly across the site. The evaluator needs to identify how they select the random sample and making sure it is really random - without or without a tool

MH: we provide some suggestions to the evaluators - such as make a script that will generate a list of all pages, use search engines. We do provide some guidance. Do we need more guidance?

AG: it is not perfectly random if someone is involved in the selection process
... if there are 10,000 how random is the selection going to be? Mostly it won't be random, will be skewed.
... we have emphasized that we need random pages

MH: we do state that even though the random is not strictly scientifically random, it can fill the importance for the applicability and reliability of the results. It adds to the reliability of the evlauation.

<shadi> [[While the random sample need not be selected according to strictly scientific criteria, the scope of the selection needs to span the entire scope of the website (any web page and web page state on the website may be selected), and the selection of individual web page and web page states does not follow a predictable pattern.]]

SA: I agree. We decided on the above sentence
... it will depend on the website. If you have a tool that will generate random-like pages, that would be fine. Otherwise your method of random samplng should span the entire score of 10,000 pages.
... we decided not to overdo this bit as the effort will outweigh the benefits. It is to cross-check the results.
... this differs from the structured sample. We might want to skew the structured section to look at certain types of pages.
... the guidaance here is sufficient. You as an evaluator need to decide what random is in the context - a tool can be helpful if you have something at hand. You need to ensure that your own structured sample is a sufficient example of the website

MH: we don't want to stipulate an algorithm. We need to be clear in that the random sample may not be really random

AG: understand its a good idea to pick another selection of pages to cross-check yours. But if we say random, it becomes a task to discover what random really is. You might want to say to select a few other pages to validate your pages

SA: maybe another term? randomly picked. Unstructured doesn't describe it. It shouldn't be just 'select a couple of pages'. We should say something about what kind of pages and how you should select them.

Kathy: as far as the random pages, its hard to select random, especially for applications. What if you say that you need to select additional pages and give guidance on how to select those pages. When we do an audit to make sure that people have made fixes across the website we want to choose pages that weren't chosen the same way as the first sample. Just give additional direction. They are

still somewhat randomly picked by the evaluator.

SA: seems to be going counter to Alistair. Which is to not to pick something intentially. That would have the tendency to have some kind of bias.

Kathy: how are you going to be able to know what the pages are in an application?

SA: using an application would be different - pages belong to the complete process requirement. If you select a page from the application that happens to be on the website, would be different. It will be too difficult to define what 'random' is.

Kathy: it isn't possible to create a random sample from an application - due to dynamic creation of pages

MH: maybe the blind-folded colleague in that application
... what do most people mean as 'random' is not scientifically 'random'

AG: you could say 'pick some pages at random' means your actions are random, rather than the pages are random

I like that Alistair

Kathy: good suggestion

SA: a good suggestion
... 'picking at random a further selection'

MH: shadi can you prepare something along that line


<martijnhoutepen> +1

<shadi> +1

<Kathy> +1

MH: that's all of the agenda items. No meeting next week because of European holiday. Next meetingJune 5th

next time pick up on discussion 4C on the list

MH: please work on this items on the list

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140514#step4c

<martijnhoutepen> http://www.w3.org/WAI/ER/conformance/ED-methodology-20140514#step4c

SA: there were several issues relating to this in the discussion. AG raised the issue that there is a notion where you check a page and find it is non-conforming and then you find an alternate version. This isn't the same as WCAG. Section 4C and 4A need to be rolled together.
... the definition of 'conforming alternate version' has certain requirements for both the page and the aternate version in order for it to qualify as an alternate version. A brief clarifying statement is needed - 'a web page has an alternate version that meets the requirement of a conforming alternate version' to encourage people to look at the WCAG requirements. This is more editorial change.

Mh: in WCAG is it more of a process - together they may conform. Although the web page needs to have certain accessibility requirements - you need to be able to get to the alternate version. If the initial page is too broken, it may not qualify.

AG: I agree. You only need to check any web page and its conforming alternate version that it is done properly. You have to check the page 100% against all the SC. I expected you to be able to check that the link is provided and then to check that alternate version.

SA: on a case by case basis. It may not always be apparent that there are alternate verisons. Sometimes the page is designed to work with certain technologies or user groups. One page willnot have something, but the other page could have that. EG higher contrast and lower contrast. They would need to identify this.

AG: if they present a page with a conforming version and you include it in your sample, its going to fail and throw off the statistics for that website. YOu have been told that this page will fail, that's why you give an alternate version
... when you provide a link to a conforming alternative version. Together they conform. You know the page with the link to the other page will fail.

MH: maybe the page does not fail because they have provided a passing version

AG: why would you check the original page if you know there is a conforming alternate version

SA: if you put them together you will get the results

AG: I provided some wording

SA: we might actually pull that together within 4A. We'll look at the wording suggestion and propose something for people to review.

Review May 14 Editor's draft for next week

<Sarah_Swierenga> good discussion. have a good holiday weekend (US) and/or a good holiday next week.

MH: everyone please read the emails and if you can carry on the discussion.

AG: a lot of the misunderstandings that get into the list are from people who don't read the emails correctly

Mh: please be patient with each other, remember not everyone has English as their first language. If it isn't clear, then ask on the call.
... see you all in 2 weeks. Please read the discussions on the list and the editor draft.

Summary of Action Items

[End of minutes]

Minutes formatted by David Booth's scribe.perl version 1.138 (CVS log)
$Date: 2014/05/23 15:07:34 $