WCAG 2.0 Evaluation Methodology Task Force Teleconference

Eric: Discussion online about fact that a lot of text is coming from the UWEM document, explained in earlier calls

Welcome

Eric: delivered a document (Unifed Web Evaluation Methodology = UM)
... what happened (three years or so ago) was really valuable, but most of discussion was with WCAG 1.0, terminology is usable, but can we use this for WCAG 2.0?

Eric: do see text coming from there, but could change completely as a result of this task force group discussion
... will discuss further with shadi on copyright issues, but not needed for this call

shadi: wasn't aware that there was specific text directly coming from UM, so if the text is specifically taken from UM or structure, we may need a small acknowledgement section, let's follow up separately and see how much of that, but yes, should certainly try to use existing work

Eric: have a read, look for "UM" - will send a link, Alice also sent direct link in the email discussions, see if there's valuable stuff that we might want to use or use for inspiration
... side products, one is on indicators and indicator refinement, discussion about how can you get a random sample that is really a good sample of a website?
... saw a few papers that had links, read them, found another paper that was made inside this cluster that also addresses this, still looking based on discussion last week, When do we stop, how to we find the sample? trying to find scientific data before shutting down that discussion
... specifically in this tool, assistance-part, there's a possibility to use a tool to get a real random sample, question is, how many pages would be a good random sample? if you know of literature about that, please send a link to use in discussion

Eric: trying to close down how many pages is large enough or small enough

allistair: there have been other discussions about sample pages, where are we storing these ideas via email?

Eric: they are stored on the mailing list, problem with putting them in the document is that we didn't really agree about how many pages

allistair: in the end, I said I'd be happy with a smaller sample, but there's no mechanism to know what we're agreeing on if we're storing it in the emails
... other things that we are deciding on are not getting into the document
... can weekly updates note what we're agreeing on?

Eric: trying to update document following outcome of the teleconferences, but to your point, it should be earlier than what was done recently, try to do this sooner

allistair: what about putting it into the W3C?

Eric: still very much in an editor draft, tools to see differences in documents (change logs), but could increase administration

shadi: not entirely sure that subversion is very easy for everyone to use, what you're talking about from HTML5 group has three co-chairs, don't think it's fully automatically generated, but we can look at more stuff to support
... would really suggest that you point out things that have been dropped or not yet integrated, but with approximately 80 emails a week, it takes time to integrate these comments into the document

Eric: if there's something that wasn't put in the document, please just add it to the list and I'll add it

<agarrison> sounds good

Eric: try to put all the outcome of these discussions into the document, if there's something missing, please let me know what I missed

Specific discussion on section 3.3, 3.4 and 3.8

Eric: did make some changes that we discussed in the last call, short discussion (side discussion) that urged me to make a change, dropped the barrier section and added a section in the sample section
... added a section (4.3) in the sampling of pages called "stop criteria" so that is new and it came out of the discussion we had in the past meetings, dropping barrier recognition (5.4) - still there, but noted as we will drop this
... in the discussion, we said that "barrier recognition" was not the right terminology, and we didn't want to make a choice about what was more/less important as a barrier
... added the idea that we could stop evaluating more pages, so created temporary title "Stop Criteria" with a really short description
... so if you're sampling pages, core sample (homepage, search page, etc), the task oriented sample, and the random sample
... for the random sample, the question is how many pages would be in the random sample to cover the website as a whole
... introduced the term "stop criteria" here, propose a discussion (not on this call) but note that it's totally new in the document

allistair: when did we actually decided on these sampling things?
... did we actually decide and finalize that?
... stop criteria seems the same as barrier criteria, are we carrying that over from the previous document or do we actually need it?

Eric: as far as sampling is concerned, we agreed on 3 forms (core, task, and random)
... put text there for the discussion

allistair: can't remember sending agreement, pretty much copied from the UM

Eric: yes, from UM with changes to fit WCAG 2, but still need discussion

allistair: very important area

Eric: closed for part, start of scope, entering flesh into the document based on what people responded, the direction put in, but not definitive text in any way

shadi: from a process perspective, if there is text coming from UM, we have to acknowledge that and add a small acknowledgement that those sections are from UM
... regarding a group resolution, this is an editor draft at this stage, quite normal for things to appear as new, changed in an ad-hoc basis, but when we publish a working draft there will be an opportunity for the group to approve, small W3C process
... there won't be a formal publication without group agreement
... having it in there doesn't imply approval, just there for everyone to discuss

<Detlev> Conformance is defined only for Web pages. However, a conformance claim may be made to cover one page, a series of pages, or multiple related Web pages.

Detlev: touching on sampling, wondered whether the methodology would only cover scenarios where you want to evaluate a whole website or part of a website, defining core resources, random pages would apply, reading the WCAG text it seems that there's also the option to evaluate one single page
... would our methodology include that, or conditional decision?
... conformance claim and include one URL, for example

Eric: we have the methodology for a full website, but discussion about what is a full website (where it starts and ends) still need more discussion on it, so that two or more people who evaluate the same website and have the same result

Detlev: can you make a conformance claim for a single web page? would that be something that the methodology could support or would it be ruled out?

Eric: right now, it's ruled out, complete website

Vivienne: when talking about websites, does that encompass web applications as well? we do things that are hosted by a company and maybe made available to specific members of the public or kept for in house applications
... know that WCAG covers intranet, with our methodology, will that cover web applications?

<Detlev> please send link to the list, Vivienne!

Vivienne: 10% of websites need to be evaluated, will we be including that?

<shadi> http://www.w3.org/WAI/ER/conformance/ED-methodology-20120125.html#definitionsandterminology

Eric: can you send a link?

<shadi> [[Website - A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.]]

<vivienne> okay guys, I'll send it to the list

Eric: regarding web applications, specifies dynamic and static applications, so it's covered, yes

Kerstin: if we have a one-pager, will it be possible to evaluate the single page with our methodology?

Eric: website would only have one page?

Kerstin: yes

Eric: I think yes, but not quite sure, don't see a lot of websites with only one page, but we do have language in the sampling with "if available" so if not available, then the website is only one page, then that'd be the full website

Kerstin: I think it would be a good solution

shadi: I think a one-page or even small 5 page site is really not the challenge, since chances are your sample will cover all the requirements if you just select 5 pages (you'd meet the requirements of the sampling)
... if I understand Detlev correctly, if you have a website and you only choose a sub-section (like a subdomain or a specific part that belongs to a department), it might be useful to be able to say "this part of a website is accessible" but potential for misuse, but might be interesting

Detlev: that's what I was talking about, if you want to look at one particular page or aspect, would it be okay to be able to say "this page conforms" but the rest do not, would the methodology support that?

shadi: if it's one page, why not use WCAG? what does the methodology provide for added value?

Detlev: maybe not

Eric: maybe not a page, but a technology perhaps (ex. forms), some people request for testing according to specific types of technology, need to look at the text to see if this is possible
... 3.8 is specifically on possibility to divide the evaluation into multiple evaluations
... this is a question we'd have to answer, will that be okay or not

<agarrison> The methodology should be able to verify a conformance claim - which could be a single page, sub domain or whole website.

allistair: should be the case that the methodology can verify that it's a conformance claim to be made, whether single page or website, can see that as quite useful

<Detlev> agree

allistair: uses the conformance claim as a kind of scope, we would be able to say that had done what they said they'd done, would support WCAG 2 in a much better way

Eric: what about a smaller amount of pages?

allistair: conformance claim is design to support single or multiple pages, using this as a scoping document, reproducable for other people to use

Eric: then you go away from a full website

allistair: you could do either, and you could do it in any case

Eric: methodology to support one-page evaluation or a full website evaluation?

allistair: do you want make a conformance claim for a single web page or a full website?

<Tim> ATAG also has conformance claim info (may be relevant to our scoping discussion)

allistair: could be quite useful, could be passed around, and you can specifiy the URLs, so if you update your website, you only need to test the new content, might optimize the process

<Tim> http://www.w3.org/TR/ATAG20/#conf-claim

Eric: so the conformance claim would not be for the full website, but only the sample?

allistair: no, whatever the claim is is the scope, defines it, and then you could take a sample out of that

Detlev: I agree fully with allistair, we increasingly have a situation with websites combined from several bits and pieces, different sources, it's much better to be able to make a specific conformance claim about something

<agarrison> For clarity the WCAG 2.0 conformance claim is used as the scope, and the sample is taken from that

Detlev: rather than ruling out a full conformance because you can't encompass all of it, statement needs to be clearly tied to the pages tested, don't want to lead people to think it's the full website that conforms, to specify what's included and what's excluded

Eric: a really different approach to scope than what we have currently
... the conformance claim decides what the scope is? so we don't first state scope and then choose a sample, but the conformance statement of claim defines what the scope is, not necessarily a full website review

<kerstin> don't understand

allistair: that seems to be the intention behind WCAG 2.0

+1 I agree because of how large applications and corporate websites are created (multiple content providers) - I think you should have both options, parallel paths

Vivienne: there's already guidelines about how to evaluate and put conformance claims on specific pages in WCAG, thought we'd ruled that out and our methodology would encompass full websites

Eric: the idea is now that you'd make a conformance claim based on choice, which would define scope, but you'd only be doing part of our methodology

shadi: not absolutely sure that it's really that far away from what we've been discussing, depends on what the big scope is, take a website with many applications or subdomains or parts, and even if you wanted to evaluate a subpart (ex. library section), you still have to do quite a bit of sampling and work to determine even the scope

<agarrison> A website owner makes a conformance claim for their website - surely they must be able to use the evaluation method we are producing to see if this conformance claim is verifiable

shadi: so you may not be doing the entire site, but still fairly large set of pages/resources that you need to put in sampling strategy to make a claim about that subsection, so for individual pages, if you can count the pages you're evaluating, this methodology doesn't make sense (not the point that we're doing), but we need to be careful not to exclude important usage
... (ex. purchase a new application for your company site) need a way to verify conformance and not the entire site

Eric: but then it's not a full website evaluation anymore

shadi: depends on what the conformance claim is for, if I want to make a claim about the entire website, then the selection/sampling would have to be for the entire website

<Detlev> agree

shadi: can't do a sampling of subsection and say it applies to the entire website, but the sampling used would apply only to the subsection

Eric: does make things much easier, we can drop a lot of areas about scope, since scope would be dependent on the conformance claim

allistair: if it's a broad scope, you still need to be able to sample

Eric: the scope section can be replaced by much shorter text saying it's dependent on conformance claim, but also drop the idea of a full website evaluation

<Detlev> Can we go back to working wit hthe queue?

Eric: this isn't a full website evaluation then, are we still planning the same thing

<kerstin> +1 am confused also

shadi: approach is in there, scope section would not be less useful, contextualized and have it either open-ended (entire website or specific subsection)

Eric: not if you say that the conformance claim is the scope
... could leave it there, but if we take this approach, we're not talking about full websites anymore, so a lot of these may not be relevant

shadi: WCAG statement of partial conformance is different than what we're talking about here

<Detlev> Thanks Shadi, that'S what I was getting at!

shadi: if I specify the web pages, still have to use the steps, the samples, etc that define scoping, not okay to say "take pages 1,2,3 and drop 4" and arbitrarily say that it conforms
... this could apply on smaller parts of a site

Mike: maybe part of the problem is how conformance is defined in WCAG 2.0 (lowest common denominator method)
... can't claim a conformance level of entire site unless you've looked at the entire site
... that doesn't exclude sampling, don't know that we'd be in conflict if we said that a portion of the site is in conformance, maybe we say it meets the WCAG 2.0 AA criteria

Eric: so you could split the website into different parts (one part WCAG AA, another part WCAG A)

Mike: if we don't use the word "conformance" we could slide by that issue of whether the site conforms or not, but don't think you can say conformance unless you look at the entire site

Martin: we'd never have a conformance claim on a large site then, which is why we're looking at sampling
... if we start splitting up into smaller sections, then the claim for a website will be very confusing for everyone who reads it

<agarrison> http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Dec/0023.html

<agarrison> This exact issue was raised in 5 dec

<shadi> +1 to Elle

allistair: this has been raised on Dec 5th about this exact issue, how many people have read the conformance claims of WCAG 2? you can say your entire website is AA compliant and use your own methodology to back it up, but if we can't allow our methodology to support that, it'll be silly

<Detlev> good stuff to discuss in the next call!

allistair: if we limit our methodology to full websites, we're telling people what they can use that for and not what they need it for, what's the benefit, is there a seal for full website conformance?

Mike: on conformance page, thought it had to do with entire site, but really has to do with a single page

Eric: to your earlier comment, allistair, WCAG 2 conformance is at a page level, but our methodology is for full websites

allistair: what's the benefit of that for full websites? why would I want to pay you money to tell me if my entire website is compliant in terms of you giving me a conformance claim?

Eric: you see a lot of evaluations of websites, can imagine that someone wants a conformance claim on a site, people with disabilities would like to know if someone's making that claim, it's for the full website

allistair: conformance claim is what you will give out at the end of your methodology
... what about adding new content after a full website evaluation, then I have to do the full website again?

Eric: you don't need a full methodology to choose samples, scopes, and you want to have a conformance claim for one page

<vivienne> In Australia, government departments need to be able to state what level their website conforms to. We need a full website evaluation.

Eric: the question is, do we want people to give a conformance claim for a 1,000,000 pages, or do we want people to use the methodology to make a conformance claim for a full website?

allistair: I'd like a website only to define what they're claiming conformance for, and to say whether or not those pages do what they say with regards to conformance
... I'd like to validate conformance claims and use the methodology to support WCAG 2

Eric: I would like to do the full website and not limit it to just whatever a website owner claims conformance over

shadi: call has to be closed now

<Mike_Elledge> Can we say "Conformance subject to the scope of the review"?

Eric: will put this topic on agenda for next meeting
... missed at least 3 items on the agenda, but this seems to be a really important thing to talk about, agenda for list this week and next meeting

Eric: thanks for joining and giving opinions