14:42:32 RRSAgent has joined #eval 14:42:32 logging to http://www.w3.org/2012/01/26-eval-irc 14:42:34 RRSAgent, make logs world 14:42:34 Zakim has joined #eval 14:42:36 Zakim, this will be 3825 14:42:36 ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 18 minutes 14:42:37 Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference 14:42:37 Date: 26 January 2012 14:42:58 chair: Eric 14:43:47 regrets: Kathy 14:44:23 agenda+ Specific discussion on section 3.3, 3.4 and 3.8 14:44:35 agenda+ Specific discussion on section 5.1 Manual and machine evaluation 14:44:56 agenda+ Evaluation section 14:53:46 WAI_ERTWG(Eval TF)10:00AM has now started 14:53:52 +houtepen 14:54:29 MartijnHoutepen has joined #eval 14:55:06 zakim, call shadi-617 14:55:06 ok, shadi; the call is being made 14:55:07 +Shadi 14:56:56 zakim, mute me 14:56:56 sorry, MartijnHoutepen, I do not know which phone connection belongs to you 14:57:03 +[IPcaller] 14:57:16 zakim, IPcaller is me 14:57:16 +vivienne; got it 14:57:34 zakim, who is on the phone 14:57:34 I don't understand 'who is on the phone', MartijnHoutepen 14:58:45 +[IPcaller] 14:59:10 zakim, ipcaller is Kerstin 14:59:10 +Kerstin; got it 14:59:14 kerstin has joined #eval 14:59:18 zakim, mute me 14:59:18 vivienne should now be muted 14:59:36 zakim, mute me 14:59:36 Kerstin should now be muted 14:59:56 Detlev has joined #eval 15:00:52 + +1.502.632.aaaa 15:01:09 zakim, aaaa is Elle 15:01:09 +Elle; got it 15:01:19 +Mike 15:01:19 ericvelleman has joined #eval 15:01:21 +Detlev 15:01:46 Elle has joined #eval 15:01:56 unmute me 15:02:02 mute me 15:02:09 zakim, mute me 15:02:18 Detlev should now be muted 15:02:23 zakim, mute me 15:02:24 scribe: Elle 15:02:34 sorry, MartijnHoutepen, I do not know which phone connection belongs to you 15:02:49 zakim, who is on the phone? 15:02:54 On the phone I see houtepen, Shadi, vivienne (muted), Kerstin (muted), Elle, Mike, Detlev (muted) 15:02:57 Mike_Elledge has joined #eval 15:03:00 +ericvelleman 15:03:04 zakim, houtepen is MartijnHoutepen 15:03:06 +MartijnHoutepen; got it 15:03:15 zakim, mute me 15:03:15 MartijnHoutepen should now be muted 15:03:31 + +1.301.221.aabb 15:03:44 zakim, aabb is Liz 15:03:44 +Liz; got it 15:03:59 agenda? 15:04:35 liz has joined #eval 15:04:43 +1 for voice recognition :-) 15:04:59 SarahSwierenga has joined #eval 15:05:08 no, I can't go 15:06:17 agarrison has joined #eval 15:06:32 agenda? 15:06:50 zakim, mute me 15:06:50 Shadi should now be muted 15:06:56 I'll be a little late joining - maybe another 5 mins 15:07:07 Eric: Discussion online about fact that a lot of text is coming from the UM document, explained in earlier calls 15:07:11 Topic: Welcome 15:07:24 s/UM/UWEM 15:07:32 Eric: delivered a document (Unifed Web Evaluation Methodology = UM) 15:08:11 Eric: what happened (three years or so ago) was really valuable, but most of discussion was with WCAG 1.0, terminology is usable, but can we use this for WCAG 2.0? 15:08:18 +??P35 15:08:39 just joined 15:08:52 zakim, ??p35 is agarrison 15:08:52 +agarrison; got it 15:08:54 Eric: do see text coming from there, but could change completely as a result of this task force group discussion 15:08:59 ack me 15:09:14 Eric: will discuss further with shadi on copyright issues, but not needed for this call 15:10:11 shadi: wasn't aware that there was specific text directly coming from UM, so if the text is specifically taken from UM or structure, we may need a small acknowledgement section, let's follow up separately and see how much of that, but yes, should certainly try to use existing work 15:10:13 zakim, mute me 15:10:14 Shadi should now be muted 15:10:47 Eric: have a read, look for "UM" - will send a link, Alice also sent direct link in the email discussions, see if there's valuable stuff that we might want to use or use for inspiration 15:11:26 Eric: side products, one is on indicators and indicator refinement, discussion about how can you get a random sample that is really a good sample of a website? 15:12:18 Eric: saw a few papers that had links, read them, found another paper that was made inside this cluster that also addresses this, still looking based on discussion last week, When do we stop, how to we find the sample? trying to find scientific data before shutting down that discussion 15:12:33 q+ 15:12:53 Eric: specifically in this tool, assistance-part, there's a possibility to use a tool to get a real random sample, question is, how many pages would be a good random sample? if you know of literature about that, please send a link to use in discussion 15:13:01 links to you, eric, or to the mailing list? 15:13:06 Eric: trying to close down how many pages is large enough or small enough 15:13:10 q? 15:13:39 allistair: there have been other discussions about sample pages, where are we storing these ideas via email? 15:14:07 Eric: they are stored on the mailing list, problem with putting them in the document is that we didn't really agree about how many pages 15:14:41 allistair: in the end, I said I'd be happy with a smaller sample, but there's no mechanism to know what we're agreeing on if we're storing it in the emails 15:14:50 +Sarah 15:14:54 allistair: other things that we are deciding on are not getting into the document 15:15:13 allistair: can weekly updates note what we're agreeing on? 15:15:41 Eric: trying to update document following outcome of the teleconferences, but to your point, it should be earlier than what was done recently, try to do this sooner 15:15:50 ack me 15:15:52 allistair: what about putting it into the W3C? 15:16:26 Eric: still very much in an editor draft, tools to see differences in documents (change logs), but could increase administration 15:17:12 shadi: not entirely sure that subversion is very easy for everyone to use, what you're talking about from HTML5 group has three co-chairs, don't think it's fully automatically generated, but we can look at more stuff to support 15:17:40 zakim, mute me 15:17:40 Shadi should now be muted 15:17:40 shadi: would really suggest that you point out things that have been dropped or not yet integrated, but with approximately 80 emails a week, it takes time to integrate these comments into the document 15:17:55 Eric: if there's something that wasn't put in the document, please just add it to the list and I'll add it 15:18:19 sounds good 15:18:20 Eric: try to put all the outcome of these discussions into the document, if there's something missing, please let me know what I missed 15:18:35 q? 15:18:44 q- 15:18:53 zakim, take up next 15:18:53 agendum 1. "Specific discussion on section 3.3, 3.4 and 3.8" taken up [from shadi] 15:19:23 regrets+ Vincent 15:19:24 Eric: did make some changes that we discussed in the last call, short discussion (side discussion) that urged me to make a change, dropped the barrier section and added a section in the sample section 15:20:22 Eric: added a section (4.3) in the sampling of pages called "stop criteria" so that is new and it came out of the discussion we had in the past meetings, dropping barrier recognition (5.4) - still there, but noted as we will drop this 15:20:42 Eric: in the discussion, we said that "barrier recognition" was not the right terminology, and we didn't want to make a choice about what was more/less important as a barrier 15:21:07 Eric: added the idea that we could stop evaluating more pages, so created temporary title "Stop Criteria" with a really short description 15:21:27 Eric: so if you're sampling pages, core sample (homepage, search page, etc), the task oriented sample, and the random sample 15:21:33 q+ 15:21:39 Eric: for the random sample, the question is how many pages would be in the random sample to cover the website as a whole 15:21:57 Eric: introduced the term "stop criteria" here, propose a discussion (not on this call) but note that it's totally new in the document 15:21:57 q? 15:22:10 ack ag 15:22:18 allistair: when did we actually decided on these sampling things? 15:22:26 allistair: did we actually decide and finalize that? 15:22:46 allistair: stop criteria seems the same as barrier criteria, are we carrying that over from the previous document or do we actually need it? 15:22:57 Eric: as far as sampling is concerned, we agreed on 3 forms (core, task, and random) 15:23:06 Eric: put text there for the discussion 15:23:11 ack me 15:23:17 q+ 15:23:17 allistair: can't remember sending agreement, pretty much copied from the UM 15:23:26 Eric: yes, from UM with changes to fit WCAG 2, but still need discussion 15:23:31 allistair: very important area 15:23:36 zakim, unmute me 15:23:37 Kerstin should no longer be muted 15:24:05 Eric: closed for part, start of scope, entering flesh into the document based on what people responded, the direction put in, but not definitive text in any way 15:24:12 q- 15:24:14 q? 15:24:44 shadi: from a process perspective, if there is text coming from UM, we have to acknowledge that and add a small acknowledgement that those sections are from UM 15:25:14 zakim, unmute me 15:25:14 Kerstin was not muted, kerstin 15:25:22 shadi: regarding a group resolution, this is an editor draft at this stage, quite normal for things to appear as new, changed in an ad-hoc basis, but when we publish a working draft there will be an opportunity for the group to approve, small W3C process 15:25:34 shadi: there won't be a formal publication without group agreement 15:25:36 zakim, mute me 15:25:36 Kerstin should now be muted 15:26:19 q- 15:26:23 zakim, mute me 15:26:23 Shadi should now be muted 15:26:25 q+ 15:26:25 q? 15:26:28 shadi: having it in there doesn't imply approval, just there for everyone to discuss 15:26:32 ack me 15:27:23 Conformance is defined only for Web pages. However, a conformance claim may be made to cover one page, a series of pages, or multiple related Web pages. 15:27:25 Detlev: touching on sampling, wondered whether the methodology would only cover scenarios where you want to evaluate a whole website or part of a website, defining core resources, random pages would apply, reading the WCAG text it seems that there's also the option to evaluate one single page 15:27:45 q? 15:27:49 Detlev: would our methodology include that, or conditional decision? 15:28:05 Detlev: conformance claim and include one URL, for example 15:28:30 q+ 15:28:39 q? 15:28:45 Eric: we have the methodology for a full website, but discussion about what is a full website (where it starts and ends) still need more discussion on it, so that two or more people who evaluate the same website and have the same result 15:28:49 Tim has joined #eval 15:29:11 Detlev: can you make a conformance claim for a single web page? would that be something that the methodology could support or would it be ruled out? 15:29:26 Eric: right now, it's ruled out, complete website 15:29:29 q? 15:29:31 ack me 15:30:04 Vivienne: when talking about websites, does that encompass web applications as well? we do things that are hosted by a company and maybe made available to specific members of the public or kept for in house applications 15:30:21 q+ 15:30:27 Vivienne: know that WCAG covers intranet, with our methodology, will that cover web applications? 15:30:53 please send link to the list, Vivienne! 15:30:57 Vivienne: 10% of websites need to be evaluated, will we be including that? 15:30:57 http://www.w3.org/WAI/ER/conformance/ED-methodology-20120125.html#definitionsandterminology 15:31:04 Eric: can you send a link? 15:31:10 q? 15:31:13 zakim, unmute me 15:31:13 Kerstin should no longer be muted 15:31:16 [[Website - A coherent collection of one or more related web pages that together provide common use or functionality. It includes static web pages, dynamically generated web pages, and web applications.]] 15:31:17 okay guys, I'll send it to the list 15:31:19 Eric: regarding web applications, specifies dynamic and static applications, so it's covered, yes 15:31:39 Kerstin: if we have a one-pager, will it be possible to evaluate the single page with our methodology? 15:31:45 q+ 15:31:46 Eric: website would only have one page? 15:31:49 Kerstin: yes 15:31:51 q- k 15:32:21 +Tim_Boland 15:32:28 Eric: I think yes, but not quite sure, don't see a lot of websites with only one page, but we do have language in the sampling with "if available" so if not available, then the website is only one page, then that'd be the full website 15:32:33 q? 15:32:39 ack me 15:32:39 Kerstin: I think it would be a good solution 15:33:02 zakim, unmute me 15:33:02 Kerstin was not muted, kerstin 15:33:10 shadi: I think a one-page or even small 5 page site is really not the challenge, since chances are your sample will cover all the requirements if you just select 5 pages (you'd meet the requirements of the sampling) 15:33:49 ack me 15:33:53 shadi: if I understand Detlev correctly, if you have a website and you only choose a sub-section (like a subdomain or a specific part that belongs to a department), it might be useful to be able to say "this part of a website is accessible" but potential for misuse, but might be interesting 15:34:05 q+ 15:34:20 zakim, mute me 15:34:20 Kerstin should now be muted 15:34:23 Detlev: that's what I was talking about, if you want to look at one particular page or aspect, would it be okay to be able to say "this page conforms" but the rest do not, would the methodology support that? 15:34:42 shadi: if it's one page, why not use WCAG? what does the methodology provide for added value? 15:34:48 Detlev: maybe not 15:35:21 zakim, mute me 15:35:21 Shadi should now be muted 15:35:26 Eric: maybe not a page, but a technology perhaps (ex. forms), some people request for testing according to specific types of technology, need to look at the text to see if this is possible 15:35:48 Eric: 3.8 is specifically on possibility to divide the evaluation into multiple evaluations 15:36:00 q? 15:36:02 Eric: this is a question we'd have to answer, will that be okay or not 15:36:20 Zakim, mute me 15:36:20 Detlev should now be muted 15:36:23 The methodology should be able to verify a conformance claim - which could be a single page, sub domain or whole website. 15:36:27 allistair: should be the case that the methodology can verify that it's a conformance claim to be made, whether single page or website, can see that as quite useful 15:37:09 agree 15:37:17 allistair: uses the conformance claim as a kind of scope, we would be able to say that had done what they said they'd done, would support WCAG 2 in a much better way 15:37:24 Eric: what about a smaller amount of pages? 15:37:45 allistair: conformance claim is design to support single or multiple pages, using this as a scoping document, reproducable for other people to use 15:37:54 Eric: then you go away from a full website 15:38:04 allistair: you could do either, and you could do it in any case 15:38:14 q+ 15:38:19 q+ 15:38:20 Eric: methodology to support one-page evaluation or a full website evaluation? 15:38:23 q? 15:38:36 allistair: do you want make a conformance claim for a single web page or a full website? 15:38:52 ATAG also has conformance claim info (may be relevant to our scoping discussion) 15:39:02 allistair: could be quite useful, could be passed around, and you can specifiy the URLs, so if you update your website, you only need to test the new content, might optimize the process 15:39:03 http://www.w3.org/TR/ATAG20/#conf-claim 15:39:19 Eric: so the conformance claim would not be for the full website, but only the sample? 15:39:25 zakim, mute me 15:39:25 Shadi was already muted, shadi 15:39:32 q? 15:39:35 ack me 15:39:35 allistair: no, whatever the claim is is the scope, defines it, and then you could take a sample out of that 15:40:03 Detlev: I agree fully with allistair, we increasingly have a situation with websites combined from several bits and pieces, different sources, it's much better to be able to make a specific conformance claim about something 15:40:21 q? 15:40:48 For clarity the WCAG 2.0 conformance claim is used as the scope, and the sample is taken from that 15:40:49 Detlev: rather than ruling out a full conformance because you can't encompass all of it, statement needs to be clearly tied to the pages tested, don't want to lead people to think it's the full website that conforms, to specify what's included and what's excluded 15:40:52 ack me 15:41:00 Eric: a really different approach to scope than what we have currently 15:41:13 q- 15:41:14 q? 15:41:15 Zakim, mute me 15:41:15 Detlev should now be muted 15:41:18 q+ 15:41:21 agenda? 15:41:27 q+ 15:41:50 q+ 15:42:00 ack me 15:42:02 Eric: the conformance claim decides what the scope is? so we don't first state scope and then choose a sample, but the conformance statement of claim defines what the scope is, not necessarily a full website review 15:42:03 q? 15:42:08 q+ 15:42:09 don't understand 15:42:20 allistair: that seems to be the intention behind WCAG 2.0 15:42:57 +1 I agree because of how large applications and corporate websites are created (multiple content providers) - I think you should have both options, parallel paths 15:43:31 Vivienne: there's already guidelines about how to evaluate and put conformance claims on specific pages in WCAG, thought we'd ruled that out and our methodology would encompass full websites 15:44:02 q+ 15:44:08 Eric: the idea is now that you'd make a conformance claim based on choice, which would define scope, but you'd only be doing part of our methodology 15:44:16 q+ 15:44:16 ack me 15:44:22 zakim, mute me 15:44:23 vivienne should now be muted 15:44:26 q- de 15:44:32 q- vi 15:45:20 shadi: not absolutely sure that it's really that far away from what we've been discussing, depends on what the big scope is, take a website with many applications or subdomains or parts, and even if you wanted to evaluate a subpart (ex. library section), you still have to do quite a bit of sampling and work to determine even the scope 15:45:50 A website owner makes a conformance claim for their website - surely they must be able to use the evaluation method we are producing to see if this conformance claim is verifiable 15:46:11 shadi: so you may not be doing the entire site, but still fairly large set of pages/resources that you need to put in sampling strategy to make a claim about that subsection, so for individual pages, if you can count the pages you're evaluating, this methodology doesn't make sense (not the point that we're doing), but we need to be careful not to exclude important usage 15:46:33 shadi: (ex. purchase a new application for your company site) need a way to verify conformance and not the entire site 15:46:44 Eric: but then it's not a full website evaluation anymore 15:46:53 q+ 15:47:05 shadi: depends on what the conformance claim is for, if I want to make a claim about the entire website, then the selection/sampling would have to be for the entire website 15:47:30 agree 15:47:33 shadi: can't do a sampling of subsection and say it applies to the entire website, but the sampling used would apply only to the subsection 15:47:56 Eric: does make things much easier, we can drop a lot of areas about scope, since scope would be dependent on the conformance claim 15:48:14 allistair: if it's a broad scope, you still need to be able to sample 15:48:45 Eric: the scope section can be replaced by much shorter text saying it's dependent on conformance claim, but also drop the idea of a full website evaluation 15:48:55 Can we go back to working wit hthe queue? 15:48:59 q+ 15:49:15 q+ 15:49:34 q+ 15:49:39 Eric: this isn't a full website evaluation then, are we still planning the same thing 15:49:40 q? 15:50:01 +1 am confused also 15:50:22 q- 15:50:36 q- 15:50:38 shadi: approach is in there, scope section would not be less useful, contextualized and have it either open-ended (entire website or specific subsection) 15:50:56 Eric: not if you say that the conformance claim is the scope 15:51:12 q+ 15:51:19 Eric: could leave it there, but if we take this approach, we're not talking about full websites anymore, so a lot of these may not be relevant 15:51:40 shadi: WCAG statement of partial conformance is different than what we're talking about here 15:52:24 Thanks Shadi, that'S what I was getting at! 15:52:48 shadi: if I specify the web pages, still have to use the steps, the samples, etc that define scoping, not okay to say "take pages 1,2,3 and drop 4" and arbitrarily say that it conforms 15:53:07 zakim, mute me 15:53:07 Shadi should now be muted 15:53:14 shadi: this could apply on smaller parts of a site 15:53:16 ack mike 15:53:20 q- 15:53:44 Mike: maybe part of the problem is how conformance is defined in WCAG 2.0 (lowest common denominator method) 15:54:03 Mike: can't claim a conformance level of entire site unless you've looked at the entire site 15:54:21 regrets+ Emmanuelle 15:54:35 Mike: that doesn't exclude sampling, don't know that we'd be in conflict if we said that a portion of the site is in conformance, maybe we say it meets the WCAG 2.0 AA criteria 15:54:57 Eric: so you could split the website into different parts (one part WCAG AA, another part WCAG A) 15:55:19 q+ 15:55:20 q+ 15:55:24 ack me 15:55:27 q+ 15:55:37 Mike: if we don't use the word "conformance" we could slide by that issue of whether the site conforms or not, but don't think you can say conformance unless you look at the entire site 15:55:48 ack ma 15:55:53 Martin: we'd never have a conformance claim on a large site then, which is why we're looking at sampling 15:56:16 Martin: if we start splitting up into smaller sections, then the claim for a website will be very confusing for everyone who reads it 15:56:29 q? 15:56:30 zakim, mute me 15:56:30 MartijnHoutepen should now be muted 15:56:31 ack elle 15:57:11 http://lists.w3.org/Archives/Public/public-wai-evaltf/2011Dec/0023.html 15:57:24 This exact issue was raised in 5 dec 15:57:28 q- 15:57:53 +1 to Elle 15:58:34 q? 15:58:52 q- 15:59:08 zakim, close queue 15:59:08 ok, shadi, the speaker queue is closed 15:59:50 allistair: this has been raised on Dec 5th about this exact issue, how many people have read the conformance claims of WCAG 2? you can say your entire website is AA compliant and use your own methodology to back it up, but if we can't allow our methodology to support that, it'll be silly 16:00:43 q? 16:00:46 q? 16:00:46 q- 16:00:51 good stuff to discuss in the next call! 16:00:53 allistair: if we limit our methodology to full websites, we're telling people what they can use that for and not what they need it for, what's the benefit, is there a seal for full website conformance? 16:01:02 ack ag 16:01:28 Mike: on conformance page, thought it had to do with entire site, but really has to do with a single page 16:02:32 Eric: to your earlier comment, allistair, WCAG 2 conformance is at a page level, but our methodology is for full websites 16:02:56 allistair: what's the benefit of that for full websites? why would I want to pay you money to tell me if my entire website is compliant in terms of you giving me a conformance claim? 16:03:31 q+ 16:03:34 q+ 16:03:49 Eric: you see a lot of evaluations of websites, can imagine that someone wants a conformance claim on a site, people with disabilities would like to know if someone's making that claim, it's for the full website 16:04:09 allistair: conformance claim is what you will give out at the end of your methodology 16:04:32 q- 16:04:34 allistair: what about adding new content after a full website evaluation, then I have to do the full website again? 16:05:06 Eric: you don't need a full methodology to choose samples, scopes, and you want to have a conformance claim for one page 16:05:06 In Australia, government departments need to be able to state what level their website conforms to. We need a full website evaluation. 16:05:38 Eric: the question is, do we want people to give a conformance claim for a 1,000,000 pages, or do we want people to use the methodology to make a conformance claim for a full website? 16:06:20 allistair: I'd like a website only to define what they're claiming conformance for, and to say whether or not those pages do what they say with regards to conformance 16:06:39 allistair: I'd like to validate conformance claims and use the methodology to support WCAG 2 16:07:03 ack me 16:07:03 Eric: I would like to do the full website and not limit it to just whatever a website owner claims conformance over 16:07:15 shadi: call has to be closed now 16:07:22 Can we say "Conformance subject to the scope of the review"? 16:07:37 zakim, unmute me 16:07:41 Eric: will put this topic on agenda for next meeting 16:08:08 MartijnHoutepen should no longer be muted 16:08:14 Eric: missed at least 3 items on the agenda, but this seems to be a really important thing to talk about, agenda for list this week and next meeting 16:08:22 bye 16:08:23 bye! 16:08:23 thanks, see you next week 16:08:23 Eric: thanks for joining and giving opinions 16:08:23 bye 16:08:25 MartijnHoutepen has left #eval 16:08:26 bye! :) 16:08:27 vivienne has left #eval 16:08:32 -kerstin 16:08:36 -Tim_Boland 16:08:38 -agarrison 16:08:39 -Liz 16:08:44 -Sarah 16:08:46 -Detlev 16:08:47 -Mike 16:08:52 -Shadi 16:08:54 -Elle 16:08:56 -vivienne 16:08:58 -Kerstin 16:08:59 -ericvelleman 16:09:01 -MartijnHoutepen 16:09:04 WAI_ERTWG(Eval TF)10:00AM has ended 16:09:06 Attendees were Shadi, vivienne, Kerstin, +1.502.632.aaaa, Elle, Mike, Detlev, ericvelleman, MartijnHoutepen, +1.301.221.aabb, Liz, agarrison, Sarah, Tim_Boland 16:09:17 ericvelleman has left #eval 16:21:50 trackbot, end meeting 16:21:50 Zakim, list attendees 16:21:50 sorry, trackbot, I don't know what conference this is 16:21:53 RRSAgent, please draft minutes 16:21:53 I have made the request to generate http://www.w3.org/2012/01/26-eval-minutes.html trackbot 16:21:54 RRSAgent, bye 16:21:54 I see no action items