14:12:05 <RRSAgent> RRSAgent has joined #eval
14:12:05 <RRSAgent> logging to http://www.w3.org/2012/04/19-eval-irc
14:12:07 <trackbot> RRSAgent, make logs world
14:12:09 <trackbot> Zakim, this will be 3825
14:12:09 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start 12 minutes ago
14:12:10 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
14:12:10 <trackbot> Date: 19 April 2012
14:12:21 <MartijnHoutepen> sorry, started this a bit  to late
14:12:37 <Elle> Eric: important to describe scope, chosen target, context, use, etc. and sequence of steps, so that another evaluator might try to do the same thing following the same sequence of steps and variables (time of day, user profile, etc.)
14:12:42 <Detlev> q+
14:12:53 <korn> q+
14:13:01 <Elle> Eric: about step 5a, provide documentation for each step, opinions? sufficient description?
14:13:02 <ericvelleman> q?
14:13:05 <Detlev> ack me
14:13:20 <Detlev> ack me
14:13:32 <Detlev> Zakim, unmute me
14:13:32 <Zakim> sorry, Detlev, I don't know what conference this is
14:13:48 <Detlev> will redial
14:13:55 <Moe> q+
14:14:02 <Mike_Elledge> Mike_Elledge has joined #eval
14:14:04 <Sarah_Swierenga> q+
14:14:10 <ericvelleman> q?
14:14:32 <Elle> Peter: couple of comments on the housekeeping level (evaluation is misspelled), documentation for each step, might want to say "as part of documentation"
14:15:13 <Elle> Peter: concerned about documenting, the size of this documentation may be large for large websites, important before we finalize this to have some idea of how large this might be and decide if we're comfortable with that
14:15:25 <Elle> Peter: other concern is that we may be overloading the word "step
14:15:28 <Detlev> q+
14:15:49 <Detlev> Had the same comment about step
14:15:55 <Detlev> q-
14:15:57 <Elle> Peter: concerned about what is a step of walking through the application versus a step of the evaluation, we need to be careful about when we mean which
14:15:59 <korn> q-
14:16:06 <Elle> Eric: agree, should be clarified
14:16:28 <Elle> Eric: also size is necessary to put in the minutes as something to look at, we do say publishing is optional
14:16:29 <Detlev> q+
14:16:45 <MartijnHoutepen> q- moe
14:16:49 <Elle> Moe: question regarding bullet #3, web pages included in the sample
14:17:11 <Elle> Moe: are we thinking they'd just record the URL of the page? as Peter mentioned, documentation could get quite lengthy... what level of recording the pages are we looking at?
14:17:40 <Elle> Eric: thinking we do describe (Step 3 if I'm not mistaken) the sample, also the sample used for the audit, so what you'd point to in the Appendix
14:17:53 <Elle> Eric: also dependent on goal of evaluation, if just fail/pass, not sure if it's necessary to publish the sample
14:18:21 <Elle> Moe: if there is a dynamic application, one base URL but content consistently changes, do they record that base page of the application or parts of the application tested?
14:19:00 <Elle> Eric: how do we record it, if we sample the website and it's one page but it's an extensively dynamic page, not enough to say "this is the page address" because it could just be checking part of what happens in that page
14:19:19 <korn> q+
14:19:24 <Elle> Eric: WCAG 2 is on the page level, so if it's all on one page, we have to test all that's there, but how to record it, not sure
14:19:35 <korn> I have a direct response to that question
14:19:44 <MartijnHoutepen> q- Detlev
14:19:45 <ericvelleman> q?
14:20:25 <Elle> Detlev: part of confusion on section on documentation was that step would be a nice one for complete process, fill in form, error message, confirmation page... but we have used Step for overall methodology
14:20:40 <Elle> Detlev: maybe the sections can be renamed so we can reserve Step
14:21:04 <Elle> Detlev: checking dynamic things, easiest thing might be to document the start/landing page for the process and then describe in the documentation the number of steps you have to follow to arrive at the end of the process
14:21:25 <Elle> Detlev: say, put something in the search form, what you input, then follow hit links on the page
14:21:49 <Elle> Detlev: describing it succinctly means you describe the process that you have to follow, making it independent of the base URL
14:22:18 <Elle> Eric: clear that there is confusion about the words Step and Sequence, maybe renaming the sections would be better and using Step as described
14:22:43 <Elle> Eric: the question still arises about the size of the documentation if you have to describe in such detail, could make this in a large document
14:23:09 <Elle> Detlev: when we do this, it's usually just a few bullet points, but complex applications may get very complicated, though, yes
14:23:57 <Elle> Sarah: thinking about the amount of documentation as well, but many evaluators would like to capture just the URL and call it good, but this approach here is more like what we do in our labs, describing the process, so that the person coming after you can identify dynamic components on the page
14:24:38 <ericvelleman> q?
14:24:42 <Elle> Sarah: sometimes evaluators don't know everything that could be described with just that base URL, but a general description of what worked and didn't, our overall goal is "does dynamic functionality work" - we may not get every one, though
14:24:50 <MartijnHoutepen> q- Sarah_Swierenga
14:24:58 <Detlev> zakim, mute me
14:24:58 <Zakim> sorry, Detlev, I don't know what conference this is
14:25:07 <Elle> Peter: on the notion of Steps, we can also address this with an adjective "evaluation steps" versus "testing steps"  as an option
14:25:35 <MartijnHoutepen> Zakim, this is WAI_ERTWG
14:25:35 <Zakim> ok, MartijnHoutepen; that matches WAI_ERTWG(Eval TF)10:00AM
14:25:38 <Elle> Peter: echo comments that others have made about web applications, we're seeing more and more of those and it's really one of the core use cases for the evaluation methodology, more than just straight WCAG evaluation of a single plage
14:25:40 <Elle> page*
14:25:43 <Detlev> Zakim, mute me
14:25:43 <Zakim> Detlev should now be muted
14:25:54 <Detlev> Zakim, unmute me
14:25:54 <Zakim> Detlev should no longer be muted
14:26:03 <korn> q-
14:26:04 <Elle> Peter: the whole concept of "page" is breaking down, and we really need to keep this in the back of our minds
14:26:07 <Detlev> Zakim, mute me
14:26:07 <Zakim> Detlev should now be muted
14:26:40 <Elle> Eric: something we describe also in the requirements, yes, I think then the solution given by people previously (documenting with a general description and process) might be a good solution
14:26:40 <agarrison> q+
14:26:49 <MartijnHoutepen> sorry Detlev, seems to be my mix-up. This is what happens if Shadi is not here ;-)
14:26:54 <ericvelleman> q?
14:27:20 <Elle> Allistair: just spent 2 weeks documenting a very small application, it does take a substantial amount of time to do that
14:27:36 <MartijnHoutepen> q- agarrison
14:27:52 <Elle> Allistair: a lot of people who have that documentation already have that in play, can we not get a copy of their use cases document and work through specific things that they intended the web application to be used for, and we can then say if those paths are accessible or not?
14:28:12 <Elle> Eric: this can be something we cover in an earlier step, reporting step as well?
14:28:38 <Elle> Allistair: pick up this documentation from those who have built the application, problem is longer time writing about website than testing it
14:29:06 <Elle> Allistair: if we spend that much time, fewer assessments and possible changes by that time, so needs to be very rapid, have mechanisms to take in as much information as is already there
14:29:24 <Sarah_Swierenga> q+
14:29:42 <Elle> Eric: would be great to gather them, and an evaluator could always decide if there's another use case needed, then you could point to it on step on reporting and use it on step on auditing
14:29:49 <Detlev> q+
14:30:02 <Tim> Tim has joined #eval
14:30:19 <ericvelleman> q?
14:30:20 <Elle> Eric: marked in Step 2, if people agree that it would be good to ask the owner of the website for the use cases that they have for their website, and include that in deciding what use cases to use for the evaluation
14:30:57 <Elle> Sarah: the use cases may not actually point to individual URLs, but developers know where the dynamic functionality is on the page, maybe we can leverage that information
14:31:01 <Elle> +q
14:31:32 <Elle> Sarah: when you're describing, if you use this level of detail in reporting, the reader can tell where in the page they were focusing on for the pass/fail of the particular component or part of the page
14:31:33 <Detlev> Zakim, unmute me
14:31:33 <Zakim> Detlev should no longer be muted
14:31:47 <MartijnHoutepen> q- Sarah_Swierenga
14:31:53 <MartijnHoutepen> q- Detlev
14:32:18 <Elle> Detlev: reporting where to report findings, if you have a process that spans several pages, the question is where to put that, our tool is to select pages as page sample and attach all your criteria to that page
14:32:45 <Zakim> -Sarah_Swierenga
14:32:48 <Elle> Detlev: if that page is a starting page of a longer process, we get some slight confusion on whether all those pages have a sample, where to document
14:33:09 <Elle> Detlev: not sure if we can go into such specifics, or leave it up to the application reporting web evaluation, or how it will be organized
14:33:22 <Detlev> Zakim, mute me
14:33:22 <Zakim> Detlev should now be muted
14:33:52 <Elle> Eric: tried to cover this in specific sequences, maybe not clear enough, add examples perhaps, bullet point 1 says "include complete process"
14:34:29 <agarrison> q+
14:34:40 <ericvelleman> q?
14:34:45 <MartijnHoutepen> q- Elle
14:35:47 <korn> 10 Euro cents?
14:36:34 <Elle> Allistair: with regards to reporting, collecting all the different pages that we have in our sample, we used to do all of this, and no one ever re-checks the audit reports after they were given to the client
14:36:46 <Elle> Allistair: the idea of keeping all those pages, who will audit the auditor?
14:36:57 <Elle> +q
14:37:12 <Elle> Eric: do we want to keep all these pages?
14:37:28 <Elle> Eric: relates to Step 4D: archiving web pages for reference
14:37:37 <Elle> Eric: direct result of that in 5, add web pages into sample
14:37:52 <ericvelleman> q?
14:37:54 <Elle> Eric: does anyone ever use this, or is there a real reason to keep this?
14:38:00 <MartijnHoutepen> -agarrison
14:38:03 <Detlev> q+
14:38:07 <MartijnHoutepen> q- agarrison
14:38:16 <ericvelleman> q?
14:38:20 <Mike_Elledge> q+
14:38:51 <Zakim> +Tim_Boland
14:39:35 <MartijnHoutepen> Zakim, Tim_Boland is Tim
14:39:35 <Zakim> +Tim; got it
14:39:47 <ericvelleman> q?
14:39:48 <Moe> q+
14:39:52 <Detlev> Zakim, unmute me
14:39:52 <Zakim> Detlev should no longer be muted
14:40:48 <Elle> Detlev: just wanted to mention that there can be cases where it's useful to have at least the URL of the page tested, complaints and discussions about how you rated certain points, it's good to be able to return to that page, some things cannot be covered in a screenshot, but URLs is certainly a minimum
14:40:50 <agarrison> q+
14:40:57 <Elle> -q
14:41:00 <MartijnHoutepen> q- Elle
14:41:04 <Detlev> Zakim, mute me
14:41:04 <Zakim> Detlev should now be muted
14:41:06 <MartijnHoutepen> q- Detlev
14:41:17 <MartijnHoutepen> q- Mike_Elledge
14:41:18 <korn> q+
14:42:09 <Elle> Mike: understand the need to keep things as light weight as possible, a matter of course to have set of URLs and screenshots, really helpful for reference in a report (able to take a look at the page in particular), may also be helpful to identifiy what was reviewed last time to set up a plan for revisiting at a later date, memory costs are low, so URLs and screenshots seem to be a good way to track
14:42:12 <MartijnHoutepen> q- Moe
14:43:10 <Elle> Moe: other scenarios: retail application, pages are based on templates, if the URLs change, templates don't change.... another is a search application, where the content changes but the template is essentially the same
14:43:25 <Elle> Moe: we are talking about a sample, but if we record these pages, the content may change but the template will not
14:43:47 <Elle> Moe: taking a snapshot of a page in time that may change, gets trickier when the content regularly changes
14:44:29 <Detlev> q+
14:44:41 <Elle> Eric: Step 2 A identifies naming templates, something we're looking at, but should add to reporting
14:45:00 <Elle> Moe: talking about something slightly different, an actual web page template
14:45:42 <Elle> Eric: 2A should mean the web page templates (not evaluator templates) where content is inside, but forgot to move it to reporting section, so will add it there
14:45:51 <ericvelleman> q?
14:46:14 <Elle> Allistair: we used to download the pages and they didn't work as they did previously, so we had to correct it all to work locally, which was painful, and also changed the actual pages
14:46:30 <Elle> Allistair: a lot of things were code violations, so screenshots didn't really help
14:46:48 <Elle> Allistair: recording the URLs are a no-brainer, and a lot of the report points to URLs
14:47:25 <Elle> Allistair: could we ask people who are in charge of the content management system (CMS) to give us templates used for the website so that we can separately look at all the content in the empty template, and then look at them in context somewhere else?
14:47:39 <Elle> Allistair: how do we as evaluators discover all the templates on a 2 million page website?
14:48:02 <Elle> Eric: we need to clarify this in the text, it's more or less what we mean in 2A about identify all the templates
14:48:24 <Elle> Eric: this would be the best moment to ask the website owner: where are you templates, use cases, etc.?
14:48:45 <Elle> Eric: identified in Step 1 and Step 2, and then it should also be in the reporting section
14:48:54 <Elle> Eric: will try to clarify that in Step 1 and 2
14:48:56 <MartijnHoutepen> q- agarrison
14:49:53 <Elle> Eric: evaluator still has the responsibility to look beyond just the templates, but agreed, needs clarification
14:50:14 <korn> q+
14:50:17 <Detlev> Zakim, unmute me
14:50:17 <Zakim> Detlev should no longer be muted
14:50:17 <Elle> Peter: time check, request about other items
14:50:25 <Elle> Eric: closing down 5A for now
14:50:42 <Elle> Detlev: regarding the word template, pages that consist of different elements created by various teams
14:51:11 <Elle> Detlev: impossible to basically track all of them, so we need definition about what the word "template" means, perhaps not always available
14:51:23 <Elle> Eric: proposal in the next version?
14:51:26 <Elle> Detlev: yes please
14:51:39 <Detlev> Zakim, mute me
14:51:39 <Zakim> Detlev should now be muted
14:52:01 <Elle> Eric: worked on more than 5A, but discussion also rendered interesting points for earlier sections, valuable discussion
14:52:32 <Elle> Eric: Step 5B, provide an accessibility statement (optional), looked at a lot of statements, tried to see what would be completely necessary to put in an accessibility statement
14:53:10 <Elle> Eric: what we use for ISO documents: scope, evaluation date, conformance level, use of non-WCAG supported techniques, version of WCAG, version of this Conformance Methodology used
14:53:35 <Elle> Eric: these are points that have to be in official documents in the Netherlands, welcome any additions or discussion
14:53:38 <agarrison> When I use templates I usually am talking about templates in the context of content management systems - the mold into which an editors contents sits.
14:53:53 <ericvelleman> q?
14:54:00 <MartijnHoutepen> q- korn
14:54:25 <Elle> Peter: what does conformance level mean? understanding of WCAG is either you do or you don't conform, is that what's meant?
14:55:00 <Elle> Peter: because this is a statistical sample of a site, we don't actually know perfection, only what we sampled had no issues or had issues... would be more interesting if the accessibility statement was based on what was found and not a simple conformance statement
14:55:12 <korn> q-
14:55:19 <Elle> Peter: a range of statements "everything found conformed" or "everything found largely conformed" etc
14:55:30 <Detlev> q-
14:55:43 <Elle> Eric: we could add how near you get to this conformance level
14:55:55 <Elle> Peter: what does conformance level mean?
14:55:57 <ericvelleman> q?
14:55:58 <Elle> Eric: A, AA, AAA
14:56:07 <Elle> Peter: would be helpful to have this in the text
14:56:37 <agarrison> q+
14:56:43 <Elle> Eric: next one is 5C, performance score
14:56:48 <korn> q+
14:56:57 <Elle> Eric: did not add any text there, will do so in the coming week, information on findings is an optional addition
14:57:09 <ericvelleman> q?
14:57:49 <Elle> Allistair: should read sample conformed to certain level of WCAG 2.0
14:58:01 <MartijnHoutepen> q- agarrison
14:58:17 <Elle> Peter: you mention if something is a failure, provide at least 3 examples of what that error is
14:58:27 <korn> q-
14:58:31 <Elle> Peter: we may not have 3 of the same error or even 3 pages, text needs to be reviewed
14:58:38 <Elle> Eric: yes, will look into that
14:58:50 <Elle> Eric: will work on new version and make additions to conformance methodology in Section 4
14:58:59 <Elle> Eric: next version will be ready by Monday or early Tuesday
14:59:23 <Zakim> -Detlev
14:59:38 <Elle> Eric: thank you all for being here and discussing, please go on discussing the other sections we didn't have time for (5E, etc.) on the list
14:59:48 <Elle> Eric: have a good weekend!
14:59:50 <MartijnHoutepen> Zakim, unmute me
14:59:50 <Zakim> MartijnHoutepen should no longer be muted
14:59:51 <Zakim> -korn
14:59:51 <Elle> bye, all!
14:59:52 <Zakim> -agarrison
14:59:52 <Zakim> -Don
14:59:52 <Zakim> -Maureen_Kraft
14:59:54 <Zakim> -Mike
14:59:55 <Sarah_Swierenga> bye
14:59:56 <Zakim> -Tim
14:59:56 <Zakim> -Liz
15:00:03 <korn> korn has left #eval
15:00:19 <MartijnHoutepen> trackbot, end meeting
15:00:19 <trackbot> Zakim, list attendees
15:00:19 <Zakim> As of this point the attendees have been Don, ericvelleman, +31.30.239.aaaa, +49.403.17.aabb, Liz, +1.517.927.aacc, +49.403.17.aadd, MartijnHoutepen, korn, Detlev, Sarah_Swierenga,
15:00:22 <Zakim> ... Maureen_Kraft, +1.502.632.aaee, Elle, agarrison, Mike, Tim
15:00:23 <Zakim> -Elle
15:00:27 <trackbot> RRSAgent, please draft minutes
15:00:27 <RRSAgent> I have made the request to generate http://www.w3.org/2012/04/19-eval-minutes.html trackbot
15:00:28 <trackbot> RRSAgent, bye
15:00:28 <RRSAgent> I see no action items