13:54:35 <RRSAgent> RRSAgent has joined #eval
13:54:35 <RRSAgent> logging to http://www.w3.org/2013/07/11-eval-irc
13:54:37 <trackbot> RRSAgent, make logs world
13:54:37 <Zakim> Zakim has joined #eval
13:54:39 <trackbot> Zakim, this will be 3825
13:54:39 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 6 minutes
13:54:40 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
13:54:40 <trackbot> Date: 11 July 2013
13:56:53 <Liz> Liz has joined #eval
13:57:26 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
13:57:33 <Zakim> + +1.301.975.aaaa
13:57:59 <Liz> zakim; aaaa is Liz
13:59:34 <ericvelleman> ericvelleman has joined #eval
14:00:33 <Zakim> +??P10
14:00:39 <Zakim> +[IPcaller]
14:00:46 <ericvelleman> Zakim, P10 is me
14:00:48 <Zakim> sorry, ericvelleman, I do not recognize a party named 'P10'
14:01:06 <ericvelleman> Zakim, ??P10 is me
14:01:06 <Zakim> +ericvelleman; got it
14:01:12 <richard> zakim; ipcaller is richard
14:01:46 <Vivienne> Vivienne has joined #eval
14:02:00 <Zakim> +[IPcaller.a]
14:02:07 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
14:02:09 <shadi> zakim, ipcaller.a is me
14:02:09 <Zakim> +shadi; got it
14:02:15 <shadi> zakim, aaaa is Liz
14:02:15 <Zakim> +Liz; got it
14:02:38 <richard> zakim, ipcaller is richard
14:02:38 <Zakim> +richard; got it
14:02:45 <Zakim> + +1.517.432.aabb
14:02:47 <Zakim> -shadi
14:03:12 <shadi> zakim, who is on the phone?
14:03:12 <Zakim> On the phone I see Liz, ericvelleman, richard, +1.517.432.aabb
14:03:41 <Sarah_Swierenga> aabb is sarah swierenga
14:03:59 <Zakim> +Shadi
14:04:16 <Zakim> +[IPcaller]
14:04:19 <shadi> zakim, aabb is Sarah_Swierenga
14:04:19 <Zakim> +Sarah_Swierenga; got it
14:04:21 <Vivienne> zakim, IPcaller is me
14:04:22 <Zakim> +Vivienne; got it
14:04:24 <korn> korn has joined #eval
14:04:48 <shadi> scribe: Vivienne
14:04:51 <Zakim> +Peter_Korn
14:07:08 <Vivienne> EV: has not been able to get latest editor draft ready yet
14:07:18 <Vivienne> EV: disposition of comments
14:07:49 <Vivienne> EV: last week's use case study and also question in the disposition of comments on how methdology decide which states to consider
14:07:54 <Vivienne> zakim, mute me
14:07:54 <Zakim> Vivienne should now be muted
14:08:32 <Vivienne> EV: also missing consistency with the diagram - naming steps / stage
14:08:39 <MoeKraft> MoeKraft has joined #eval
14:09:03 <Vivienne> EV: status of use case revision - section 1(b)
14:09:17 <Zakim> +MoeKraft
14:09:55 <Vivienne> SA: have 4 use cases according to thread - discussion has stopped - what is required - should it go out to review again?
14:10:35 <Vivienne> EV: seems to have skipped over this section and its comments
14:11:00 <Vivienne> EV: also detailed overview of reports which is to be replaced with the use cases
14:11:21 <Vivienne> SA: tie in with remaining sections is important
14:11:57 <Vivienne> SA: primary thing to address is how this all relates throughout the document - that isn't very well worked out yet
14:12:00 <ericvelleman> q?
14:12:09 <Vivienne> SA: in the end in the report, there is no tie-in with the other sections
14:12:25 <Vivienne> EV: shall we work offline and get a proposal to the task force
14:12:30 <Vivienne> SA: yes, that's the suggestion
14:13:08 <Vivienne> EV: zakim, take up agendum 3
14:13:18 <Vivienne> zakim, take up agendum 3
14:13:18 <Zakim> I see nothing on the agenda
14:13:26 <ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130611#specialcases
14:13:32 <Vivienne> EV: types of websites
14:14:14 <Vivienne> EV: 'common stages' of the websites - 3 times
14:14:50 <Vivienne> EV: generated states of the web applications can really differ - what does this mean - what more significant place can we give it?
14:15:02 <Vivienne> Shadi:  did you want to take over scribing?
14:15:45 <Vivienne> EV: easiest way would be to add it to one of the last sections - 'considerations for particular situations' at the end of the document
14:16:19 <Vivienne> EV: this is right at the end - Step 5.d
14:16:32 <ericvelleman> q?
14:16:35 <Vivienne> q+
14:16:36 <korn> q+
14:16:41 <Vivienne> ack me
14:17:17 <Vivienne> EV: it comes after 5.d
14:17:25 <Vivienne> zakim, mute me
14:17:25 <Zakim> Vivienne should now be muted
14:17:37 <Vivienne> SA: what are the comments that we're trying to address?
14:18:13 <Vivienne> EV: the comment is that the methodology doesn't help you decide which state to consider
14:18:59 <Vivienne> SA: talking more about the sampling aspect and how pages/applications and states should be addressed
14:19:33 <Vivienne> SA: we're talking about the states of pages and how to select them
14:20:07 <Vivienne> EV: 2c identify the variety of web pages, then 3b include exemplar as identified in 2c
14:20:10 <Vivienne> q+
14:20:10 <richard> q+
14:22:27 <Vivienne> PK: we're really talking about having a web application that without changing the url can do different things - how do we track that and where in the steps do we do that.  Is it more important for a longer discourse about the use of testing software and how you record the steps you took in the software to get to various places.  It makes me start to think that stages is a better term than steps
14:22:27 <Vivienne> - that you took through the ap to record various paths - may be a new body of text
14:22:32 <korn> q-
14:22:41 <Vivienne> zakim, me
14:22:41 <Zakim> I don't understand 'me', Vivienne
14:22:46 <Vivienne> zakim, ack me
14:22:46 <Zakim> unmuting Vivienne
14:22:47 <Zakim> I see richard, Shadi on the speaker queue
14:23:43 <Vivienne> zakim, mute me
14:23:43 <Zakim> Vivienne should now be muted
14:24:03 <Vivienne> VC: we could also put it in the complete processes as each step needs to be recorded
14:24:30 <Vivienne> EV: You could get that when the user's profile is known and they get dynamically generated pages according to that preference
14:24:52 <Vivienne> Richard: government sites offer different processes according to your needs - you get a different set of pages
14:26:20 <Vivienne> Richard: with a large set of pages you could then sample - including a random sample.  You can apply the same idea to these multi-state sets of pages - we need to know the range of profiles and then select a random sample.  We'd need to talk to the web developer/owner and ask about those profiles - which are your priority targets etc.  We need to talk to the owner and then once you've got the
14:26:21 <Vivienne> range you can use your standard random sampling technicque.
14:26:41 <Vivienne> EV: then you do it in the scoping of the website and testing - getting the range of profiles
14:26:57 <Vivienne> Richard: it would be one of the loop-backs that you discover when scoping the analysis
14:27:16 <Vivienne> Richard: you'd identify that they are using profiles and you would contact the web owner to clarify the list of profiles
14:27:24 <ericvelleman> q?
14:30:04 <Vivienne> SA: recall that we had a section on web applications that we talked about the different states of an application and when it is considered to be a web page.  That might have got lost in the beginning.  It was also focused on types of websites that are called applications and doesn't really cover the different deliveries of content based on who you are.  Do we need to spell that out within the
14:30:04 <Vivienne> sections themselves.  "Also make sure that you identify how the web pages are generated and try to find with the help of the developer the range of different pages that can be generated, acknowledging that this might not be exhaustive".  You can see how the web pages are being generated, being put together." in some cases it might be part of a complete process or part of an individual state.
14:30:22 <ericvelleman> q?
14:30:26 <Vivienne> SA: we have the ideas, but it's getting lost because it's in the very beginning because we don't explain it too well how to do it
14:31:09 <Vivienne> EV: yes, that sounds good - it has be an integral part of what we're doing
14:31:26 <Vivienne> Richard: it's actually covered in the very first part, agree with Shadi
14:32:06 <Vivienne> EV: propose that we start with what's already there and which sections should have some information added.  Shadi can we prepare this together?
14:32:26 <Vivienne> SA: quite a bit of editing to do throughout - can work on that
14:32:58 <Vivienne> EV: maybe not in the next edited draft - we need to see what has changed to get it into the document
14:33:53 <ericvelleman> http://www.w3.org/WAI/ER/conformance/ED-methodology-20130611#procedure
14:33:55 <Vivienne> EV: next agenda point - section performance evaluation procedure - the diagram
14:34:30 <Zakim> +Tim_Boland
14:34:51 <shadi> shadi has joined #eval
14:35:30 <korn> q+
14:35:34 <Vivienne> EV: In thei intorduction to the conformance evaluation procedure - intro. top part says 'stages' and the lower part says 'steps'.  Should 1a,b,c, be steps or stages?
14:35:37 <ericvelleman> q?
14:35:46 <shadi> q-
14:35:49 <shadi> q- rich
14:35:56 <richard> q-
14:36:05 <Zakim> +[IPcaller]
14:36:12 <shadi> zakim, ipcaller is me
14:36:12 <Zakim> +shadi; got it
14:36:16 <Zakim> -Shadi
14:36:44 <korn> q-
14:36:50 <Vivienne> PK: like the idea of renaming the stages - wonder about reserving the idea of 'step' for 'steps through a web application' but that might be going a bit far.  The stages are significantly different from each other and I'm happy calling them stages.
14:37:02 <Sarah_Swierenga> q+
14:37:10 <Vivienne> EV: in the diagram - step 1 - change to stage 1,2,3?
14:37:14 <Vivienne> PK: yes
14:37:19 <richard> q+
14:37:55 <shadi> +1 to Sarah's comment
14:37:59 <Vivienne> SS: like the way that it is now in terms of steps - clearer from the unfamiliar user's perspection - the first step is this.  Stages implies they are independent.  That way the numbering can stay with step 1,2
14:37:59 <shadi> ack sarah
14:38:08 <Vivienne> Richard: keep it as steps - it is a user process
14:38:12 <shadi> ack r
14:38:20 <richard> steps +1
14:38:29 <Liz>  steps +1
14:38:31 <MoeKraft> +1 steps
14:38:41 <Tim> Tim has joined #eval
14:38:43 <Vivienne> I'm okay with steps, but I'm not concerned too much
14:39:08 <Vivienne> EV: in the survey I'll put this in to see if everyone agrees
14:39:30 <Vivienne> EV: you saw a difference in stage and steps, and you wanted to reserve steps?
14:40:28 <shadi> q+
14:41:14 <Vivienne> PK: use steps through an application - record the 5 steps I took to get to this screen where I had a problem etc.  When I think about the amount of work in these 5 steps, I like to call them stages.  But whether it is stages or steps, it is more a question of having a variety of nouns for the different things to stop it becoming confusing.  I personally like stage better for this, but it is not
14:41:14 <Vivienne> a significant issue.  More significant that we use a different word for different things.
14:42:16 <Vivienne> SA: the issue is that we're trying to fix is that we're using the word steps too much.  Keep them in the title as that's easy to communicate.  We can say using the technique 4.1... and not use the word 'step' within the context of the document itself.
14:42:56 <Vivienne> EV: we agreed that we can keep steps as it keeps it clear and understandable.  In this text we use 'stage' and 'steps' - should we drop the word 'stage'?
14:43:00 <shadi> ack me
14:43:28 <Vivienne> EV: if you go through a web application you follow a route to get to a point or screen - what do you call this if we already use 'steps' for this, we cannot use 'steps' for the things you do to get through a web application.
14:43:33 <Vivienne> What about path?
14:43:55 <shadi> +1 to path (+ parameters?)
14:44:18 <Vivienne> You can have the points on a path
14:44:58 <Vivienne> SA: for the sections of this document - we should keep it to steps.  Do we use 'steps' of the document and also 'steps' of the application where it is confusing?
14:45:11 <Liz> q+
14:45:12 <Vivienne> SA: otherwise it might be okay to say the 'steps that you took to get to this page'
14:45:33 <Vivienne> SA: depends on how we write it and don't use it in an ambiguous way
14:46:18 <shadi> +1 to Liz
14:46:27 <Vivienne> Liz: I had the feeling that step is an action - it is what you should do and stage is an environment.  In the diagram you are talking about the proceduring - they are action, exploring.  After you've done that you set the stage for the next step.  They are different, not synonyms
14:47:18 <Vivienne> EV: it is important to keep this in the back of our minds, have to do more to describe the ways you found the pages - the way that you go to a destination - depends on the decisions you make.  Maybe the path, or the stpes on that path
14:47:32 <Vivienne> EV: if they see the word step -do they get confused
14:47:46 <Vivienne> EV: we can just make sure we describe it clearly
14:47:52 <korn> q+
14:47:59 <shadi> q- liz
14:48:51 <korn> q-
14:48:56 <Vivienne> PK: it's a small issue compared to some of those we still have to deal with.  We haven't figured out what a good statistical sampling method is and how you ensure you have sampled enough and how you have described the stages through a web app.  This is small compared to those bigger issues
14:48:59 <Vivienne> EV: agree
14:49:37 <Vivienne> EV: regarding statistical sampling we've got some things from Giorgio that I'm putting into the dissolution of comments
14:50:13 <Vivienne> EV: we haven't described the states far enough - even in the reporting section.  It currently looks like you don't have to report how you got somewhere.  We need to add more sections to make this clearer.
14:51:20 <Vivienne> EV: other issues.  Anyone have anything to add or discuss?
14:51:32 <Vivienne> SA: yes, can we pick up on the statistical sampling?
14:52:23 <Vivienne> EV: we have added comments from Giorgio and have tried to add it into the comments and we still have to decide on
14:53:29 <Vivienne> SA: I think coming up with a number of statistical will be close to impossible - depends on type of website etc and lots of parameters. Depends on how many authors, tools used etc.  If we use a different approach - Detlev commented on this.  Relates to the purpose of the evaluation.
14:53:45 <korn> q+
14:54:06 <Vivienne> SA: after a couple of web pages you usually know if it's compliant or not.  If the first few are already so few of errors it might already answer the question the evaluation is trying to address.
14:54:20 <ericvelleman> q?
14:55:29 <Vivienne> SA: but if we say that there is a website that we don't know how it is going to fare and you're selecting and evaluating pages and continuing to find errors.  How repeated are those errors - you'd expect the number of types of errors to level off sometime.  If you go to someone and they ask how much it will cost, you need to be able to say how many pages.  You should be able to come up with a
14:55:29 <Vivienne> number of pages that should give you a reliable indication of the status of the website.
14:56:18 <Vivienne> SA: the process where you have a selection of validated pages that is a cross-section with randomly generated pages.  The evaluator will be making those decisions - they know when they have seen enough.  It needs to be clear in the reporting.
14:57:24 <Vivienne> PK: purpose if very important, but we don't have different types of reports in our document based on the different purposes.  If you're going to report on the basis of imperfection it's easy to stop on the first imperfection.  If you're trying to get an accurate overall sense you need to get a better sense of the homgeneity of heterogeneity of the site.
14:58:47 <Vivienne> PK: sometimes the pages look visually similar, you notice that there are differences that aren't apparent visually.  There is something to be said for some analysis of the heterogeneity of your sample.  If in a random sample you see that the pages are coded in the same way, you have confidence to say that the pages are similar.  If there is more heterogeneity, you need to look further to see how
14:58:47 <Vivienne> much deviation there is the sample types.
14:59:05 <Vivienne> PK: Statistically this may be very important
14:59:11 <korn> q-
15:00:00 <Vivienne> SA: Peter has a good point.  Calculating the deviation may be important, but the difficult thing is getting the parameters.  What constitutes hetergeneity?
15:00:09 <Vivienne> q+
15:00:23 <Vivienne> EV: this is probably the most important question about the sampling
15:00:48 <korn> q+
15:01:03 <Vivienne> EV: one of the questions could be the homogeneity or hetergeneity of the website.  The other is the purpose of the evaluation - acceptance, compliance.  Lots of parameters to consider - perhaps we need to make a list.
15:01:22 <Vivienne> zakim, ack me
15:01:22 <Zakim> unmuting Vivienne
15:01:22 <Zakim> I see korn on the speaker queue
15:02:08 <Sarah_Swierenga> have a good week. i need to head to another meeting.....
15:02:14 <Zakim> -Sarah_Swierenga
15:02:19 <Zakim> -Tim_Boland
15:03:48 <Vivienne> PK: also points out the important of the what we're calling the report.  It is critical we review the use of the word 'conformance'.  Are you just checking the pulse to see if there is anything major in the acceptance test you don't know if it conforms to WCAG.  We need to be careful in what we call the output of this - don't think that 'conformance' is the right word.
15:04:16 <Vivienne> EV: these parameters are important to get in a list and see what we have to work on and what is important or not.
15:04:30 <Vivienne> EV: let's start this on the list this week and get some short explanations there
15:04:58 <Vivienne> Ev: hope to have editors' draft and survey by the weekend if possible.  That's the homework for next week.
15:05:05 <Vivienne> ack me
15:05:05 <Liz> bye
15:05:06 <Zakim> -Peter_Korn
15:05:10 <korn> korn has left #eval
15:05:14 <richard> bye
15:05:16 <ericvelleman> ericvelleman has left #eval
15:05:20 <shadi> trackbot, end meeting
15:05:20 <trackbot> Zakim, list attendees
15:05:20 <Zakim> As of this point the attendees have been +1.301.975.aaaa, ericvelleman, shadi, Liz, richard, +1.517.432.aabb, Sarah_Swierenga, Vivienne, Peter_Korn, MoeKraft, Tim_Boland
15:05:23 <Zakim> -MoeKraft
15:05:23 <Zakim> -shadi
15:05:24 <Zakim> -ericvelleman
15:05:24 <Zakim> -Vivienne
15:05:28 <trackbot> RRSAgent, please draft minutes
15:05:28 <RRSAgent> I have made the request to generate http://www.w3.org/2013/07/11-eval-minutes.html trackbot
15:05:28 <Zakim> -Liz
15:05:29 <trackbot> RRSAgent, bye
15:05:29 <RRSAgent> I see no action items
15:05:30 <Zakim> -richard
15:05:30 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
15:05:30 <Zakim> Attendees were +1.301.975.aaaa, ericvelleman, shadi, Liz, richard, +1.517.432.aabb, Sarah_Swierenga, Vivienne, Peter_Korn, MoeKraft, Tim_Boland