13:54:12 <RRSAgent> RRSAgent has joined #eval
13:54:12 <RRSAgent> logging to http://www.w3.org/2012/10/04-eval-irc
13:56:10 <MartijnHoutepen> Zakim, this will be eval
13:56:10 <Zakim> ok, MartijnHoutepen, I see WAI_ERTWG(Eval TF)10:00AM already started
13:57:02 <vivienne> ack me
13:57:28 <MartijnHoutepen> zakim, who is on the phone?
13:57:28 <Zakim> On the phone I see ??P3, +1.978.443.aaaa
13:57:51 <Kathy> zakim, aaaa is me
13:57:51 <Zakim> +Kathy; got it
13:57:58 <vivienne> zakim, ??P3 is me
13:57:58 <Zakim> +vivienne; got it
13:58:19 <Liz> Liz has joined #eval
13:59:03 <Zakim> +Liz
13:59:33 <Zakim> +MartijnHoutepen
14:00:07 <MartijnHoutepen> zakim, who is on the phone?
14:00:07 <Zakim> On the phone I see vivienne, Kathy, Liz, MartijnHoutepen
14:00:28 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
14:01:16 <ericvelleman> ericvelleman has joined #eval
14:01:19 <vivienne> zakim, mute me
14:01:19 <Zakim> vivienne should now be muted
14:01:33 <MartijnHoutepen> Zakim, mute me
14:01:34 <shadi> regrets: Tim, Alistair, Moe
14:01:40 <Zakim> MartijnHoutepen should now be muted
14:02:07 <shadi> chair: Eric
14:02:26 <Zakim> +Sarah
14:02:28 <MartijnHoutepen> ack me
14:03:03 <Zakim> +Eric_Velleman
14:03:24 <MartijnHoutepen> scribe: Sarah
14:05:21 <MartijnHoutepen> zakim, mute me
14:05:21 <Zakim> MartijnHoutepen should now be muted
14:05:23 <Kathy> zakim, mute me
14:05:23 <Zakim> Kathy should now be muted
14:05:53 <MartijnHoutepen> topic: comments until now
14:06:11 <korn> korn has joined #eval
14:06:33 <Sarah_Swierenga> scribe: sarah
14:06:49 <MartijnHoutepen> EV: We will collect all the received comments into a disposition of comments and publish this after the comments close
14:07:34 <MartijnHoutepen> ack me
14:07:49 <Zakim> +[Oracle]
14:07:51 <korn> Zakim, Oracle has Peter_Korn
14:07:51 <Zakim> +Peter_Korn; got it
14:07:59 <MartijnHoutepen> zakim, mute me
14:07:59 <Zakim> MartijnHoutepen should now be muted
14:08:07 <Sarah_Swierenga> item #3 random sampling survey
14:08:44 <MartijnHoutepen> http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Oct/0017.html
14:09:07 <Kathy> yes
14:09:09 <vivienne> q+
14:09:10 <Sarah_Swierenga> yes
14:09:15 <Liz> yes
14:09:29 <Zakim> +Katie_Haritos-Shea
14:09:36 <korn> q+
14:09:40 <vivienne> ack me
14:10:29 <Kathy> q+
14:10:57 <Sarah_Swierenga> vivienne: evaluation commissioner - maybe need question; to what extend do the views of the commissioner change the sample selection? 
14:11:10 <Ryladog> Ryladog has joined #eval
14:11:12 <vivienne> zakim, mute me
14:11:12 <Zakim> vivienne should now be muted
14:11:16 <korn> q?
14:11:16 <Ryladog> q+
14:11:34 <MartijnHoutepen> ack me
14:11:45 <MartijnHoutepen> q- korn
14:12:19 <mike_elledge> mike_elledge has joined #eval
14:13:00 <Sarah_Swierenga> peter: concern with survey is that is focused on external evaluation and sampling. If this the only view we have then we'll miss other users and use cases, e.g., web apps, templates, boundary condition testing.
14:13:12 <korn> q-
14:13:24 <Zakim> +Mike
14:13:37 <Sarah_Swierenga> martijn: add questions if you like
14:14:02 <ericvelleman> ericvelleman has joined #eval
14:14:08 <Sarah_Swierenga> peter: the questionnaire would have to expand significantly, so we might want to break them up into two questionnaires
14:14:10 <Zakim> -Eric_Velleman
14:14:22 <vivienne> q+
14:14:35 <ericvelleman> sorry, my phone was also on internet, so I will have to dial in again
14:15:22 <ericvelleman> ericvelleman has left #eval
14:15:34 <Sarah_Swierenga> peter: different survey for structured reviews for websites that are too  big, or have too many permutations
14:15:38 <ericvelleman> ericvelleman has joined #eval
14:16:15 <Sarah_Swierenga> martijn: maybe another questionnaire for in-house testing
14:16:36 <Zakim> +Eric_Velleman
14:17:07 <Sarah_Swierenga> peter: craft a placeholder for random sampling for other types of website
14:17:11 <shadi> [[I wonder if we need two questionnaires that follow each other to dig deeper into particular areas (such as web applications) rather than splitting by in-house/external modalities?]]
14:17:34 <ericvelleman> q?
14:17:56 <Kathy> ack me
14:18:11 <MartijnHoutepen> zakim, mute me
14:18:11 <Zakim> MartijnHoutepen should now be muted
14:18:40 <Sarah_Swierenga> Kathy: need more for web applications. right now we have #12 that can start the discussions. is that enough?
14:19:43 <Sarah_Swierenga> peter: how do you adjust your sampling approach? your testing approach? adjustment could be to remove Random from the title and add it to specific items.
14:21:08 <Sarah_Swierenga> kathy: #3 - this would change depending on what type of evaluation is done
14:21:16 <ericvelleman> q?
14:22:27 <vivienne> * vivienne agrees that we should change the title and make the survey about sampling, not just random
14:22:35 <vivienne> q-
14:23:28 <Sarah_Swierenga> kathy: mobile apps, websites, web apps, etc.
14:24:10 <Sarah_Swierenga> kathy: methodology is very focused on websites; internal vs external views of applications should be addressed
14:24:39 <Kathy> zakim, mute me
14:24:39 <Zakim> Kathy should now be muted
14:24:42 <ericvelleman> q?
14:24:45 <Sarah_Swierenga> ev: need to account for these ideas
14:26:13 <korn> q+
14:26:14 <Sarah_Swierenga> katie: agrees with peter and kathy re: internal vs external views. when we figure out the universe of this document, we need to consider historic components of the project
14:26:30 <Kathy> q+
14:26:44 <Sarah_Swierenga> shadi: mobile has to be in the scope
14:26:45 <MartijnHoutepen> q- Ryladog
14:26:46 <Sarah_Swierenga> +1
14:26:49 <ericvelleman> q?
14:26:50 <vivienne> yes, mobile has to be there
14:26:57 <Ryladog> +1
14:27:17 <Ryladog> +1 mobile
14:27:47 <shadi> [[not advocating to remove mobile but wondering about terminology difference between "mobile app" versus "mobile website"]]
14:28:33 <Sarah_Swierenga> peter: include mobile as part of the universe. question about re-review: white box testing where the engineering group tells you we made lots of changes in these areas, so we focus on the new areas. how would this adjust the sampling?
14:28:42 <ericvelleman> q?
14:28:45 <Kathy> ach me
14:28:49 <MartijnHoutepen> q- korn
14:28:53 <MartijnHoutepen> ack kathy
14:29:30 <vivienne> q+
14:29:36 <Sarah_Swierenga> kathy: also missing is the role of automated testing. do automated tests change the sampling strategy?
14:29:57 <ericvelleman> q?
14:30:31 <vivienne> ack me
14:30:38 <vivienne> q+
14:30:40 <Sarah_Swierenga> ev: good idea for a new question
14:31:14 <Kathy> zakim, mute me
14:31:14 <Zakim> Kathy should now be muted
14:31:17 <shadi> +1
14:31:40 <MartijnHoutepen> +1
14:32:02 <Sarah_Swierenga> vivienne: likes adding question about automated testing. sometimes uses automated tools to help decide on pages to test for manual review.
14:32:03 <vivienne> zakim, mute me
14:32:03 <Zakim> vivienne should now be muted
14:32:04 <ericvelleman> q?
14:32:09 <Sarah_Swierenga> +1
14:32:11 <MartijnHoutepen> q- viv
14:33:06 <Sarah_Swierenga> ev: send proposed questions/revisions to the list
14:33:15 <vivienne> sure
14:34:42 <ericvelleman> q?
14:34:43 <korn> q+
14:34:52 <MartijnHoutepen> q- korn
14:35:50 <Sarah_Swierenga> peter: native apps shouldn't be within the scope. 
14:36:58 <Sarah_Swierenga> katie: agrees that native apps shouldn't be included, but if it opens in a browser it should be included.
14:38:12 <Sarah_Swierenga> peter: if it's not something that users can direct their browsers to view, then it shouldn't be included
14:38:36 <Kathy> q+
14:38:53 <Sarah_Swierenga> peter: e.g., iTunes is not within scope
14:38:53 <shadi> [[propose a question to compare *sampling* in mobile web apps vs in traditional web apps]]
14:40:06 <Sarah_Swierenga> katie: the excell version (plugin) that opens in the browser is covered by WCAG. so, bottom line is 'if it opens in a browser it is included.'
14:40:27 <ericvelleman> q?
14:40:27 <Kathy> ack me
14:41:45 <Kathy> zakim, mute me
14:41:46 <Zakim> Kathy should now be muted
14:41:58 <Sarah_Swierenga> kathy: add on to Katie's comment, there are instances where the majority of the application runs in a browser but calls out a native app, then that should be included. 
14:42:55 <Sarah_Swierenga> ev: add your questions to the questionnaire list by next Tuesday
14:43:52 <Sarah_Swierenga> ev: next agenda item: goodness criteria - propose discussing this on the next call
14:44:18 <Sarah_Swierenga> ev: agenda item 5: approaches for tolerance
14:45:55 <ericvelleman> http://www.w3.org/WAI/ER/2011/eval/track/issues/1
14:47:42 <Sarah_Swierenga> peter: aggregating individual results into a conformance statement - my fundamental concern is how to convey the results in a meaningful fashion; red, yellow, green ratings for the outputs are minimally helpful. 
14:47:55 <vivienne> q+
14:48:22 <ramoncorominas> ramoncorominas has joined #eval
14:48:35 <Sarah_Swierenga> peter: confidence level is key
14:48:55 <vivienne> ack me
14:50:24 <Sarah_Swierenga> vivienne: does failing one item means failing overall - we have decided yes, but we may want to consider a 'not quite there' rating, a 'conditional pass'
14:50:49 <Kathy> q+
14:50:49 <korn> q+
14:50:52 <vivienne> zakim, mute me
14:50:52 <Zakim> vivienne should now be muted
14:50:53 <Kathy> ack me
14:51:04 <Sarah_Swierenga> vivienne: this approach encourages developers to get those items fixed to get a complete pass
14:52:39 <Sarah_Swierenga> katie: uses a similar approach. uses high, medium, low severity for the items; also has rating for impact for persons with disabilities, e.g., descriptive text on a spacer image vs an unlabeled link that would impact PWD more. 
14:53:04 <korn> q?
14:53:41 <shadi> [[I see three parts of the evaluation "output": (1) conformance to WCAG - yes/no; (2) some type of score - indicative to help "motivate" developers and decision makers; (3) report - to show the types of issues to explain the "severity" and guide developers on how to fix issues]]
14:54:02 <Sarah_Swierenga> ev: two discussions on tolerance - 1) confidence level or 2) severity/impact for PWD
14:55:25 <Kathy> q+
14:56:04 <Ryladog> q+
14:56:20 <korn> q-
14:56:26 <Sarah_Swierenga> peter: these two approaches are closely linked; confidence of what you are reporting is a given. but how do you convey 'good, but not perfect' or 'not horrible' vs sites that are mostly inaccessible, since this is very valuable info?
14:56:46 <Sarah_Swierenga> ev: we should continue this discussion on the list
14:57:31 <Sarah_Swierenga> ev: agenda item #6: graphic
14:59:33 <Sarah_Swierenga> kathy: sent a couple of graphics to the list. want feedback on 1) what do we see as the differences in interactions with the different steps? and 2) how to convey this in the graphic?. e.g., the arrows don't overlap, but they should.
14:59:59 <Ryladog> Nice job Kathy
15:00:07 <Sarah_Swierenga> ev: graphics are colorful and clear, so let's get them finalized
15:00:29 <Sarah_Swierenga> kathy: working with WCAG group on the graphic, too.
15:00:45 <MartijnHoutepen> ack me
15:00:47 <Sarah_Swierenga> bye - have a good week
15:00:48 <Liz> bye
15:00:49 <vivienne> okay, bye all
15:00:51 <shadi> [[I really like the lify-cycle approach]]
15:00:52 <vivienne> -vivienne
15:00:55 <vivienne> vivienne has left #eval
15:00:55 <Kathy> bye
15:00:58 <Zakim> -Kathy
15:01:06 <Zakim> -Katie_Haritos-Shea
15:01:11 <Zakim> -Mike
15:01:13 <Zakim> -[Oracle]
15:01:15 <Zakim> -Sarah
15:01:17 <Zakim> -vivienne
15:01:18 <ericvelleman> Shadi, are you making the minutes?
15:01:34 <Zakim> -Eric_Velleman
15:01:35 <korn> korn has left #eval
15:01:35 <ericvelleman> ericvelleman has left #eval
15:01:35 <Zakim> -Liz
15:01:41 <MartijnHoutepen> ok thanks shadi
15:01:45 <MartijnHoutepen> bye
15:01:46 <shadi> trackbot, end meeting
15:01:46 <trackbot> Zakim, list attendees
15:01:49 <Zakim> As of this point the attendees have been +1.978.443.aaaa, Kathy, vivienne, Liz, MartijnHoutepen, Sarah, Eric_Velleman, Peter_Korn, Katie_Haritos-Shea, Mike
15:01:49 <MartijnHoutepen> MartijnHoutepen has left #eval
15:01:54 <trackbot> RRSAgent, please draft minutes
15:01:54 <RRSAgent> I have made the request to generate http://www.w3.org/2012/10/04-eval-minutes.html trackbot
15:01:55 <trackbot> RRSAgent, bye
15:02:12 <Zakim> -MartijnHoutepen
15:02:16 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
15:02:19 <shadi> rrsagent, make logs world
15:02:21 <Zakim> Attendees were +1.978.443.aaaa, Kathy, vivienne, Liz, MartijnHoutepen, Sarah, Eric_Velleman, Peter_Korn, Katie_Haritos-Shea, Mike
15:02:27 <shadi> rrsagent, make mnutes
15:02:27 <RRSAgent> I'm logging. I don't understand 'make mnutes', shadi.  Try /msg RRSAgent help
15:02:32 <shadi> rrsagent, make minutes
15:02:32 <RRSAgent> I have made the request to generate http://www.w3.org/2012/10/04-eval-minutes.html shadi
15:02:42 <shadi> rrsagent, bye
15:02:42 <RRSAgent> I see no action items