IRC log of eval on 2012-10-04

Timestamps are in UTC.

13:54:12 [RRSAgent]

RRSAgent has joined #eval

13:54:12 [RRSAgent]

logging to http://www.w3.org/2012/10/04-eval-irc

13:56:10 [MartijnHoutepen]

Zakim, this will be eval

13:56:10 [Zakim]

ok, MartijnHoutepen, I see WAI_ERTWG(Eval TF)10:00AM already started

13:57:02 [vivienne]

ack me

13:57:28 [MartijnHoutepen]

zakim, who is on the phone?

13:57:28 [Zakim]

On the phone I see ??P3, +1.978.443.aaaa

13:57:51 [Kathy]

zakim, aaaa is me

13:57:51 [Zakim]

+Kathy; got it

13:57:58 [vivienne]

zakim, ??P3 is me

13:57:58 [Zakim]

+vivienne; got it

13:58:19 [Liz]

Liz has joined #eval

13:59:03 [Zakim]

+Liz

13:59:33 [Zakim]

+MartijnHoutepen

14:00:07 [MartijnHoutepen]

zakim, who is on the phone?

14:00:07 [Zakim]

On the phone I see vivienne, Kathy, Liz, MartijnHoutepen

14:00:28 [Sarah_Swierenga]

Sarah_Swierenga has joined #eval

14:01:16 [ericvelleman]

ericvelleman has joined #eval

14:01:19 [vivienne]

zakim, mute me

14:01:19 [Zakim]

vivienne should now be muted

14:01:33 [MartijnHoutepen]

Zakim, mute me

14:01:34 [shadi]

regrets: Tim, Alistair, Moe

14:01:40 [Zakim]

MartijnHoutepen should now be muted

14:02:07 [shadi]

chair: Eric

14:02:26 [Zakim]

+Sarah

14:02:28 [MartijnHoutepen]

ack me

14:03:03 [Zakim]

+Eric_Velleman

14:03:24 [MartijnHoutepen]

scribe: Sarah

14:05:21 [MartijnHoutepen]

zakim, mute me

14:05:21 [Zakim]

MartijnHoutepen should now be muted

14:05:23 [Kathy]

zakim, mute me

14:05:23 [Zakim]

Kathy should now be muted

14:05:53 [MartijnHoutepen]

topic: comments until now

14:06:11 [korn]

korn has joined #eval

14:06:33 [Sarah_Swierenga]

scribe: sarah

14:06:49 [MartijnHoutepen]

EV: We will collect all the received comments into a disposition of comments and publish this after the comments close

14:07:34 [MartijnHoutepen]

ack me

14:07:49 [Zakim]

+[Oracle]

14:07:51 [korn]

Zakim, Oracle has Peter_Korn

14:07:51 [Zakim]

+Peter_Korn; got it

14:07:59 [MartijnHoutepen]

zakim, mute me

14:07:59 [Zakim]

MartijnHoutepen should now be muted

14:08:07 [Sarah_Swierenga]

item #3 random sampling survey

14:08:44 [MartijnHoutepen]

http://lists.w3.org/Archives/Public/public-wai-evaltf/2012Oct/0017.html

14:09:07 [Kathy]

yes

14:09:09 [vivienne]

q+

14:09:10 [Sarah_Swierenga]

yes

14:09:15 [Liz]

yes

14:09:29 [Zakim]

+Katie_Haritos-Shea

14:09:36 [korn]

q+

14:09:40 [vivienne]

ack me

14:10:29 [Kathy]

q+

14:10:57 [Sarah_Swierenga]

vivienne: evaluation commissioner - maybe need question; to what extend do the views of the commissioner change the sample selection?

14:11:10 [Ryladog]

Ryladog has joined #eval

14:11:12 [vivienne]

zakim, mute me

14:11:12 [Zakim]

vivienne should now be muted

14:11:16 [korn]

q?

14:11:16 [Ryladog]

q+

14:11:34 [MartijnHoutepen]

ack me

14:11:45 [MartijnHoutepen]

q- korn

14:12:19 [mike_elledge]

mike_elledge has joined #eval

14:13:00 [Sarah_Swierenga]

peter: concern with survey is that is focused on external evaluation and sampling. If this the only view we have then we'll miss other users and use cases, e.g., web apps, templates, boundary condition testing.

14:13:12 [korn]

q-

14:13:24 [Zakim]

+Mike

14:13:37 [Sarah_Swierenga]

martijn: add questions if you like

14:14:02 [ericvelleman]

ericvelleman has joined #eval

14:14:08 [Sarah_Swierenga]

peter: the questionnaire would have to expand significantly, so we might want to break them up into two questionnaires

14:14:10 [Zakim]

-Eric_Velleman

14:14:22 [vivienne]

q+

14:14:35 [ericvelleman]

sorry, my phone was also on internet, so I will have to dial in again

14:15:22 [ericvelleman]

ericvelleman has left #eval

14:15:34 [Sarah_Swierenga]

peter: different survey for structured reviews for websites that are too big, or have too many permutations

14:15:38 [ericvelleman]

ericvelleman has joined #eval

14:16:15 [Sarah_Swierenga]

martijn: maybe another questionnaire for in-house testing

14:16:36 [Zakim]

+Eric_Velleman

14:17:07 [Sarah_Swierenga]

peter: craft a placeholder for random sampling for other types of website

14:17:11 [shadi]

[[I wonder if we need two questionnaires that follow each other to dig deeper into particular areas (such as web applications) rather than splitting by in-house/external modalities?]]

14:17:34 [ericvelleman]

q?

14:17:56 [Kathy]

ack me

14:18:11 [MartijnHoutepen]

zakim, mute me

14:18:11 [Zakim]

MartijnHoutepen should now be muted

14:18:40 [Sarah_Swierenga]

Kathy: need more for web applications. right now we have #12 that can start the discussions. is that enough?

14:19:43 [Sarah_Swierenga]

peter: how do you adjust your sampling approach? your testing approach? adjustment could be to remove Random from the title and add it to specific items.

14:21:08 [Sarah_Swierenga]

kathy: #3 - this would change depending on what type of evaluation is done

14:21:16 [ericvelleman]

q?

14:22:27 [vivienne]

* vivienne agrees that we should change the title and make the survey about sampling, not just random

14:22:35 [vivienne]

q-

14:23:28 [Sarah_Swierenga]

kathy: mobile apps, websites, web apps, etc.

14:24:10 [Sarah_Swierenga]

kathy: methodology is very focused on websites; internal vs external views of applications should be addressed

14:24:39 [Kathy]

zakim, mute me

14:24:39 [Zakim]

Kathy should now be muted

14:24:42 [ericvelleman]

q?

14:24:45 [Sarah_Swierenga]

ev: need to account for these ideas

14:26:13 [korn]

q+

14:26:14 [Sarah_Swierenga]

katie: agrees with peter and kathy re: internal vs external views. when we figure out the universe of this document, we need to consider historic components of the project

14:26:30 [Kathy]

q+

14:26:44 [Sarah_Swierenga]

shadi: mobile has to be in the scope

14:26:45 [MartijnHoutepen]

q- Ryladog

14:26:46 [Sarah_Swierenga]

+1

14:26:49 [ericvelleman]

q?

14:26:50 [vivienne]

yes, mobile has to be there

14:26:57 [Ryladog]

+1

14:27:17 [Ryladog]

+1 mobile

14:27:47 [shadi]

[[not advocating to remove mobile but wondering about terminology difference between "mobile app" versus "mobile website"]]

14:28:33 [Sarah_Swierenga]

peter: include mobile as part of the universe. question about re-review: white box testing where the engineering group tells you we made lots of changes in these areas, so we focus on the new areas. how would this adjust the sampling?

14:28:42 [ericvelleman]

q?

14:28:45 [Kathy]

ach me

14:28:49 [MartijnHoutepen]

q- korn

14:28:53 [MartijnHoutepen]

ack kathy

14:29:30 [vivienne]

q+

14:29:36 [Sarah_Swierenga]

kathy: also missing is the role of automated testing. do automated tests change the sampling strategy?

14:29:57 [ericvelleman]

q?

14:30:31 [vivienne]

ack me

14:30:38 [vivienne]

q+

14:30:40 [Sarah_Swierenga]

ev: good idea for a new question

14:31:14 [Kathy]

zakim, mute me

14:31:14 [Zakim]

Kathy should now be muted

14:31:17 [shadi]

+1

14:31:40 [MartijnHoutepen]

+1

14:32:02 [Sarah_Swierenga]

vivienne: likes adding question about automated testing. sometimes uses automated tools to help decide on pages to test for manual review.

14:32:03 [vivienne]

zakim, mute me

14:32:03 [Zakim]

vivienne should now be muted

14:32:04 [ericvelleman]

q?

14:32:09 [Sarah_Swierenga]

+1

14:32:11 [MartijnHoutepen]

q- viv

14:33:06 [Sarah_Swierenga]

ev: send proposed questions/revisions to the list

14:33:15 [vivienne]

sure

14:34:42 [ericvelleman]

q?

14:34:43 [korn]

q+

14:34:52 [MartijnHoutepen]

q- korn

14:35:50 [Sarah_Swierenga]

peter: native apps shouldn't be within the scope.

14:36:58 [Sarah_Swierenga]

katie: agrees that native apps shouldn't be included, but if it opens in a browser it should be included.

14:38:12 [Sarah_Swierenga]

peter: if it's not something that users can direct their browsers to view, then it shouldn't be included

14:38:36 [Kathy]

q+

14:38:53 [Sarah_Swierenga]

peter: e.g., iTunes is not within scope

14:38:53 [shadi]

[[propose a question to compare *sampling* in mobile web apps vs in traditional web apps]]

14:40:06 [Sarah_Swierenga]

katie: the excell version (plugin) that opens in the browser is covered by WCAG. so, bottom line is 'if it opens in a browser it is included.'

14:40:27 [ericvelleman]

q?

14:40:27 [Kathy]

ack me

14:41:45 [Kathy]

zakim, mute me

14:41:46 [Zakim]

Kathy should now be muted

14:41:58 [Sarah_Swierenga]

kathy: add on to Katie's comment, there are instances where the majority of the application runs in a browser but calls out a native app, then that should be included.

14:42:55 [Sarah_Swierenga]

ev: add your questions to the questionnaire list by next Tuesday

14:43:52 [Sarah_Swierenga]

ev: next agenda item: goodness criteria - propose discussing this on the next call

14:44:18 [Sarah_Swierenga]

ev: agenda item 5: approaches for tolerance

14:45:55 [ericvelleman]

http://www.w3.org/WAI/ER/2011/eval/track/issues/1

14:47:42 [Sarah_Swierenga]

peter: aggregating individual results into a conformance statement - my fundamental concern is how to convey the results in a meaningful fashion; red, yellow, green ratings for the outputs are minimally helpful.

14:47:55 [vivienne]

q+

14:48:22 [ramoncorominas]

ramoncorominas has joined #eval

14:48:35 [Sarah_Swierenga]

peter: confidence level is key

14:48:55 [vivienne]

ack me

14:50:24 [Sarah_Swierenga]

vivienne: does failing one item means failing overall - we have decided yes, but we may want to consider a 'not quite there' rating, a 'conditional pass'

14:50:49 [Kathy]

q+

14:50:49 [korn]

q+

14:50:52 [vivienne]

zakim, mute me

14:50:52 [Zakim]

vivienne should now be muted

14:50:53 [Kathy]

ack me

14:51:04 [Sarah_Swierenga]

vivienne: this approach encourages developers to get those items fixed to get a complete pass

14:52:39 [Sarah_Swierenga]

katie: uses a similar approach. uses high, medium, low severity for the items; also has rating for impact for persons with disabilities, e.g., descriptive text on a spacer image vs an unlabeled link that would impact PWD more.

14:53:04 [korn]

q?

14:53:41 [shadi]

[[I see three parts of the evaluation "output": (1) conformance to WCAG - yes/no; (2) some type of score - indicative to help "motivate" developers and decision makers; (3) report - to show the types of issues to explain the "severity" and guide developers on how to fix issues]]

14:54:02 [Sarah_Swierenga]

ev: two discussions on tolerance - 1) confidence level or 2) severity/impact for PWD

14:55:25 [Kathy]

q+

14:56:04 [Ryladog]

q+

14:56:20 [korn]

q-

14:56:26 [Sarah_Swierenga]

peter: these two approaches are closely linked; confidence of what you are reporting is a given. but how do you convey 'good, but not perfect' or 'not horrible' vs sites that are mostly inaccessible, since this is very valuable info?

14:56:46 [Sarah_Swierenga]

ev: we should continue this discussion on the list

14:57:31 [Sarah_Swierenga]

ev: agenda item #6: graphic

14:59:33 [Sarah_Swierenga]

kathy: sent a couple of graphics to the list. want feedback on 1) what do we see as the differences in interactions with the different steps? and 2) how to convey this in the graphic?. e.g., the arrows don't overlap, but they should.

14:59:59 [Ryladog]

Nice job Kathy

15:00:07 [Sarah_Swierenga]

ev: graphics are colorful and clear, so let's get them finalized

15:00:29 [Sarah_Swierenga]

kathy: working with WCAG group on the graphic, too.

15:00:45 [MartijnHoutepen]

ack me

15:00:47 [Sarah_Swierenga]

bye - have a good week

15:00:48 [Liz]

bye

15:00:49 [vivienne]

okay, bye all

15:00:51 [shadi]

[[I really like the lify-cycle approach]]

15:00:52 [vivienne]

-vivienne

15:00:55 [vivienne]

vivienne has left #eval

15:00:55 [Kathy]

bye

15:00:58 [Zakim]

-Kathy

15:01:06 [Zakim]

-Katie_Haritos-Shea

15:01:11 [Zakim]

-Mike

15:01:13 [Zakim]

-[Oracle]

15:01:15 [Zakim]

-Sarah

15:01:17 [Zakim]

-vivienne

15:01:18 [ericvelleman]

Shadi, are you making the minutes?

15:01:34 [Zakim]

-Eric_Velleman

15:01:35 [korn]

korn has left #eval

15:01:35 [ericvelleman]

ericvelleman has left #eval

15:01:35 [Zakim]

-Liz

15:01:41 [MartijnHoutepen]

ok thanks shadi

15:01:45 [MartijnHoutepen]

bye

15:01:46 [shadi]

trackbot, end meeting

15:01:46 [trackbot]

Zakim, list attendees

15:01:49 [Zakim]

As of this point the attendees have been +1.978.443.aaaa, Kathy, vivienne, Liz, MartijnHoutepen, Sarah, Eric_Velleman, Peter_Korn, Katie_Haritos-Shea, Mike

15:01:49 [MartijnHoutepen]

MartijnHoutepen has left #eval

15:01:54 [trackbot]

RRSAgent, please draft minutes

15:01:54 [RRSAgent]

I have made the request to generate http://www.w3.org/2012/10/04-eval-minutes.html trackbot

15:01:55 [trackbot]

RRSAgent, bye

15:02:12 [Zakim]

-MartijnHoutepen

15:02:16 [Zakim]

WAI_ERTWG(Eval TF)10:00AM has ended

15:02:19 [shadi]

rrsagent, make logs world

15:02:21 [Zakim]

Attendees were +1.978.443.aaaa, Kathy, vivienne, Liz, MartijnHoutepen, Sarah, Eric_Velleman, Peter_Korn, Katie_Haritos-Shea, Mike

15:02:27 [shadi]

rrsagent, make mnutes

15:02:27 [RRSAgent]

I'm logging. I don't understand 'make mnutes', shadi. Try /msg RRSAgent help

15:02:32 [shadi]

rrsagent, make minutes

15:02:32 [RRSAgent]

I have made the request to generate http://www.w3.org/2012/10/04-eval-minutes.html shadi

15:02:42 [shadi]

rrsagent, bye

15:02:42 [RRSAgent]

I see no action items