14:48:38 <RRSAgent> RRSAgent has joined #rd
14:48:38 <RRSAgent> logging to http://www.w3.org/2011/07/28-rd-irc
14:48:40 <trackbot> RRSAgent, make logs world
14:48:40 <Zakim> Zakim has joined #rd
14:48:42 <trackbot> Zakim, this will be 7394
14:48:42 <Zakim> ok, trackbot; I see WAI_RDWG()11:00AM scheduled to start in 12 minutes
14:48:43 <trackbot> Meeting: Research and Development Working Group Teleconference
14:48:43 <trackbot> Date: 28 July 2011
14:49:08 <sharper> Chair: Harper_Simon
14:49:16 <sharper> Agenda+ Welcome
14:49:16 <sharper> Agenda+ Logistics (Regrets, Agenda Requests, Comments)
14:49:16 <sharper> Agenda+ Round the table introductions - each participant introduces themselves briefly
14:49:16 <sharper> Agenda+ First Topic Selection - http://www.w3.org/2002/09/wbs/47076/TopicSelection01/results
14:49:17 <sharper> Agenda+ Any Other Business 
14:49:27 <sharper> zakim, save agenda
14:49:33 <Zakim> ok, sharper, the agenda has been written to http://www.w3.org/2011/07/28-rd-agenda.rdf
14:49:41 <sharper> rrsagent, make logs public
14:49:41 <sharper> rrsagent, draft minutes
14:49:41 <RRSAgent> I have made the request to generate http://www.w3.org/2011/07/28-rd-minutes.html sharper
14:50:02 <sharper> present+ Harper_Simon
14:50:12 <sharper> zakim, take up item 1
14:50:12 <Zakim> agendum 1. "Welcome" taken up [from sharper]
14:56:23 <Zakim> WAI_RDWG()11:00AM has now started
14:56:30 <Zakim> + +1.703.225.aaaa
14:59:53 <KarlGroves> KarlGroves has joined #rd
15:00:00 <markel> markel has joined #rd
15:00:04 <yeliz> yeliz has joined #rd
15:00:52 <Zakim> +??P58
15:00:56 <sharper> zakim, ??P58 is sharper
15:00:56 <Zakim> +sharper; got it
15:01:40 <Zakim> +??P61
15:03:39 <Zakim> +??P16
15:14:05 <Zakim> +??P25
15:14:12 <yeliz> zakim, ??P25 is yeliz
15:14:13 <Zakim> +yeliz; got it
15:14:36 <yeliz> zakim, mute yeliz
15:14:36 <Zakim> yeliz should now be muted
15:14:46 <yeliz> zakim, unmute yeliz
15:14:46 <Zakim> yeliz should no longer be muted
15:15:04 <shadi> zakim, call shadi-617
15:15:05 <Zakim> ok, shadi; the call is being made
15:15:05 <Zakim> +Shadi
15:15:08 <yeliz> zakim, mute yeliz
15:15:11 <Zakim> yeliz should now be muted
15:15:21 <shadi> zakim, who is on the phone?
15:15:21 <Zakim> On the phone I see +1.703.225.aaaa, sharper, ??P61, ??P16, yeliz (muted), Shadi
15:15:35 <yeliz> Agenda+ Welcome
15:15:35 <yeliz> Agenda+ Logistics (Regrets, Agenda Requests, Comments)
15:15:36 <yeliz> Agenda+ Round the table introductions - each participant introduces themselves briefly
15:16:07 <yeliz> SH is introducing himself, he works on web accessibility
15:16:36 <yeliz> Arun is introducing himself, they do technology for disabled people
15:17:03 <yeliz> Markel is introducing himself, he is from Manchester, he is interested in understanding behaviour of blind users
15:17:22 <yeliz> KarlGroves is from New York, he does training and evaluation
15:17:33 <yeliz> zakim, unmute yeliz
15:17:33 <Zakim> yeliz should no longer be muted
15:17:50 <KarlGroves> @yeliz  Actually I'm from Washington DC area, thx
15:17:52 <shadi> shadi has joined #rd
15:17:58 <yeliz> zakim, mute yeliz
15:17:58 <Zakim> yeliz should now be muted
15:18:11 <sharper> yeliz, introduces herself from METU, looks at blind users
15:18:20 <shadi> zakim, aaaa is Karl
15:18:20 <Zakim> +Karl; got it
15:18:22 <yeliz> zakim, mute me
15:18:22 <Zakim> yeliz was already muted, yeliz
15:18:42 <shadi> zakim, ? is Markel
15:18:42 <Zakim> sorry, shadi, I do not recognize a party named '?'
15:18:49 <yeliz> scribe, yeliz
15:18:59 <shadi> zakim, ??p61 Markel
15:18:59 <Zakim> I don't understand '??p61 Markel', shadi
15:19:07 <shadi> zakim, ??p16 Arun
15:19:07 <Zakim> I don't understand '??p16 Arun', shadi
15:19:16 <arun> arun has joined #rd
15:19:41 <yeliz> Agenda+ First Topic Selection - http://www.w3.org/2002/09/wbs/47076/TopicSelection01/results
15:19:59 <yeliz> SH: main topic today is choosing the topic to focus
15:20:07 <yeliz> SH: main topics are benchmarking topics
15:20:12 <yeliz> SH: let discuss these
15:20:18 <shadi> shadi has joined #rd
15:20:23 <yeliz> SH: different opinions? objections?
15:21:25 <yeliz> Karl: What about doing another survey to choose the first one?
15:21:35 <yeliz> SH: We have two with the same ordering
15:21:47 <yeliz> SH: includes asking for editors
15:22:07 <yeliz> SH: should we say run off or lets choose now
15:22:39 <yeliz> SAZ: Lets discuss more, may be we can join some, etc
15:22:54 <yeliz> SAZ: it's some people's idea, who put it on the WIki
15:23:08 <yeliz> SAZ: but would be good to discuss them here
15:23:20 <yeliz> SH: Markel put these benchmarking items
15:23:38 <yeliz> SH: Markel, can we do eiither of them? Join them or do you prefer to do them together?
15:23:52 <yeliz> MV: would be good to do them seperately. but I am not sure
15:24:03 <yeliz> MV: I prefer to choose one and then go for that
15:24:15 <yeliz> SH: is the ordering of these topics important? 
15:24:21 <yeliz> MV: there is no specific order
15:24:40 <yeliz> SAZ: What is the expected output of sucj exploration? What do we hope that comes out of that?
15:24:50 <yeliz> SAZ: What impact such an outcome would have?
15:25:18 <yeliz> SAZ: and based on the impact, how can we maximize the output for web accessibility 
15:25:24 <yeliz> SAZ: we need to be more structured
15:25:24 <shadi> shadi has joined #rd
15:25:34 <yeliz> SAZ: what kind of outcome would make a different
15:25:51 <yeliz> SAZ: we would have 3/4 topics per year so we have to carefully choose the topic
15:26:23 <yeliz> MV: Metrics, the main outcome would be there are so many metrics out there, but we cannot say which one is the best or most appropriate in a defined scernario
15:26:45 <yeliz> MV: there is a paper that address these questions, we propose a framework that includes these metrics
15:27:06 <yeliz> MV: based on the requirements, this framework allow you to choose the metrics
15:27:19 <yeliz> MV: but there are many open questions
15:27:29 <yeliz> MV: the outcome would be a tool to support decision
15:27:57 <yeliz> KG: I tend to agree, what would give the best benefit, having experience with clients
15:28:10 <yeliz> KG: having such a tool would be very useful
15:28:21 <yeliz> KG: some other topics would also be useful for my clients
15:29:30 <yeliz> SAZ: I don't really understand what does metric mean in this case? Aggregation? Ranking? Test participants (subjective)? This is a complex topic....
15:30:05 <yeliz> SAZ: We should focus the topic and what is the expected outcome, or if we decide to go in a certain direction, we have to look at it carefully
15:30:38 <yeliz> KG: I agree with Shari, especially if you look at the description, it is very broad, would be good to explain which metrics we focus
15:31:23 <yeliz> SZ: I agree with you, but instead of discussing these in the telecon, we propose a wide topic such that we can all contribute, and then as the telecons go, we can narrow down the topic
15:31:39 <yeliz> SZ: this might be more useful and more appropriate
15:31:59 <yeliz> SZ: before looking at the background and finding editors hard to make this decision
15:32:03 <yeliz> KG: I agree
15:32:24 <yeliz> SAZ: working out scientific questions with a background survey
15:33:04 <yeliz> SAZ: one of the concerns I have, WCAG working group put a lot of work into combining the requirements of disabilities, and there is a potential here to undo it
15:33:51 <shadi> shadi has joined #rd
15:34:04 <yeliz> SAZ: instead we need a scientific approach, we come up with questions
15:34:14 <yeliz> SH: Markel, can you give us more info
15:34:47 <yeliz> MV: the purpose was to focus on automatic metrics, those are included by the automated evaluation tools in the reports
15:35:14 <yeliz> MV: one question, how do these metrics reflect the experience of disabled people? 
15:35:35 <yeliz> MV: not only using automatic tools or users, how about the opinion of experts?
15:35:57 <yeliz> MV: how do expert metrics compare to the user experience
15:35:59 <yeliz> ?
15:36:21 <yeliz> SAZ: what is the impact of this with the lives of disabled people?
15:36:31 <yeliz> SAZ: how can we maximize the impact of our work?
15:36:48 <yeliz> SAZ: may be we need to think how to improve the impact?
15:37:09 <yeliz> SH: Its difficult to think of that in massive detail because the process is new to all of us
15:37:30 <yeliz> SH: we can look at how different evaluation tools perform, 
15:37:54 <yeliz> SH: evaluating pages for cognitive disabilities, metrics that would be useful for all users
15:38:20 <yeliz> SAZ: the metrics that you are referring would they be any different from WCAG
15:38:46 <shadi> shadi has joined #rd
15:38:59 <yeliz> SH: we have AI metric, or Barrier Walkthrough, Metrics from MV's thesis, or metrics that WAVE uses
15:39:31 <yeliz> SH: we also know that with WCAG we have metrics that cannot be automated
15:39:50 <yeliz> SH: and we know that getting 80% agreement between people is very difficult
15:40:09 <yeliz> SH: so what would be the effect of combining them
15:40:23 <yeliz> MV: metrics are not specifically attached to guidelines
15:40:57 <yeliz> MV: I tried to focus on automated metrics, because they are quicker to get, you don't need to wait for experts, or user evaluation
15:41:07 <yeliz> MV: automated metrics are much quicker to get
15:41:21 <yeliz> MV: you can also combine manual, user or automated tools
15:41:27 <yeliz> SH: I would also agree with that
15:42:04 <yeliz> SH: may be we need to look at the pages that evolve, may be some metrics can be still kept
15:42:39 <yeliz> SH: or may be some evaluation aspect are still valid, but may be some dynamic part of the page can be evaluated
15:42:49 <shadi> shadi has joined #rd
15:42:55 <yeliz> SH: as the MV says there are so many things like this that have not been investigated
15:43:05 <yeliz> SH: this is not a one shot deal
15:43:20 <yeliz> SH: there might be a lot of metrics that people use in different context
15:43:34 <yeliz> SAZ: why do we have so few references
15:43:57 <yeliz> SAZ: we have to be careful because we do not have resource to carry experiements
15:44:10 <yeliz> SAZ: we mainly invite existing research
15:44:18 <yeliz> SAZ: how would we go about this
15:44:20 <yeliz> ?
15:44:36 <yeliz> SH: there might be some stuff or might be nothing
15:45:15 <yeliz> SH: may be we look into this and then bring them to the group
15:45:28 <yeliz> SH: published research, published papers, experiment conducted
15:45:38 <yeliz> SH: MV have you already done some work on this?
15:45:54 <yeliz> MV: we already have a related work section on metrics in our paper
15:46:07 <yeliz> MV: we compared them, discussed them..
15:46:19 <yeliz> MV: may be we missed some...
15:46:30 <yeliz> SAZ: what is the conclusion?
15:46:47 <shadi> shadi has joined #rd
15:46:53 <yeliz> MV: all metrics are different, that is the main outcome
15:47:08 <yeliz> MV: some are good distinguishing accessible/non-accessible pages
15:47:20 <yeliz> MV: most of them are weak, and some of them are strong
15:47:41 <yeliz> SH: we can invite people to give a seminar who actually created these metrics
15:48:02 <yeliz> SAZ: the problem is what is it that we want to achieve?
15:48:14 <yeliz> SAZ: do we want to recompare them? What is it that we want to do?
15:48:24 <yeliz> SAZ: I am not sure what we are going to do?
15:49:00 <yeliz> MV: we have downloaded a lot of pages, if we use LIFT, or WAVE, how does the score change? An example direction
15:49:28 <yeliz> KG: when you do that don't you have a lot of unctrollable variables
15:50:11 <yeliz> KG: for example, do they crawl pages properly? If you use experts, what is the method used? 
15:50:38 <yeliz> KG: metrics and their strength, depend on the tool used, or depend on the expert used, their experiences
15:50:39 <shadi> shadi has joined #rd
15:50:57 <yeliz> KG: this could be something much bigger than what we think or chew...
15:51:19 <yeliz> SAZ: we want evaluation or authoring tools to apply and use these metrics
15:51:25 <yeliz> SAZ: why do we want these metrics
15:51:46 <yeliz> SAZ: purpose is to improve the experience of disabled people, right
15:51:57 <yeliz> SH & MV: +1
15:52:17 <yeliz> SAZ: we want to advice people on the quality of metrics, right?
15:52:31 <yeliz> SAZ: I am trying to come up with a goal for this work
15:53:03 <yeliz> MV: a framework to support the decision making on metrics
15:53:18 <yeliz> SAZ: I am confused about the metrics and WCAG 2.0 techniques
15:53:51 <yeliz> SAZ: guide people to write techniques, may be we can guidance on writing techniques
15:54:08 <shadi> shadi has joined #rd
15:54:13 <yeliz> SAZ: making the link between WCAG criteria and techniques to achieve them
15:54:34 <yeliz> SAZ: we can provide guidance for people to create good techniques
15:54:45 <yeliz> SH: not sure about techniques?
15:55:43 <yeliz> SH: WCAG 2.0 is one way of evaluating pages, but how we express the result of evaluation? all the tools out there they all looking at WCAG 2.0 but the metrics used are all different
15:56:16 <yeliz> SH: not saying we should analyse the tools existing, but understand the metrics better and provide guidance on providing metrics at the end
15:56:50 <yeliz> SH: guidelines are there and the success criteria fixed, but how do we look at these evaluation results
15:57:08 <yeliz> SH: as the KG says, we don't know really how people do the evaluation
15:57:33 <yeliz> SH: may be if we provide metrics, then you can better guide people to do and present the tests
15:58:04 <yeliz> SAZ: if we do testing with experts, or with users or with automated tools, do they peresent different measurement?
15:58:27 <yeliz> SH: some system assume best or worst in applying a criteria? different kinds of users?
15:58:51 <yeliz> SH: if the users from different kinds of disabilities, may be the metrics used would be very different?
15:59:05 <yeliz> SH: but if you have one disabled user the metrics would be different?
15:59:18 <yeliz> YY: what is a metric? I am confused!!!
15:59:31 <yeliz> SAZ: this is very different from what we have on Wiki
15:59:47 <yeliz> zakim, unmute yeliz
15:59:47 <Zakim> yeliz should no longer be muted
16:00:12 <yeliz> SH: may we should have an action on writing what would be the outcome?
16:00:47 <yeliz> YY: what is a metric?
16:00:54 <arun> have to leave now, bye
16:00:56 <yeliz> zakim, mute me
16:00:56 <Zakim> yeliz should now be muted
16:01:00 <Zakim> -??P16
16:01:13 <shadi> shadi has joined #rd
16:01:36 <yeliz> SAZ: what would be the goal of such exploration?
16:01:52 <yeliz> SAZ: to define/ or provide your definition of key terms such as metric
16:02:04 <yeliz> SAZ: think of a set of questions that can be investigated?
16:02:13 <yeliz> SAZ: elaborate the questions we have on Wiki
16:02:42 <yeliz> SAZ: it might worthwhile to work on the page on Wiki and expand i
16:02:43 <yeliz> t
16:02:50 <yeliz> based on the discussion we have 
16:03:01 <sharper> Action: markel to capture the meaning (definition) and outcome expected of 'Benchmarking Web Accessibility Metrics'.
16:03:02 <trackbot> Created ACTION-1 - Capture the meaning (definition) and outcome expected of 'Benchmarking Web Accessibility Metrics'. [on Markel Vigo - due 2011-08-04].
16:03:06 <yeliz> MV: what would be the right place for dicsussing definitons
16:03:13 <yeliz> ?
16:03:24 <shadi> shadi has joined #rd
16:03:30 <yeliz> SAZ: some key issues/terms need to be further explained
16:03:41 <yeliz> SAZ: bring out the purpose/goal and research question
16:03:57 <yeliz> SAZ: if we god to public and ask them to contriute, what is it that we want to ask people
16:04:06 <yeliz> MV: OK, I got it
16:04:16 <yeliz> SAZ: it seems like we already have our first topic
16:04:25 <yeliz> SH: it seems like it, 
16:04:42 <yeliz> SH & SAZ: please continue to contribute to Wiki
16:04:47 <yeliz> SH: anything else?
16:04:55 <Zakim> -Karl
16:04:58 <yeliz> SH: thanks for joining, see you next week
16:05:03 <yeliz> zakim unmute yeliz
16:05:09 <yeliz> zakim, unmute yeliz
16:05:09 <Zakim> yeliz should no longer be muted
16:05:18 <markel> markel has left #rd
16:05:19 <yeliz> yeliz has left #rd
16:05:20 <Zakim> -sharper
16:05:40 <Zakim> -yeliz
16:06:00 <Zakim> -??P61
16:10:37 <Zakim> disconnecting the lone participant, Shadi, in WAI_RDWG()11:00AM
16:10:39 <Zakim> WAI_RDWG()11:00AM has ended
16:10:41 <Zakim> Attendees were +1.703.225.aaaa, sharper, yeliz, Shadi, Karl
16:36:14 <shadi> shadi has joined #rd
16:36:50 <shadi> trackbot, end meeting
16:36:50 <trackbot> Zakim, list attendees
16:36:50 <Zakim> sorry, trackbot, I don't know what conference this is
16:36:51 <trackbot> RRSAgent, please draft minutes
16:36:51 <RRSAgent> I have made the request to generate http://www.w3.org/2011/07/28-rd-minutes.html trackbot
16:36:52 <trackbot> RRSAgent, bye
16:36:52 <RRSAgent> I see 1 open action item saved in http://www.w3.org/2011/07/28-rd-actions.rdf :
16:36:52 <RRSAgent> ACTION: markel to capture the meaning (definition) and outcome expected of 'Benchmarking Web Accessibility Metrics'. [1]
16:36:52 <RRSAgent>   recorded in http://www.w3.org/2011/07/28-rd-irc#T16-03-01