14:56:45 <RRSAgent> RRSAgent has joined #eval
14:56:45 <RRSAgent> logging to http://www.w3.org/2014/02/27-eval-irc
14:56:47 <trackbot> RRSAgent, make logs world
14:56:47 <Zakim> Zakim has joined #eval
14:56:49 <trackbot> Zakim, this will be 3825
14:56:49 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 4 minutes
14:56:50 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
14:56:50 <trackbot> Date: 27 February 2014
14:59:58 <Detlev> Detlev has joined #eval
15:00:05 <alistair> alistair has joined #eval
15:00:38 <Mike_Elledge> Mike_Elledge has joined #eval
15:00:54 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
15:01:03 <Zakim> +Detlev
15:01:14 <Zakim> -Detlev
15:01:15 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
15:01:15 <Zakim> Attendees were Detlev
15:01:16 <Detlev> Zakim
15:01:27 <Detlev> Zakim, mute me
15:01:27 <Zakim> sorry, Detlev, I don't know what conference this is
15:01:45 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
15:01:51 <Zakim> +[IPcaller]
15:01:55 <Vivienne> zakim, IPcaller is me
15:01:55 <Zakim> +Vivienne; got it
15:01:59 <Zakim> +Mike_Elledge
15:02:03 <Liz> Liz has joined #eval
15:02:03 <Zakim> +[IPcaller]
15:02:09 <Zakim> +Kathy_Wahlbin
15:02:10 <alistair> zakim, ipcaller is alistair
15:02:10 <Zakim> +alistair; got it
15:02:49 <Mike_Elledge> Congratulations, Vivienne!
15:03:05 <Zakim> +Liz
15:03:13 <Zakim> +Detlev
15:03:15 <Sarah_Swierenga> Sarah_Swierenga has joined #eval
15:03:30 <Zakim> +??P24
15:03:34 <Detlev> Zakim, mute me
15:03:34 <Zakim> Detlev should now be muted
15:03:46 <MartijnHoutepen> Zakim, ??P24 is me
15:03:46 <Zakim> +MartijnHoutepen; got it
15:03:49 <Zakim> +Sarah_Swierenga
15:04:13 <MoeKraft> MoeKraft has joined #eval
15:05:10 <MartijnHoutepen> http://www.w3.org/WAI/ER/2011/eval/minutes
15:05:32 <Vivienne> scribe: vivienne
15:05:34 <Zakim> +MoeKraft
15:05:55 <Vivienne> take up aggendum 1
15:06:14 <MartijnHoutepen> https://utwentebs.eu.qualtrics.com/SE/?SID=SV_cGAAMLdmjk3FmHX
15:06:14 <MartijnHoutepen> 
15:06:50 <Vivienne> topic: test-run
15:07:21 <Vivienne> Eric has sent the link to the survey and we should all get this answered
15:07:46 <Vivienne> Question 1: does the survey have accessibility problems?  We can make some changes.
15:08:13 <Vivienne> Kathy: have access to accessibility people at Qualtrics so can get things fixed
15:08:33 <Vivienne> MH: we used the accessibility guidelines and the accessible question guides, but we need to know if there are any other problems
15:09:06 <Vivienne> MH: please have a look and send accessibility issues to Eric
15:09:20 <Detlev> q+
15:09:28 <MartijnHoutepen> ack detlev
15:09:32 <Detlev> ack me
15:09:49 <Zakim> +Mary_Jo_Mueller
15:10:18 <MaryJo> MaryJo has joined #eval
15:11:03 <Vivienne> DF: had a look at the options for the success criteria - checkboxes or radio buttons.  Probably not clear how to separate the different comments on the different pages - can paste in the URL.  problem is that if we don't separate the pages it isn't quite clear whether people make mistakes because we have to put all the comments for all of the pages together.  If would have been better to have a
15:11:04 <Vivienne> tool that could separate the pages.
15:11:27 <Vivienne> Df: it's not necessary to ask people to make comments on areas where things were not present - that could be dropped
15:11:50 <Vivienne> DF: regarding radio buttons or checkboxes a test that goes across a number of pages, checkboxes may be the better option
15:12:16 <Vivienne> DF: should just have 1 comment box for pass/fail
15:12:31 <Sarah_Swierenga> q+
15:12:41 <Vivienne> MH: wanted to ask if you liked checkboxes or radio buttons better
15:12:42 <MartijnHoutepen> ack sarah
15:13:04 <Vivienne> Sarah: looks like it's been changed to checkboxes except for 1.1.1 and 1.2.2 but everything else was a checkbox
15:13:15 <Vivienne> MH: told Eric to change 1 to a radio button so we can compare
15:13:22 <Vivienne> Sarah: prefer checkboxes
15:13:42 <Vivienne> Sarah: we want to know which pages people thought passed, and which pages people thought failed
15:14:10 <Vivienne> MH: you can have pages that fail and pages that pass - pro for checkbox approach
15:14:29 <Detlev> q+
15:14:32 <Vivienne> MH: if you have comments later, put it on mailing list or email Eric
15:14:33 <Detlev> ack me
15:14:39 <Vivienne> MH: leaning toward checkboxes
15:15:41 <Vivienne> DF: maybe its good to give a general short instruction before you start step 4 explaining to people how you explain input - e.g. checkboxes that you can check both so you're not restricted to pass or fail, but could pick both and copy URL and extend the appropriate comments for that URL consistently.  If we provide guidance we will get better answers.
15:16:41 <Vivienne> MH:  what about the length of the questionnaire - 10 pages is quite a lot.  We do have a timeout and you can log off and get back to it
15:17:00 <Detlev> Zakim, mute me
15:17:00 <Zakim> Detlev should now be muted
15:17:06 <MartijnHoutepen> q?
15:17:23 <Vivienne> MH: any other opinions?
15:17:58 <Vivienne> MH: timing - when should it be launched - comments for last version is up tomorrow.  Ideally we should start tomorrow to get the survey done - keep it open for a month.
15:18:03 <Vivienne> zakim, mute me
15:18:03 <Zakim> Vivienne should now be muted
15:18:13 <Vivienne> MH: any objections to starting soon?
15:18:17 <Sarah_Swierenga> q+
15:18:18 <Detlev> fine with me
15:18:46 <Vivienne> SS: not sure what the issue is about when we do the testing.  How long is the testing period for the group?
15:19:02 <Vivienne> MH: thinking about keeping it open for the whole of March - is that too long or too short?
15:19:09 <Vivienne> would be a bit short for me
15:19:10 <Sarah_Swierenga> +1 on a month
15:19:11 <Detlev> Is it just one site we will be testing?
15:19:21 <Vivienne> MH: just 1 site
15:19:55 <Vivienne> MH: would be good to finish in April and process the results in May
15:20:21 <Vivienne> zakim, ack me
15:20:21 <Zakim> unmuting Vivienne
15:20:22 <Zakim> I see Sarah_Swierenga on the speaker queue
15:20:52 <Detlev> why just one site? Will that not a bit one-sided?
15:21:28 <Detlev> q+
15:21:32 <MartijnHoutepen> q?
15:21:50 <MartijnHoutepen> q- sara
15:21:51 <Detlev> ack me
15:21:58 <Mike_Elledge> +1 for a month
15:22:00 <Vivienne> I will be away for last 2 weeks in March and then the first couple of weeks in April for WWW 2014
15:22:17 <Vivienne> DF: not sure why just 1 site.  2 sites might be better
15:22:37 <Vivienne> q+
15:22:46 <Mike_Elledge> q+
15:23:25 <Vivienne> DF: if we had different sites, we might get more coverage e.g. dynamic content, applications etc.  Many issues can only be appreciated when you test more than 1 site
15:24:11 <Vivienne> MH:  mostly a time issue.  We have 2 volunteers for sites.  We're asking a lot of people to do a full site.
15:24:30 <Detlev> Zakim, mute me
15:24:30 <Zakim> Detlev should now be muted
15:25:15 <MartijnHoutepen> q- vivienne
15:25:53 <Detlev> q+
15:26:03 <Detlev> ack me
15:26:03 <Vivienne> ME: could consider doing them sequentially.  Do 1 site first and get the feed-back and then follow this with a more complicated or different elements
15:26:32 <Sarah_Swierenga> q+
15:26:39 <Vivienne> DF: wonder whether we could extend the group of testers beyond the group here.  Is there anythng against having some kind of open request for outside help?  More people might be willing to have a go at it
15:26:54 <Vivienne> DF: it might be good to have 3 types of sites and people can pick which one they'd like to do
15:27:13 <Vivienne> MH: it was my idea that this would be open to the public so everyone could have a chance to use it and feed the results back to us
15:27:22 <alistair> q+
15:28:03 <MartijnHoutepen> q- mike
15:28:04 <Vivienne> MH: if there are more people involved, it wouldn't be so much of a problem
15:28:33 <Vivienne> SS: I have a question about opening it wide up for the general public.  In the earlier discussions we had talked about keeping the site and the results anonymous.
15:28:38 <Vivienne> q+
15:29:03 <Detlev> well, site nominations given in spite of that openness would be fine, wouldn't they?
15:29:07 <Vivienne> SS: reach out to colleagues to people who would want to help but respect the anonymity of the website.
15:29:14 <Vivienne> MH: this might be necessary for W3C also
15:29:19 <MartijnHoutepen> q- sara
15:30:03 <Vivienne> Alistair: we need to maintain control - the issue is that you wouldn't be able to understand if the public understood the requirements etc.  We need to have good people doing precise checking
15:30:11 <Vivienne> zakim, ack me
15:30:11 <Zakim> I see alistair on the speaker queue
15:30:19 <Detlev> q+
15:30:20 <alistair> q-
15:30:26 <Vivienne> MH: maybe not public, but people we know who would be able to do good testing
15:30:33 <Vivienne> ack me
15:31:07 <Sarah_Swierenga> q+
15:31:29 <Vivienne> The university website would not be okay with it being open to the public
15:31:54 <Detlev> ack me
15:31:56 <Vivienne> MH: we would have to talk with W3C about it as well.  Would mean limited people answering the survey.  Maybe not do too many websites in that case
15:33:05 <Vivienne> DF: if a site owner agrees to offering the site for testing by the general public, then there shouldn't be a problem.  We wouldn't mind to have that openly tested for the site we've offered.  Let people know that we're testing the methodology and not the site.  We might be able to get propositions for sites.
15:33:15 <alistair> q+
15:33:44 <Vivienne> Df:  we can keep it open then and let anyone test it.  The methodology is a public document and aims to help more people learn to test websites.  This would also tell us how people with less experience would test and whether they would get meaningful results.
15:34:01 <Tim> Tim has joined #eval
15:34:19 <alistair> q-
15:34:19 <Vivienne> MH: we could combine with public of some sites and more private for others.  Will check with Eric to see if it was meant to be public or internal.
15:34:55 <Vivienne> SS: ask Eric and Shadi what they think
15:35:05 <Vivienne> Could get the website owner some negative publicity
15:36:25 <Vivienne> Alistair: there is no problem in running the methodology on anyone's website as it's public.  The issue is who would say whether the EM was run correctly.  Think it should be a limited number of websites and keep it within a controlled group and then put it out for the general public also later.
15:37:06 <Detlev> q+
15:37:15 <Detlev> ack me
15:37:15 <MartijnHoutepen> q- sara
15:37:23 <Sarah_Swierenga> q-
15:38:17 <Vivienne> DF: is there any harm in publishng an open call explaining we're testing the methodology if you're a website owner and you may not mind - are you willing to propose your site and we can see if we get sites that nominate themselves for testing.  If they offer themselves, there is no harm in people test it even if they're not experienced - tells us something about the methodology
15:38:21 <Vivienne> zakim, mute me
15:38:21 <Zakim> Vivienne should now be muted
15:39:08 <alistair> q+
15:39:26 <Vivienne> Df: you can's constrain it too much because we don't have a way of tracking how they did the testing.  They may accept minor flaws or they may not.  That will be something we can't keep a tab on it.  I don't think we'll have much control anyway  - better to open it up to get more exposure - more testers.  Would let us know where the problems are.
15:39:42 <Vivienne> MH:  some good ideas.  Have to check whether W3C will allow us to do public testing.
15:40:05 <Vivienne> DF: how could they not?  It's an open document and should be exposed as widely as possible to find the problems
15:40:32 <MartijnHoutepen> ack alistair
15:41:08 <Vivienne> Alistair: the point is to have a controlled group so that you can compare the results.  Without that you can't compare the results.  If you open this up, who would validate whether the results for that website is reasonable.
15:42:01 <Vivienne> DF: you would probably see a pattern.  People would have some common misunderstandings, a pattern could appear.  We could see whether people misunderstand instructions.  Test can be used as a heuristic to guide you to things that you want to change
15:42:21 <Vivienne> Alistair: I can't see how you can do this with no control group
15:42:27 <Vivienne> DF: there would be obvious errors
15:43:30 <Vivienne> MH:  question about the number of sites - small number of sites gives more opportunity to compare.  We want to include the general public as much as possible.
15:43:58 <Vivienne> DF: about control group - would mean that we would need a group that have the same understanding - unlikely to happen even within this group of experts.
15:45:20 <Vivienne> Alistair: that is the purpose of the methodology.   Pick 1 or 2 sites.  Should be a controlled experiment to some extent.  We could then have another group besides the control to open to the public and that way we've got something to compare.
15:45:37 <Vivienne> MH: let's move this subject to the list
15:45:55 <Vivienne> Topic 3: choice of website
15:46:06 <Detlev> Zakim, mute me
15:46:06 <Zakim> Detlev should now be muted
15:46:51 <Vivienne> MH: 2 websites have been suggested and they score about the same
15:46:52 <Detlev> ack me
15:47:00 <Detlev> www.incobs.de
15:47:06 <Sarah_Swierenga> q+
15:48:21 <Vivienne> DF: described the website - a modern site, not huge.  Some changes to DOM scripting with the navigation.  One case of a modern clean site which tries to be accessible and meets WCAG 2.0 AA.
15:49:18 <Vivienne> MH: if we ask the public for a site, we would have to make sure that we can have consent.
15:50:39 <Vivienne> SS: yesterday I sent an email to Eric & Shadi -happy to forward the note.  Got a receptive web master in charge of all of the medical related websites at the university.  He suggested we take a part of 1 of the websites.
15:50:59 <Vivienne> SS: He would be happy to get feedback
15:51:23 <Sarah_Swierenga> http://msue.anr.msu.edu/topic/info/4_h
15:51:41 <Vivienne> http://www.ecu.edu.au/centres/library-services/overview
15:52:16 <Vivienne> q+
15:52:57 <Zakim> +Tim_Boland
15:53:11 <Vivienne> SS: this part of the website would be more than big enough
15:53:14 <Vivienne> zakim, ack me
15:53:14 <Zakim> unmuting Vivienne
15:53:15 <Zakim> I see Sarah_Swierenga on the speaker queue
15:54:33 <MartijnHoutepen> q- sara
15:55:39 <Detlev> ack me
15:56:06 <Vivienne> some parts of the library website are quite difficult to make compliant because of third-party product such as databases, logins, interactive campus maps etc.
15:56:23 <Vivienne> DF: no harm in looking at our websites, it's all good information
15:56:49 <Vivienne> MH: that gives us 3 good candidates for testing the methodology.  We need to talk to Eric and Shadi about whether it is public or not and whether we have consent.
15:57:20 <Mike_Elledge> +1
15:57:29 <Vivienne> MH: topic: face to face meeting in San Diego.  Need to register asap.  How many are attending?
15:57:40 <Liz> Tim at NIST will be attending.
15:57:43 <Sarah_Swierenga> +1 for the conference, but not until Tuesday afternoon
15:57:55 <MaryJo> I will be attending.
15:57:56 <Vivienne> Eric & Shadi will be there.  Shadi has got us a room to meet.  Arrange a skype session for an hour or so
15:57:58 <Vivienne> I would like that
15:58:00 <MoeKraft> +1
15:58:08 <MartijnHoutepen> +1
15:58:14 <Detlev> maybe..
15:58:21 <Vivienne> topic: other business
15:58:37 <Vivienne> ack me
15:58:39 <Detlev> bye
15:58:39 <Liz> bye
15:58:41 <Zakim> -alistair
15:58:43 <Zakim> -Tim_Boland
15:58:43 <Zakim> -Mary_Jo_Mueller
15:58:44 <Zakim> -Detlev
15:58:44 <Zakim> -Kathy_Wahlbin
15:58:46 <Zakim> -Sarah_Swierenga
15:58:46 <Zakim> -MoeKraft
15:58:54 <Mike_Elledge> Bye! Good job!
15:58:59 <Vivienne> good night
15:59:01 <MartijnHoutepen> trackbot, end meeting
15:59:01 <trackbot> Zakim, list attendees
15:59:02 <Zakim> As of this point the attendees have been Vivienne, Mike_Elledge, Kathy_Wahlbin, alistair, Liz, Detlev, MartijnHoutepen, Sarah_Swierenga, MoeKraft, Mary_Jo_Mueller, Tim_Boland
15:59:09 <Zakim> -Mike_Elledge
15:59:09 <trackbot> RRSAgent, please draft minutes
15:59:09 <RRSAgent> I have made the request to generate http://www.w3.org/2014/02/27-eval-minutes.html trackbot
15:59:10 <trackbot> RRSAgent, bye
15:59:10 <RRSAgent> I see no action items
15:59:11 <Zakim> -Vivienne