14:54:57 <RRSAgent> RRSAgent has joined #eval
14:54:57 <RRSAgent> logging to http://www.w3.org/2014/02/13-eval-irc
14:54:59 <trackbot> RRSAgent, make logs world
14:55:01 <trackbot> Zakim, this will be 3825
14:55:01 <Zakim> ok, trackbot; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 5 minutes
14:55:02 <trackbot> Meeting: WCAG 2.0 Evaluation Methodology Task Force Teleconference
14:55:02 <trackbot> Date: 13 February 2014
14:55:13 <shadi> zakim, this will be eval
14:55:13 <Zakim> ok, shadi; I see WAI_ERTWG(Eval TF)10:00AM scheduled to start in 5 minutes
14:55:23 <shadi> zakim, who is on the phone?
14:55:23 <Zakim> apparently WAI_ERTWG(Eval TF)10:00AM has ended, shadi
14:55:24 <Zakim> On IRC I see RRSAgent, Kathy, shadi, Bim, Zakim, trackbot
14:57:35 <Zakim> WAI_ERTWG(Eval TF)10:00AM has now started
14:57:42 <Zakim> +Kathy_Wahlbin
14:57:54 <Zakim> +Mike_Elledge
14:58:49 <Mike_Elledge> Mike_Elledge has joined #eval
14:59:37 <EricVelleman> EricVelleman has joined #eval
14:59:52 <Zakim> + +1.301.990.aaaa
15:00:22 <Zakim> +Shadi
15:00:30 <Detlev> Detlev has joined #eval
15:00:43 <shadi> zakim, aaaa Liz
15:00:43 <Zakim> I don't understand 'aaaa Liz', shadi
15:00:46 <shadi> zakim, aaaa is Liz
15:00:46 <Zakim> +Liz; got it
15:01:07 <Zakim> +EricVelleman
15:02:10 <shadi> regrets: Moe, Alistair
15:02:30 <Zakim> +Detlev
15:02:38 <shadi> regrets: Sarah, Moe, Alistair
15:02:46 <Detlev> Zakim, mute me
15:02:46 <Zakim> Detlev should now be muted
15:03:32 <shadi> scribe: Mike_Elledge
15:03:34 <shadi> chair: Eric
15:03:38 <Vivienne> Vivienne has joined #eval
15:04:01 <shadi> agenda: http://lists.w3.org/Archives/Public/public-wai-evaltf/2014Feb/0004.html
15:04:25 <shadi> Topic: Test-run survey proposal
15:04:28 <Mike_Elledge> Eric sent around a draft questionnaire
15:04:33 <Zakim> +[IPcaller]
15:04:38 <shadi> http://lists.w3.org/Archives/Public/public-wai-evaltf/2014Feb/0006.html
15:04:42 <Vivienne> zakim, IPcaller is me
15:04:42 <Zakim> +Vivienne; got it
15:04:49 <Mike_Elledge> Received a few commments and included them
15:04:58 <Mike_Elledge> Uploaded new version
15:05:36 <Mike_Elledge> E: looked at methodology, tried to replicate most important steps.
15:06:14 <Mike_Elledge> E; If could put answers into spreadsheet could go through it easily.
15:06:34 <shadi> zakim, mute me
15:06:34 <Zakim> Shadi should now be muted
15:06:55 <Mike_Elledge> E: Intro: where to find working draft. Note how long it may take. That it is confidential, published anonymous.
15:07:07 <Mike_Elledge> K: What are we using.
15:07:13 <Mike_Elledge> E: Qualtrics.
15:07:26 <Mike_Elledge> K: Has some a11y bugs that they're working on.
15:07:56 <Mike_Elledge> E: In Q can indicate what is/not accessible. So only used items called accessible.
15:07:57 <gavinevans> gavinevans has joined #eval
15:08:35 <Mike_Elledge> K::Items that are accessible may take users to external pages that give directions for screen reader users. Some bugs still exist.
15:08:44 <Mike_Elledge> K: Will test drive survey.
15:09:25 <Mike_Elledge> E: Has numbered questions. Not logical numbers, but clearly indicated in tool.
15:09:48 <Mike_Elledge> E: Enter name, practical experience, days evaluating in 2013 (indicates experience) and email address.
15:09:52 <Zakim> + +44.179.281.aabb
15:10:03 <Kathy> q+
15:10:14 <Mike_Elledge> E: Some comments not to as for email address. Will suggest making information optional.
15:10:28 <Mike_Elledge> K: Please enter email address or phone number so can contact you.
15:10:39 <Kathy> q-
15:10:48 <Detlev> q+
15:11:03 <Mike_Elledge> K: Ask for email or phone number so we can contact you.
15:11:48 <Mike_Elledge> E: Survey follows WCAG Steps. Pls remember not testing you. Follow methodology as closely as possible. Then directions for how tosave contents.
15:12:24 <Mike_Elledge> E: Only saves,however, if you press Save before going back. Otherwise lose everything.
15:13:04 <Mike_Elledge> M: Is it possible to show a warning message?
15:13:14 <Mike_Elledge> E: Will look into it.
15:13:54 <Detlev> Are you using the queue today or are we supposed to speak up queueless?
15:14:03 <Mike_Elledge> E: Scope. Q105 Eval scope. Part we have to fill in. Should do on maling list. Scope of evaluation. Put @ signs to indicate we need input. Need more work.
15:14:06 <Detlev> ack me
15:14:29 <shadi> zakim, who is on the phone?
15:14:29 <Zakim> On the phone I see Kathy_Wahlbin, Mike_Elledge, Liz, Shadi (muted), EricVelleman, Detlev, Vivienne, +44.179.281.aabb
15:14:39 <Mike_Elledge> D: Comment to Q101. mentions w
15:14:49 <Detlev> Q101 How much practical experience do you have evaluating websites using the WCAG2.0 guidelines (or other schemes / regulations based o WCAG 2.0)?
15:15:12 <shadi> zakim, aabb is probably Gavin
15:15:12 <Zakim> +Gavin?; got it
15:15:32 <gavinevans> yes is me
15:15:37 <shadi> q+
15:15:43 <Mike_Elledge> D; wcag change to one pasted into survey. Test against German translation BITV.
15:15:49 <shadi> ack me
15:16:41 <Mike_Elledge> S: Maybe we can reformulate question. How much experience eval websites.
15:17:02 <Mike_Elledge> D: Just to make it clear that it counts. other countries have their own regs based on wcag.
15:17:30 <Mike_Elledge> S: Methodology does rely on WCAg 2.0. If using something else would be apples/oranges.
15:17:43 <Mike_Elledge> D: Why said or other schemes based on WCAG 2.0.
15:17:49 <Mike_Elledge> S: What's purpose of question?
15:18:09 <Detlev> Zakim, mute me
15:18:09 <Zakim> Detlev should now be muted
15:18:24 <Mike_Elledge> E: If people get different results is it caused by different pages or their experience. Not somehing that's definitive, but gives context.
15:18:39 <Mike_Elledge> E: If using differently may get different results.
15:18:53 <Mike_Elledge> S: # Days gets at it too.
15:19:17 <Mike_Elledge> E: Could use one question, but it's kind of a control. Great experience may be 5 days to some people.
15:19:36 <Mike_Elledge> S: Not disagreeing with Detlev, just wonder if it will add confusion.
15:19:52 <Kathy> q+
15:20:01 <shadi> q+
15:20:08 <Mike_Elledge> E: Could put some info that if persons use a national version with same checkpoints and success criteria would be okay.
15:20:15 <shadi> ack kathy
15:20:47 <Vivienne> I agree with Kathy.  Maybe a scale?  1-25, 26-50 etc?
15:21:04 <Mike_Elledge> K: How many days evaluating websites WCAG 2.0. do all the time, but have no idea. Suggest # of websites and perhaps ranges.
15:21:10 <Mike_Elledge> M: +1
15:21:15 <EricVelleman> +1
15:21:16 <Vivienne> list or range
15:21:23 <Mike_Elledge> E: Could turn into bulleted list.
15:21:37 <gavinevans> q+
15:21:51 <Detlev> queue
15:21:53 <shadi> q- last
15:21:58 <Mike_Elledge> E: Q104. The intro to eval scope.
15:22:05 <Mike_Elledge> S: Gavin first.
15:22:14 <shadi> ack gavin
15:22:38 <Mike_Elledge> G: How many websites. could do one, but could be very large with sections. Should take into consideration.
15:22:49 <Mike_Elledge> E: Will come up with proposal.
15:23:02 <Detlev> approx. number of days/year may be more expressive
15:23:02 <Mike_Elledge> S: Back on requirements for scope?
15:23:27 <Mike_Elledge> S: Base line seems too big. What we would usually expect?
15:23:29 <shadi> ack me
15:23:42 <shadi> s/big/vague
15:23:43 <Mike_Elledge> E: Just typed something there. <vague, not big>
15:23:58 <Mike_Elledge> E: Could put very specific things there.
15:23:59 <gavinevans> q+
15:24:13 <shadi> zakim, mute me
15:24:13 <Zakim> Shadi should now be muted
15:24:32 <Kathy> q+
15:24:39 <shadi> ack gavin
15:24:50 <Mike_Elledge> G: Massive? Gets down to browsers, versions, etc. Important that mentioned HTML, ARIA, earlier versions of AT important as well.
15:24:58 <shadi> ack kathy
15:25:26 <Mike_Elledge> K: Also add asking what kind of applications ahve been done, i.e., mobile. Gives indication of level of evaluations they'e done.
15:25:48 <shadi> q+
15:26:00 <shadi> ack me
15:26:03 <Mike_Elledge> E: will have to spend more time on this on list. Have to decide accessibility benchmark.
15:26:49 <Mike_Elledge> S: Have to be precise. Enumerate browsers and assistive technologies and combinations, to really ahve list that we assume that website should work for.
15:27:10 <EricVelleman> q?
15:27:20 <Mike_Elledge> S: Isn't that how you would do testing. Define what needs to work and that baseline is addressed during testing.
15:27:38 <Mike_Elledge> K: Have to prioritize though. There are so many combinations and limited budgets.
15:28:05 <Vivienne> q+
15:28:13 <shadi> zakim, mute me
15:28:13 <Zakim> Shadi should now be muted
15:28:16 <Vivienne> zakim, ack me
15:28:16 <Zakim> I see no one on the speaker queue
15:28:17 <Mike_Elledge> S: Agree, can make it short list. But need something so we have a target we're shooting for. Need to define minimum bar.
15:28:24 <Zakim> +Tim_Boland
15:28:46 <shadi> ack me
15:28:53 <Mike_Elledge> V: When we test give client a questionnaire if there is specific requirement they have. Ex Aus gov't site may requre only IE 9.
15:28:56 <Vivienne> zakim, mute me
15:28:56 <Zakim> Vivienne should now be muted
15:29:09 <shadi> zakim, mute me
15:29:09 <Zakim> Shadi should now be muted
15:30:04 <shadi> q+
15:30:12 <shadi> zakim, mute me
15:30:12 <Zakim> Shadi was already muted, shadi
15:30:25 <Mike_Elledge> E: If we test a website that is only required to be IE6, then will be only
15:30:32 <Mike_Elledge> E: that.
15:30:34 <shadi> ack me
15:31:35 <Mike_Elledge> S: Do have a definition of that in methodology. Can't use an intranet website so can test. Assume it will be english as well. Might be dependant on type of website.
15:31:44 <Mike_Elledge> E: Propose not to test Dutch website!
15:31:53 <Mike_Elledge> S: Put that in definition.
15:31:57 <shadi> zakim, mute me
15:31:57 <Zakim> Shadi should now be muted
15:32:42 <Mike_Elledge> E: Let's come back to scope on mailing list. Decide on what site we want to use, part, baseline, target, etc. Will be easier once we have website in front of us.
15:33:34 <shadi> [["how many minutes" -> "how much time"?]]
15:33:34 <Mike_Elledge> E: Next section. Technologies relied upon (can be selected in Qualtrics). How much time to do, easy to do, comments, etc. For Step Two.
15:33:39 <Mike_Elledge> E: Missing anything?
15:33:41 <shadi> ack me
15:33:53 <Vivienne> q+
15:34:01 <Mike_Elledge> S: Just minor. May to change from minutes to how much time.
15:34:08 <shadi> zakim, mute me
15:34:08 <Zakim> Shadi should now be muted
15:34:08 <Vivienne> zakim, ack me
15:34:09 <Zakim> unmuting Vivienne
15:34:09 <Zakim> I see no one on the speaker queue
15:34:24 <Mike_Elledge> V: Step Two, don't we have identify different templates?
15:34:58 <Mike_Elledge> E: Yes, but would have report on all the different parts that come back in Section 3. In the reporting we put others as optional. This is only non-optional one.
15:35:03 <Vivienne> zakim, mute me
15:35:03 <Zakim> Vivienne should now be muted
15:35:32 <Mike_Elledge> E: Step 3. Representative sample. Read it then paste exemplar instances, maybe past urls or descriptions instead.
15:35:38 <shadi> +1 to "provide the urls or descriptions ..."
15:35:58 <Mike_Elledge> Q: 115. Change to how much time. Then how chose random sample.
15:36:19 <Mike_Elledge> T: Have any indication of how big the site is, # pages?
15:36:28 <Mike_Elledge> E: We determine scope.
15:37:10 <Mike_Elledge> E: We will be telling them which site to review for consistency.
15:37:49 <Mike_Elledge> E: Were you able to get representative sample. Not scientific, but indicates comfort level.
15:37:55 <Mike_Elledge> T: Tehre is rationale, right.
15:38:24 <Mike_Elledge> E: Yes. In line with what usually use. Q119 How easy to follow instructions.
15:38:53 <Mike_Elledge> E: Select the urls, paste them, then go back to them.
15:39:04 <martijnhoutepen> martijnhoutepen has joined #eval
15:39:17 <shadi> q+
15:39:25 <Mike_Elledge> E: Step 4. Audit the sample. Introduction, then a page for each SC.
15:39:37 <Mike_Elledge> Q 121 not a qestions so much as remark.
15:39:38 <shadi> ack me
15:39:58 <Mike_Elledge> S: Thought earlier version was AA. Need to go with that.
15:40:05 <Detlev> agree about using AA
15:40:18 <Mike_Elledge> E: Okay.
15:40:34 <Mike_Elledge> S: A bit more work, but no way around it. Trying to make it easier.
15:40:57 <Mike_Elledge> T: If one goes through it would check A and AA accessibility.
15:41:13 <Mike_Elledge> E: If we dont SC1, then all applicable for A and AA.
15:41:27 <Mike_Elledge> S: Maybe 36, for total, not each.
15:41:30 <Mike_Elledge> +1
15:41:34 <shadi> zakim, mute me
15:41:34 <Zakim> Shadi should now be muted
15:41:45 <Mike_Elledge> E: Agree to expand to WCAG 2.0 AA.
15:42:13 <Mike_Elledge> E. Please read for audited sample. Then real tests. Have to decide, easy or hard work for E.
15:42:39 <Zakim> +??P10
15:42:56 <martijnhoutepen> zakim, ??P10 is martijnhoutepen
15:42:56 <Zakim> +martijnhoutepen; got it
15:43:13 <shadi> q+
15:43:17 <Vivienne> q+
15:43:30 <shadi> q- last
15:43:33 <Detlev> q+
15:43:33 <shadi> q- later
15:43:39 <shadi> q-
15:43:45 <Vivienne> zakim, ack me
15:43:45 <Zakim> unmuting Vivienne
15:43:45 <Mike_Elledge> E: Radio buttons or check boxes. This is relevant because on a page doesn't pass. But next page fails. Do we want possiblity for marking page pass...
15:43:45 <shadi> q+
15:43:46 <Zakim> I see Detlev on the speaker queue
15:44:26 <Mike_Elledge> V: We do both. First page is p, f, na and summarizes them according to sc
15:44:46 <Mike_Elledge> E: Can't use radio buttons.
15:44:57 <Detlev> ack me
15:44:57 <Mike_Elledge> V: dropdown for p, f, na?
15:45:02 <Vivienne> zakim, mute me
15:45:02 <Zakim> Vivienne should now be muted
15:46:07 <Vivienne> q+
15:46:40 <Mike_Elledge> D: What would be the purpose for recording each p, f for each page. May not be same pages. Not comparable. Someone may have made different decisions. Can't really compare, unless go back to every page. Waht would work is for people report common problems. Difficult to rate 1.3.1 for some reason. Detailed results for each page won't be usable.
15:47:06 <Mike_Elledge> E: Thought if we do it that way would be pushing people into one direction. Wanted to give them the possibility.
15:47:34 <Mike_Elledge> D: But will we be able to collect meaningful information. Comments, not necessarily specific results.
15:47:51 <Mike_Elledge> E: So you would prefer radio button that opens comment field?
15:48:13 <Mike_Elledge> D: Need a comments field, not so much radio button or checkbox.
15:48:55 <Mike_Elledge> D: Checking P/F no way to calculate results. Maybe one P/F for entire site. Can process since not dependant on particular pages.
15:48:58 <shadi> ack me
15:49:33 <Detlev> Zakim, mute me
15:49:33 <Zakim> Detlev should now be muted
15:49:46 <Mike_Elledge> S: I agree with Detlev. Particularly since later in process want to see how the procedure worked. Which page would require more analysis. Would skew questions about overall performance.
15:49:47 <martijnhoutepen> zakim, mute me
15:49:47 <Zakim> martijnhoutepen should now be muted
15:50:22 <Mike_Elledge> S: Step 5 A, looking at minimum requirements,
15:50:34 <shadi> http://www.w3.org/TR/WCAG-EM/#step5a
15:51:10 <Detlev> q+
15:51:10 <Mike_Elledge> S: Reading from Step 5, right after list there is a note, need to decide what the desired granularity is. can't compare two different levels. Need to set in scope.
15:51:18 <Mike_Elledge> q+
15:52:05 <Mike_Elledge> S: Would like to simplify so have greater chance of people completing the test run. Table might work. Show SC with PF comments.
15:52:42 <Mike_Elledge> S: Rather than by SC have it done by entire website. Otherwise do by page.
15:53:17 <Mike_Elledge> E: But if don't put that level of specificity may wind up with differing, but intresting results.
15:53:45 <Mike_Elledge> E: Need people to paste where people found failures. If all pass, one fail, can see what difference is.
15:54:45 <Mike_Elledge> S: If you want to see a long vs. short version, maybe we should have two tests. Was not clear to me level of granularity wanted.
15:55:02 <Mike_Elledge> S: Should not leave it open, otherwise won't know what we get.
15:55:36 <Mike_Elledge> E: But we have a method. If people have methodology A, B, C then we won't know how they'll interpreting our methodology.
15:55:43 <EricVelleman> q?
15:56:04 <shadi> zakim, mute me
15:56:05 <Zakim> Shadi should now be muted
15:56:06 <Mike_Elledge> S: Trying to test too many aspects? Clarity, efficiency, confidence. Lots of parameters.
15:56:07 <Vivienne> zakim, ack me
15:56:07 <Zakim> unmuting Vivienne
15:56:08 <Zakim> I see Detlev, Mike_Elledge on the speaker queue
15:57:01 <Mike_Elledge> V: Understand what Detlev was saying, part of it is seeing how people will use mehtoldogy. Would expect them to use all sc on each page. If they don't, won't be using it as we envision.
15:57:11 <Detlev> ack me
15:57:19 <Mike_Elledge> V: If they don't use each of guidelines then how we know what they'll do in practice?
15:57:19 <Vivienne> zakim, mute me
15:57:19 <Zakim> Vivienne should now be muted
15:59:44 <Mike_Elledge> D: Practical testing refers to wcag techniques as one approach. In practice people break items into individual checkpoints in 1.3. If we don't have that we won't know what causes 1.3 to be rated as fail. If we stipulate that any failure leads to failure of page then we will be consistent and comparable. Whether it will be meaningful is another question. If we checkbox then we can see there is variability, which would be useful.
16:00:01 <EricVelleman> q?
16:00:30 <Mike_Elledge> M: I'm confused.
16:00:32 <shadi> ack mike
16:00:42 <shadi> ack me
16:01:12 <Mike_Elledge> E: We decided not include in document that sometimes an error is okay. Every error is an error.
16:01:44 <Mike_Elledge> E: Rest will cover next week. You can send ideas for websites.
16:02:59 <Mike_Elledge> S: CSUN. Contract is still not yet signed. Scheduled to have room for both Monday and Tuesday. About 90% chance we are meeting Monday or Tuesday.
16:03:15 <Mike_Elledge> See you next week.
16:03:15 <martijnhoutepen> bye
16:03:15 <Vivienne> good night all
16:03:16 <gavinevans> many thanks bye
16:03:17 <Detlev> bye
16:03:18 <Zakim> -Tim_Boland
16:03:19 <Zakim> -Kathy_Wahlbin
16:03:20 <Zakim> -Liz
16:03:23 <Zakim> -Vivienne
16:03:24 <Zakim> -martijnhoutepen
16:03:24 <Zakim> -Mike_Elledge
16:03:26 <Zakim> -Detlev
16:03:26 <Zakim> -Gavin?
16:03:27 <Zakim> -EricVelleman
16:03:27 <Zakim> -Shadi
16:03:27 <Zakim> WAI_ERTWG(Eval TF)10:00AM has ended
16:03:27 <Zakim> Attendees were Kathy_Wahlbin, Mike_Elledge, +1.301.990.aaaa, Shadi, Liz, EricVelleman, Detlev, Vivienne, +44.179.281.aabb, Gavin?, Tim_Boland, martijnhoutepen
16:03:28 <EricVelleman> EricVelleman has left #eval
16:03:33 <gavinevans> gavinevans has left #eval
16:07:41 <shadi> trackbot, end meeting
16:07:41 <trackbot> Zakim, list attendees
16:07:41 <Zakim> sorry, trackbot, I don't know what conference this is
16:07:49 <trackbot> RRSAgent, please draft minutes
16:07:49 <RRSAgent> I have made the request to generate http://www.w3.org/2014/02/13-eval-minutes.html trackbot
16:07:50 <trackbot> RRSAgent, bye
16:07:50 <RRSAgent> I see no action items