13:52:39 RRSAgent has joined #silver 13:52:39 logging to https://www.w3.org/2021/01/19-silver-irc 13:52:41 RRSAgent, make logs Public 13:52:42 Meeting: Silver Task Force & Community Group 13:52:55 Meeting: Silver Task Force & Community Group 13:52:55 present: 13:52:55 chair: Shawn, jeanne 13:52:55 present+ 13:52:55 zakim, clear agenda 13:52:55 rrsagent, make minutes 13:52:55 I have made the request to generate https://www.w3.org/2021/01/19-silver-minutes.html jeanne 13:52:55 q? 13:52:55 agenda cleared 13:53:16 regrets+ Makoto 13:53:28 agenda+ Update on publishing 13:53:28 agenda+ Update on the approved Decision policy 13:53:28 agenda+ Proposed change to next version of Requirements (take 3) 13:53:28 agenda+ Subgroups: status on January goals? 13:53:29 agenda+ Update from leadership 14:20:34 Fazio has joined #silver 14:23:10 sajkaj has joined #silver 14:25:33 present+ 14:25:47 ChrisLoiselle has joined #silver 14:26:36 Sorry group, can't scribe today...restarted my computer 4 times already today :( 14:29:17 Jemma has joined #silver 14:29:28 present+ 14:29:35 agenda? 14:29:39 Francis_Storr has joined #silver 14:29:44 present+ 14:29:46 Lauriat has joined #silver 14:29:48 JustineP has joined #silver 14:29:56 present+ 14:30:18 scribe: JustineP 14:30:26 zakim, take up next 14:30:26 agendum 1 -- Update on publishing -- taken up [from jeanne] 14:30:27 present+ 14:30:39 JakeAbma has joined #silver 14:30:50 present+ 14:30:53 Wilco has joined #silver 14:31:01 present+ 14:31:07 present+ 14:31:31 q+ 14:31:39 present+ 14:31:57 ack Rachael 14:32:04 Jeanne: Upcoming publication in the near future. 14:32:06 q+ so formal objections handled? 14:32:15 Chuck_ has joined #silver 14:32:20 present+ 14:32:28 present+ 14:32:34 present+ 14:32:43 Janina: Have objections been handled? 14:32:45 CharlesHall has joined #silver 14:32:52 present+ 14:33:19 regrets, I have to drop at 10am ET 14:33:32 zakim, take up next 14:33:32 agendum 2 -- Update on the approved Decision policy -- taken up [from jeanne] 14:33:34 JF has joined #silver 14:34:15 KimD has joined #silver 14:34:15 Rachael: The CfC for Silver decision policy has passed. 14:34:15 https://lists.w3.org/Archives/Public/public-silver/2021Jan/0069.html 14:34:28 sarahhorton has joined #silver 14:34:38 ...one typo will be corrected. 14:34:58 present+ 14:35:08 zakim, take up next 14:35:08 agendum 3 -- Proposed change to next version of Requirements (take 3) -- taken up [from jeanne] 14:35:15 Present+ 14:35:17 Present+ 14:35:21 Sheri-B-H has joined #silver 14:35:21 present+ 14:35:27 present= 14:35:38 present+ 14:35:38 Minutes from where we left off -> https://www.w3.org/2021/01/15-silver-minutes.html#t06 14:36:07 All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This includes particular 14:36:07 attention to people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, and cognitive and learning disabilities. 14:36:44 +1 14:36:59 Jeanne: If you agree, please enter +1 14:37:14 Janina: Suggest last "and" should be "or" 14:37:21 +1 14:37:27 I agree with "or" insteand of "and" 14:37:33 I agree with or 14:37:48 Q+ to ask how user-research is measured 14:37:55 q+ 14:38:07 ack JF 14:38:07 JF, you wanted to ask how user-research is measured 14:38:34 q+ to respond to jf 14:38:40 John F: Agree that user research would be incorporated, but not sure how it would be measured. We certainly use it, but how to measure? 14:38:44 may be "usability testing"? 14:38:45 q+ to talk to user research question 14:39:05 q+ 14:39:22 Jeanne: When drafted, people who recommended its inclusion had recommendations on how to measure. Haven't written guidelines yet and as a result haven't delved into the details. 14:39:48 +1 to JF 14:39:49 John F: Suggest replacing "measure" with "evaluation" 14:39:49 +1 to Jemma, should read "usability" rather than "user" 14:39:50 +1 to evaluation 14:40:10 q? 14:40:31 I'm fine with "measurement" because it's "such as" 14:40:32 Janina: Doesn't bother me, as its a requirements document with examples. Not suggesting that all will be used. 14:40:47 ack sajkaj 14:40:47 sajkaj, you wanted to respond to jf 14:41:09 ack Rachael 14:41:09 Rachael, you wanted to talk to user research question 14:41:10 ack Rachael 14:41:36 user research and usability testing are both appropriate. user testing is not. 14:41:58 +1 to Charles, that exactly. 14:42:03 Rachael: History on "user research"...usability testing does not include heuristic evaluation. May have landed on "user testing" as terminology but we should agree on terminology and use consistently. 14:42:15 q? 14:42:18 Jeanne: We don't test users... 14:42:25 ack JakeAbma 14:42:43 I feel like user research is an umbrella term 14:42:57 it includes multiple testing methods 14:43:35 Jake: I was wondering if current approach is set in stone. 14:43:39 Charles, I agree with you but we may want to put a discussion on the to do list. The current glossary (https://w3c.github.io/silver/guidelines/#glossary) includes "User testing Evaluation of content by observation of how users with specific functional needs are able to complete a process and how the content meets the relevant outcomes." 14:43:42 @ KimD - "such as" is not included in the sentence in question - implied, but not written 14:44:22 ...or do we have opportunity to propose Silver as a set of documents with technical documents, user needs, etc.? If so, sentence might be applicable to those documents. 14:44:54 Jeanne: This is a FWPD, nothing is set in stone. We are in early stages. 14:45:31 Andy has joined #silver 14:45:32 Janina: I'd suggest that the subject of the sentence is guidelines. 14:45:39 present+ 14:46:02 Jake: First three words are for guidance. 14:46:19 Jeanne: Should we not include? 14:46:34 Jake: Needs to be part of it, but scope might be expanded. 14:46:42 ack Wilco 14:47:14 Wilco: Wonder if we can remove "procedures". In ACT, procedures are problematic to include because there are many ways to test. What matters are the results. 14:47:29 +1 to Wilco 14:47:34 ...Otherwise, we may need a conversation about why procedures are bad. 14:47:45 No objection to Wilco here 14:48:26 Maturity! 14:48:27 Jeanne: I believe "procedures" is there to leave us opening for more user research-oriented and maturity-oriented guidance. May not be in bronze level regulatory section, but could be in Silver or Gold levels. 14:48:40 ...In that case, we could include "procedures". 14:48:49 Wilco: Would still recommend against it. 14:48:58 q? 14:49:11 John F: How do we measure user research? 14:49:20 Janina: Why do we need to specify that in a requirements document? 14:49:20 User reseach includes usability testing 14:49:34 John F: "User research" is an important term in that sentence. 14:49:36 multiple ways of usability testing 14:49:56 q+ 14:50:05 Janina: If we specify a way to conform based on working with user testing, we'd need to specify many things. Its just an example of ways to evaluate. 14:50:14 ...Its exemplary, no more. 14:50:25 q+ 14:50:34 q+ 14:50:36 John F: Don't know how you measure some of these things. 14:50:55 Janina: Its not what we are doing now. 14:51:03 ack faz 14:51:45 David: The term includes brain wave mapping, eye tracking studies...abstract concepts whereas "user research" is more subjective. Its an umbrella term that covers many types of testing. 14:52:05 ...Covers cognitive disabilities and more difficult concepts to handle. 14:52:08 ack rach 14:53:04 Rachael: Acknowledge the concerns. We have concepts of measuring vs. evaluating. User research includes measurable pieces but also concepts that are more difficult to measure. We can change to "evaluating" or keep "measuring" but replace "user research" with "user testing". 14:53:21 +1 to Rachael, with a preference for using 'evaluate' 14:53:36 Jeanne: From academic viewpoint, what is the difference between measuring and evaluating? The terms seem synonymous. 14:53:50 Rachael: Evaluating doesn't have a clear scale. 14:53:50 evaluating = review. measure = tally the reuslts of the review. 14:54:01 ...Measuring is more specific than evaluating. 14:54:19 Andrew: Measuring can be done with a tool. Evaluating involves judgment/subjective perspective. 14:54:27 Oxfored Dictionary: verb - form an idea of the amount, number, or value of; assess. 14:54:35 Jeanne: Thank you. I'm fine with changing to "evaluating". 14:54:37 ack Chuck 14:55:11 s/ with "user testing / with "usability testing" 14:55:12 Propossed change: All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of evaluating (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This includes particular attention 14:55:12 to people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. 14:55:16 Chuck: Entered change so that we can see suggested change. I'm fine either way. 14:55:28 q+ 14:55:44 ...John, are you more comfortable with "evaluating"? 14:55:50 John F: Yes. 14:55:57 ack Racha 14:56:05 q+ preference for measure 14:56:19 can't we use both terms? 14:56:35 ack pref 14:56:35 preference, you wanted to discuss measure 14:56:37 ack on 14:56:38 ack the 14:56:40 Rachael: Would prefer "measuring" but its a slight preference only. Evaluating doesn't indicate comparable method of assessment. 14:56:50 q+ to say that WCAG2 isn't measurable in that sense 14:57:20 Wilco: I share Rachael's preference. "Measure" is more concrete. 14:57:20 q- 14:57:35 John F: How do you measure user research? 14:57:36 i agree with JF. if use measure, we have to indicate how that could be measured. 14:57:53 q+ 14:57:54 focus groups, eye tracking studies brain wave mapping 14:58:06 ack sarahhorton 14:58:38 Alternative would be: All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring (for example: rubrics, sliding scale, task-completion, usability testing with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This includes particular attention to people whose needs may better be met 14:58:38 with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. 14:58:50 Sarah: There are ways to measure. Depends on type of research being conducted. 14:59:20 q+ 14:59:39 All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring or evaluating (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This includes particular attention to 14:59:39 people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. 15:00:03 have to drop 15:00:33 +1 to measuring or evaluating 15:00:44 q+ to ask aren't we using and? 15:00:48 ToddLibby has joined #silver 15:00:51 ack Ch 15:00:51 Chuck_, you wanted to ask aren't we using and? 15:00:55 present+ 15:00:59 All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring and evaluating (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This includes particular attention to 15:00:59 people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. 15:00:59 Jeanne: Will adjust to "measuring and evaluating" 15:01:01 +1 to *OR*, -1 to *AND* - which implies both must happen 15:01:21 ack Jak 15:01:34 For what it is worth, for usability and ISO definition, usability requires us to measure effectiveness, efficiency and satisfaction. Two measures that will justify design changes...time on task and success rate... whether this adds to measure of evaluate is up for discussion. 15:02:12 +1 for the revised version 15:02:40 q+ to ask that as this is a first draft, it's ok to change, and even expected at some level 15:02:42 Jake: Sarah mentioned that scoring is a part of benchmarking which is a different approach than we've chosen. Are those still a part of scoring system? If so, current approach for scoring will probably change and may be completely different. 15:03:12 Jeanne: I have not objection to including "benchmarks". 15:03:31 s/not/no 15:04:04 Jake: May be a big change if we include in Bronze level scoring. Would be smaller change if included in Silver or Gold. 15:04:33 Jeanne: There are many definitions of benchmark testing. We might be coming from different perspectives. 15:04:54 ...I'm fine with including "benchmarking" in examples though. 15:05:20 ...any objections to including "benchmarking"? 15:05:46 Chuck: Slightly although not strongly against it. The more examples we include, the more detailed it gets. 15:06:16 Jake: When John F. was asking about ways to measure, Sarah gave benchmarks as an example of ways to measure. If we don't include, we should circle back to John's question. 15:06:57 Chuck: We have multiple considerations at this point. I'm fine with including "benchmark". 15:07:00 one thing is that "bench mark" can be a very broad concept and many different meanings by diciplines or domain area - bussiness or computer domain 15:07:04 Q+ 15:07:17 ack chu 15:07:17 Chuck_, you wanted to ask that as this is a first draft, it's ok to change, and even expected at some level 15:07:23 John F: No problem with "benchmark" although I do prefer "or" rather than "and". 15:07:42 ...actually, I feel quite strongly about it. 15:07:56 if we want to add, can we add it as "bench mark testing"?? 15:08:17 s/bench mark testing/benchmark testing 15:08:30 ack JF 15:08:57 JaEun: Curious if we understand benchmark testing. From a business perspective, it has a different meaning. 15:09:09 Jeanne: Good point. 15:09:14 q+ 15:09:23 q+ 15:09:29 Janina: We may not have the same understanding but this is an exemplary list. 15:09:50 Jake: We need to have consensus of what we understand the term to mean. 15:09:57 s/a business perspective/a bussiness perspective or computer science perspective 15:10:09 Jeanne: Would you be okay with not including in the list because its a list of examples, but we can work on it in the future. 15:10:50 Jake: Sure... 15:11:05 ack ja 15:11:08 ack sara 15:11:11 User Research Example: Eye Tracking study pinpoints fixation points, duraation of fixaation, scan patterns. multiple sessions with multiple users provides actionable insight 15:11:17 This may be helpful regarding Metrics etc https://www.w3.org/TR/accessibility-metrics-report/ 15:11:45 Sarah: Method that I mentioned was system usability scale. There's a question about what "benchmarking" actually means. I was using as an example of ways that we've measured accessibility via user research. 15:12:07 Jeanne: We are still at "measuring and evaluating" or "measuring or evaluating" 15:13:02 ...Who is on the "and" vs. "or" side? 15:13:12 John F: Would like "or". 15:13:33 ...struggling with term "user research". 15:13:58 ...My concern is that we're suggesting we will measure user research. 15:14:02 We could still change "user research" to "usability testing" which is a bit more specific and has clear ways to measure (success or error rate, etc) 15:14:13 Jeanne: We added "evaluating" so that we can include user research. 15:14:24 John: Prefer "or" because we can't measure everything. 15:14:35 ORs +1? 15:14:35 +1 or 15:14:36 -1 15:14:40 q- use and 15:14:42 -1 15:14:42 0 15:14:42 -1 15:14:45 0 15:14:46 -1 15:14:46 0 15:14:53 +1 for "or" 15:15:00 0 15:15:06 -1 15:15:10 I think measuring is the foundation of evaluation and it is connected to me. 15:15:22 I agree with Janina 15:15:28 s/it/both 15:15:42 For user research, you can analyze qualitative data from a field visit . You can analyze and interpret existing data. You observe the user doing "X". 15:15:50 Jeanne: We might have a connotation problem in that people are looking at the terms in different contexts. 15:16:06 maybe and/or will be the compromise 15:16:24 s/it is connected to me/both are connected closely 15:16:40 analyze and evaluate are not measure 15:17:04 + 1 to Fazio 15:17:09 Rarely in qualititative methodology do we "measure" per say. We do evaluate. Tpyically we pick qualitative because the situation lends itself to not measuring. 15:17:36 Chris: Adding context. User research can lead to quantitative and qualitative analyses. Research piece is by user tester and researcher, but its getting caught up in the word "evaluate". 15:17:51 Evaluate: *CAN* the use click the button. Measure: *HOW LONG* does it take to click the button 15:18:05 Jeanne: Could everyone who voted +1 live with "and"? 15:18:10 No 15:18:18 Or everyone who voted -1, can you love with "or"? 15:18:18 +1 I can live with "or" 15:18:26 not that big of deal so yes 15:18:32 +1 i can live with and 15:18:47 I can leave with opposite way. 15:19:20 John F: My preference remains with "or" and can't live with "and". 15:19:21 q+ to say there is a "live with" consensus for "or" 15:19:43 Jeanne: Can everyone live with "or"? 15:19:44 +1 15:19:45 +1 15:19:52 +1 15:19:55 +1 15:19:56 Who can live with Or? 15:19:58 +1 to live with or 15:20:10 +1 15:20:15 Jemma: To confirm, "or" give more flexibility? 15:20:18 John F: Yes. 15:20:23 +1 15:21:05 Sarah: I can live with "and/or". 15:21:09 John F: Same. 15:21:19 Jeanne: Let's go with "and/or". 15:21:34 All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring and/or evaluating (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This includes particular attention to 15:21:34 people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. 15:21:44 q? 15:21:44 q+ 15:21:49 ack Ch 15:21:49 Chuck_, you wanted to say there is a "live with" consensus for "or" 15:22:06 Wilco: Don't really like "and/or". 15:22:30 +1 to live with tests or procedures 15:22:31 Jeanne: Let's try to wrap this up. 15:23:02 ...Let's go with "tests or procedures". Straw poll please. 15:23:03 -1 to test or procedures 15:23:07 tests or procedures +1 15:23:11 0 15:23:14 +1 15:23:42 Jemma: Can you remind me of the issue? 15:23:55 +1 to evaluating outcomes 15:24:03 Wilco: Procedures is highly prescriptive when when only need repeatable outcomes. 15:24:18 wouldn't you measure off of an evaluation method? ie. formative vs. summative, moderated vs. unmoderated, lab v. remote test, usability tests vs. expert review? 15:25:15 Jeanne: Opposite side is that group envisioned "procedures" as something that could go in Silver or Gold conformance (a long time ago). If you follow the procedure, its more of a maturity model that could lead to Silver or Gold conformance. 15:25:22 q+ 15:25:30 benchmarking - wouldn't that be comparing one thing directly to another? not sure what the group decided on benchmarking. 15:25:35 Wilco: Still think that its not a procedure, I'd rather use a different word. 15:26:00 ack wil 15:26:06 ack sar 15:26:07 Sarah: I agree Jeanne re: keeping door open to opportunities to demonstrate commitment and progress that extends beyond tests. 15:26:29 ...wonder if we can say "Silver guidance includes tests..." if "procedures" feels too confining. 15:26:43 agenda? 15:26:50 ..or if we can generalize to give more flexibility. 15:27:11 Jeanne: That's fine with me. Can you draft a phrase that would capture your recommendation? 15:27:14 Sarah: Yes. 15:28:27 Jeanne: Sarah, can you send to the list by Friday? 15:28:35 Yep! 15:28:37 Here's where we are leaving off: Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring and/or evaluating (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This 15:28:37 includes particular attention to people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. 15:28:57 Sarah will propose a phrase to substitute "procedures". 15:29:28 Jeanne: On Friday, we'll discuss subgroups and goals for January. 15:29:52 ToddLibby has left #silver 15:29:54 rrsagent, make minutes 15:29:54 I have made the request to generate https://www.w3.org/2021/01/19-silver-minutes.html jeanne 15:30:02 Wilco: Can't be on Friday's call, but okay if word "procedures" is taken out. 15:30:09 rrsagent, make minutes 15:30:09 I have made the request to generate https://www.w3.org/2021/01/19-silver-minutes.html jeanne 15:30:20 KimD has left #silver 15:31:21 RRSAgent, make logs public 15:31:49 trackbot, end meeting 15:32:00 Zakim, please part 15:32:00 leaving. As of this point the attendees have been Sheri-B-H, Andy, ToddLibby 15:32:00 Zakim has left #silver 15:34:26 Corrected proposal for Friday "All Silver guidance includes tests or procedures. Some guidance may use true/false verification but other guidance will use other ways of measuring and/or evaluating (for example: rubrics, sliding scale, task-completion, user research with people with disabilities, and more) where appropriate so that more needs of people with disabilities may be included. This 15:34:26 includes particular attention to people whose needs may better be met with a broad testing approach, such as people with low vision, limited vision, or cognitive and learning disabilities. " 15:34:54 noting that we didn't discuss removing the first word "All". 15:34:58 rrsagent, make minutes 15:34:58 I have made the request to generate https://www.w3.org/2021/01/19-silver-minutes.html jeanne 15:47:36 jeanne has joined #silver 16:06:29 mbgower has joined #silver