W3C

Results of Questionnaire Follow On from 29 April Face to Face Meeting

The results of this questionnaire are available to anybody.

This questionnaire was open from 2021-04-29 to 2021-05-04.

17 answers have been received.

Jump to results for question:

  1. One-line description of question
  2. How should we handle scope in 3.0?

1. One-line description of question

Based on the following discussion points from the meeting, can we agree to use a rating scale (adjectival rating) at the outcome (testable statement) level?

  1. Simplicity and transparency matter especially to regulatory and legal specialists.
  2. Transparency adds granularity so increases the number of checkpoints. Simplicity will most likely come from consistency in how we handle scoring. We should ideally pick a single approach to scoring outcomes.
  3. We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this.
  4. Assuming we want a single approach, percentages do not work across all outcomes so shouldn't be our choice. Percentages can be a threshold on some rating scales but not the final scoring approach.
  5. Binary and Rating scales are the same at the outcomes level since we have to define, even in binary, where the cutoffs are.
  6. The question is how many levels should be on a rating scale. If we approach the question this way, we can run tests and aggregate test results against 2, 3, and 5 point scales (or any other number) to compare the balance between simplicity and flexibility and make a decision.

Summary

ChoiceAll responders
Results
Yes, use a rating scale at the outcome level and test how many levels works well 8
Something else (add to comments) 8

(1 response didn't contain an answer to this question)

Details

Responder One-line description of questionComments
Jennifer Delisi Something else (add to comments) Regarding list item 4 - I'm unclear of why "but not the final scoring approach" is listed here, because I believe this is about the outcome (testable statement) level. Apologies if I am misunderstanding the question.
Regarding list item 6 - I do not feel qualified without reviewing research about rating scales and levels, to answer this question. My assumption is that there is research that has tested different scoring models, in a variety of settings. If levels were selected based on this research, along with a researched-method for writing the scoping for levels (helps testers decide where their score falls when it is more ambiguous), then I would find it easier to vote on this question. If there is research the group has reviewed and is considering while answering this response, it would be helpful if a link could be shared.
Sarah Horton Something else (add to comments) Continue to research, discuss, and prototype approaches.
Stefan Schnabel Yes, use a rating scale at the outcome level and test how many levels works well
Oliver Keim Yes, use a rating scale at the outcome level and test how many levels works well
Jake Abma Something else (add to comments) Continue to research, discuss, and prototype approaches.
Justine Pascalides Yes, use a rating scale at the outcome level and test how many levels works well At a conceptual level, yes.
Makoto Ueki Something else (add to comments) Re: 4 - Percentages can be more repeatable and reliable than any other things to present a more nuanced way of showing how accessible their website, app or product is. There will be a number of guidelines (used to be "SC" in WCAG 2) where we can calculate percentages. If it is not the "final" scoring approach, I'd like to confirm how the final approach in this context would be, before I say "Yes".

And +1 to John's "concerns over subjective determinations". We should check if multiple testers will get the same results/scores at the end. It maybe at later timing, not now. But the results of multiple assessments should be repeatable.
Michael Gower Something else (add to comments) Maybe you don't want any feedback on your list, but here are a few comments:
"Simplicity will most likely come from consistency in how we handle scoring." I'd rather you used "Clarity" than 'simplicity', in almost all cases where you've used the word.
"Transparency adds granularity so increases the number of checkpoints" I'm not sure I understand this, or agree with it :)
"We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this." I'd say "consistent results" rather than "reliability"

This is kind of a radical departure from this, and I apologize for just stating it in a survey question, but I wonder if allowing teams to report on design and dev process (its existence and adherence to) isn't at least as important a metric to capture in regard to pursuing accessibility. The ISO approach, if you will.
Marc Johlic
Jeanne F Spellman Yes, use a rating scale at the outcome level and test how many levels works well We have a group working on testing WCAG3 that can test this. We need more people with testing experience to help so we can get better data.
Alastair Campbell Yes, use a rating scale at the outcome level and test how many levels works well
Andrew Kirkpatrick Something else (add to comments) Without more specific examples I can't answer this concretely. Am concerned about subjectivity (e.g., "process").
Rachael Bradley Montgomery Yes, use a rating scale at the outcome level and test how many levels works well
David MacDonald Yes, use a rating scale at the outcome level and test how many levels works well
Bruce Bailey Yes, use a rating scale at the outcome level and test how many levels works well
Laura Carlson Something else (add to comments) Continue to research, discuss, and prototype approaches.
John Foliot Something else (add to comments) Re: Bullet 4 - I reject the assertion that "percentages do not work across all outcomes so shouldn't be our choice" as pretty much anything can be measured as a percentage. (xx/100)

Re: Bullet 6 - "The question is how many levels should be on a rating scale." - Again, this can be handled by using percentages (for granularity), with minimum percentage values then mapping to Bronze, Silver and Gold.

I also continue to have grave concerns over subjective determinations, as no two experts will always agree, and we're setting up a point of disagreement between evaluators which will be problematic for regulators.

For Question 3 (Scope) the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page".

2. How should we handle scope in 3.0?

WCAG 2 defines conformance against web pages. The first public working draft of WCAG 3 defines scope in terms of views and processes. There have been comments about the need for a better definition of process.

How do you think we should define scope within conformance?

Summary

ChoiceAll responders
Results
Use WCAG 2 scope of web page adjusted for non-html.
Use the WCAG 3 FPWD scope to views and processes and better define both. 15
Something else (add to comments) 2

Details

Responder How should we handle scope in 3.0?
Jennifer Delisi Use the WCAG 3 FPWD scope to views and processes and better define both.
Sarah Horton Use the WCAG 3 FPWD scope to views and processes and better define both.
Stefan Schnabel Use the WCAG 3 FPWD scope to views and processes and better define both.
Oliver Keim Something else (add to comments)
Jake Abma Use the WCAG 3 FPWD scope to views and processes and better define both.
Justine Pascalides Use the WCAG 3 FPWD scope to views and processes and better define both.
Makoto Ueki Use the WCAG 3 FPWD scope to views and processes and better define both.
Michael Gower Use the WCAG 3 FPWD scope to views and processes and better define both.
Marc Johlic Use the WCAG 3 FPWD scope to views and processes and better define both.
Jeanne F Spellman Use the WCAG 3 FPWD scope to views and processes and better define both.
Alastair Campbell Use the WCAG 3 FPWD scope to views and processes and better define both. I think a key thing to also agree is that (like WCAG 2.x) it is up to the person claiming conformance that determines which views/processes to include. And then it's up to regulators (etc) to determine for certain industries/sectors what type of views/processes they must include.
Andrew Kirkpatrick Use the WCAG 3 FPWD scope to views and processes and better define both.
Rachael Bradley Montgomery Use the WCAG 3 FPWD scope to views and processes and better define both.
David MacDonald Something else (add to comments) It seems a little premature, because I think of this question like an exective summary. Lets get further into the actual methods before we decide what will be most appropriate
Bruce Bailey Use the WCAG 3 FPWD scope to views and processes and better define both.
Laura Carlson Use the WCAG 3 FPWD scope to views and processes and better define both.
John Foliot Use the WCAG 3 FPWD scope to views and processes and better define both. the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page".

More details on responses

  • Jennifer Delisi: last responded on 3, May 2021 at 21:42 (UTC)
  • Sarah Horton: last responded on 4, May 2021 at 09:11 (UTC)
  • Stefan Schnabel: last responded on 4, May 2021 at 09:59 (UTC)
  • Oliver Keim: last responded on 4, May 2021 at 10:07 (UTC)
  • Jake Abma: last responded on 4, May 2021 at 11:27 (UTC)
  • Justine Pascalides: last responded on 4, May 2021 at 12:34 (UTC)
  • Makoto Ueki: last responded on 4, May 2021 at 13:44 (UTC)
  • Michael Gower: last responded on 4, May 2021 at 14:24 (UTC)
  • Marc Johlic: last responded on 4, May 2021 at 14:40 (UTC)
  • Jeanne F Spellman: last responded on 4, May 2021 at 14:51 (UTC)
  • Alastair Campbell: last responded on 4, May 2021 at 14:52 (UTC)
  • Andrew Kirkpatrick: last responded on 4, May 2021 at 14:53 (UTC)
  • Rachael Bradley Montgomery: last responded on 4, May 2021 at 14:56 (UTC)
  • David MacDonald: last responded on 4, May 2021 at 15:03 (UTC)
  • Bruce Bailey: last responded on 4, May 2021 at 15:06 (UTC)
  • Laura Carlson: last responded on 4, May 2021 at 15:21 (UTC)
  • John Foliot: last responded on 4, May 2021 at 15:40 (UTC)

Non-responders

The following persons have not answered the questionnaire:

  1. Gregg Vanderheiden
  2. Chris Wilson
  3. Lisa Seeman-Horwitz
  4. Janina Sajka
  5. Shawn Lawton Henry
  6. Katie Haritos-Shea
  7. Shadi Abou-Zahra
  8. Chus Garcia
  9. Steve Faulkner
  10. Patrick Lauke
  11. Gez Lemon
  12. Peter Korn
  13. Preety Kumar
  14. Georgios Grigoriadis
  15. Romain Deltour
  16. Chris Blouch
  17. Jedi Lin
  18. Wilco Fiers
  19. Kimberly Patch
  20. Glenda Sims
  21. Ian Pouncey
  22. Léonie Watson
  23. David Sloan
  24. Mary Jo Mueller
  25. John Kirkwood
  26. Detlev Fischer
  27. Reinaldo Ferraz
  28. Matt Garrish
  29. Mike Gifford
  30. Loïc Martínez Normand
  31. Mike Pluke
  32. Chris Loiselle
  33. Tzviya Siegman
  34. Jan McSorley
  35. Sailesh Panchang
  36. Cristina Mussinelli
  37. Jonathan Avila
  38. John Rochford
  39. Sujasree Kurapati
  40. Jatin Vaishnav
  41. Sam Ogami
  42. Kevin White
  43. E.A. Draffan
  44. Paul Bohman
  45. JaEun Jemma Ku
  46. 骅 杨
  47. Victoria Clark
  48. Avneesh Singh
  49. Mitchell Evan
  50. biao liu
  51. Scott McCormack
  52. Denis Boudreau
  53. Francis Storr
  54. Rick Johnson
  55. David Swallow
  56. Aparna Pasi
  57. Gregorio Pellegrino
  58. Melanie Philipp
  59. Nicole Windmann
  60. Gundula Niemann
  61. Ruoxi Ran
  62. Wendy Reid
  63. Scott O'Hara
  64. Charles Adams
  65. Muhammad Saleem
  66. Amani Ali
  67. Trevor Bostic
  68. Jamie Herrera
  69. Shinya Takami
  70. Karen Herr
  71. Kathy Eng
  72. Cybele Sack
  73. Audrey Maniez
  74. Arthur Soroken
  75. Daniel Bjorge
  76. Kai Recke
  77. David Fazio
  78. Daniel Montalvo
  79. Mario Chacón-Rivas
  80. Michael Gilbert
  81. Caryn Pagel
  82. Achraf Othman
  83. Helen Burge
  84. Fernanda Bonnin
  85. Jared Batterman
  86. Raja Kushalnagar
  87. Jan Williams
  88. Todd Libby
  89. Isabel Holdsworth
  90. Julia Chen
  91. Marcos Franco Murillo
  92. Yutaka Suzuki
  93. Azlan Cuttilan
  94. Jennifer Strickland
  95. Joe Humbert
  96. Ben Tillyer
  97. Charu Pandhi
  98. Poornima Badhan Subramanian
  99. Alain Vagner
  100. Roberto Scano
  101. Rain Breaw Michaels
  102. Kun Zhang
  103. Jaunita George
  104. Regina Sanchez
  105. Shawn Thompson
  106. Thomas Brunet
  107. Kenny Dunsin
  108. Jen Goulden
  109. Mike Beganyi
  110. Ronny Hendriks
  111. Breixo Pastoriza Barcia
  112. Olivia Hogan-Stark
  113. Rashmi Katakwar
  114. Julie Rawe
  115. Duff Johnson
  116. Laura Miller
  117. Will Creedle
  118. Shikha Nikhil Dwivedi
  119. Marie Csanady
  120. Meenakshi Das
  121. Perrin Anto
  122. Stephanie Louraine
  123. Rachele DiTullio
  124. Jan Jaap de Groot
  125. Rebecca Monteleone
  126. Ian Kersey
  127. Peter Bossley
  128. Anastasia Lanz
  129. Michael Keane
  130. Chiara De Martin
  131. Giacomo Petri
  132. Andrew Barakat
  133. Devanshu Chandra
  134. Xiao (Helen) Zhou
  135. Joe Lamyman
  136. Bryan Trogdon
  137. Mary Ann (MJ) Jawili
  138. 禹佳 陶
  139. 锦澄 王
  140. Stephen James
  141. Jay Mullen
  142. Thorsten Katzmann
  143. Tony Holland
  144. Kent Boucher
  145. Abbey Davis
  146. Phil Day
  147. Julia Kim
  148. Michelle Lana
  149. David Williams
  150. Mikayla Thompson
  151. Catherine Droege
  152. James Edwards
  153. Eric Hind
  154. Quintin Balsdon
  155. Mario Batušić
  156. David Cox
  157. Sazzad Mahamud
  158. Katy Brickley
  159. Kimberly Sarabia
  160. Corey Hinshaw
  161. Ashley Firth
  162. Daniel Harper-Wain
  163. Kiara Stewart
  164. DJ Chase
  165. Suji Sreerama
  166. Lori Oakley
  167. David Middleton
  168. Alyssa Priddy
  169. Young Choi
  170. Nichole Bui
  171. Julie Romanowski
  172. Eloisa Guerrero
  173. Daniel Henderson-Ede
  174. George Kuan
  175. YAPING LIN
  176. Justin Wilson
  177. Tiffany Burtin
  178. Shane Dittmar
  179. Nayan Padrai
  180. Niamh Kelly
  181. Matt Argomaniz Matthew Argomaniz
  182. Frankie Wolf
  183. Kimberly McGee
  184. Ahson Rana
  185. Carolina Crespo
  186. humor927 humor927
  187. Samantha McDaniel
  188. Matthäus Rojek
  189. Phong Tony Le
  190. Bram Janssens
  191. Graham Ritchie
  192. Aleksandar Cindrikj
  193. Jeroen Hulscher
  194. Alina Vayntrub
  195. Marco Sabidussi
  196. John Toles
  197. Jeanne Erickson Cooley
  198. Theo Hale
  199. Gert-Jan Vercauteren
  200. Karla Rubiano
  201. Aashutosh K
  202. Hidde de Vries
  203. Julian Kittelson-Aldred
  204. Roland Buss
  205. Aditya Surendranath
  206. Avon Kuo
  207. Elizabeth Patrick
  208. Nat Tarnoff
  209. Filippo Zorzi
  210. Mike Pedersen
  211. Rachael Yomtoob
  212. Oliver Habersetzer

Send an email to all the non-responders.


Compact view of the results / list of email addresses of the responders

WBS home / Questionnaires / WG questionnaires / Answer this questionnaire