W3C

Results of Questionnaire Follow On from 29 April Face to Face Meeting

The results of this questionnaire are available to anybody.

This questionnaire was open from 2021-04-29 to 2021-05-04.

17 answers have been received.

Jump to results for question:

  1. One-line description of question
  2. How should we handle scope in 3.0?

1. One-line description of question

Based on the following discussion points from the meeting, can we agree to use a rating scale (adjectival rating) at the outcome (testable statement) level?

  1. Simplicity and transparency matter especially to regulatory and legal specialists.
  2. Transparency adds granularity so increases the number of checkpoints. Simplicity will most likely come from consistency in how we handle scoring. We should ideally pick a single approach to scoring outcomes.
  3. We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this.
  4. Assuming we want a single approach, percentages do not work across all outcomes so shouldn't be our choice. Percentages can be a threshold on some rating scales but not the final scoring approach.
  5. Binary and Rating scales are the same at the outcomes level since we have to define, even in binary, where the cutoffs are.
  6. The question is how many levels should be on a rating scale. If we approach the question this way, we can run tests and aggregate test results against 2, 3, and 5 point scales (or any other number) to compare the balance between simplicity and flexibility and make a decision.

Summary

ChoiceAll responders
Results
Yes, use a rating scale at the outcome level and test how many levels works well 8
Something else (add to comments) 8

(1 response didn't contain an answer to this question)

Details

Responder One-line description of questionComments
Jennifer Delisi Something else (add to comments) Regarding list item 4 - I'm unclear of why "but not the final scoring approach" is listed here, because I believe this is about the outcome (testable statement) level. Apologies if I am misunderstanding the question.
Regarding list item 6 - I do not feel qualified without reviewing research about rating scales and levels, to answer this question. My assumption is that there is research that has tested different scoring models, in a variety of settings. If levels were selected based on this research, along with a researched-method for writing the scoping for levels (helps testers decide where their score falls when it is more ambiguous), then I would find it easier to vote on this question. If there is research the group has reviewed and is considering while answering this response, it would be helpful if a link could be shared.
Sarah Horton Something else (add to comments) Continue to research, discuss, and prototype approaches.
Stefan Schnabel Yes, use a rating scale at the outcome level and test how many levels works well
Oliver Keim Yes, use a rating scale at the outcome level and test how many levels works well
Jake Abma Something else (add to comments) Continue to research, discuss, and prototype approaches.
Justine Pascalides Yes, use a rating scale at the outcome level and test how many levels works well At a conceptual level, yes.
Makoto Ueki Something else (add to comments) Re: 4 - Percentages can be more repeatable and reliable than any other things to present a more nuanced way of showing how accessible their website, app or product is. There will be a number of guidelines (used to be "SC" in WCAG 2) where we can calculate percentages. If it is not the "final" scoring approach, I'd like to confirm how the final approach in this context would be, before I say "Yes".

And +1 to John's "concerns over subjective determinations". We should check if multiple testers will get the same results/scores at the end. It maybe at later timing, not now. But the results of multiple assessments should be repeatable.
Michael Gower Something else (add to comments) Maybe you don't want any feedback on your list, but here are a few comments:
"Simplicity will most likely come from consistency in how we handle scoring." I'd rather you used "Clarity" than 'simplicity', in almost all cases where you've used the word.
"Transparency adds granularity so increases the number of checkpoints" I'm not sure I understand this, or agree with it :)
"We resolved that for WCAG 3, testing will aim to improve reliability between testers (from WCAG 2.x) and that we will work on testing to measure this." I'd say "consistent results" rather than "reliability"

This is kind of a radical departure from this, and I apologize for just stating it in a survey question, but I wonder if allowing teams to report on design and dev process (its existence and adherence to) isn't at least as important a metric to capture in regard to pursuing accessibility. The ISO approach, if you will.
Marc Johlic
Jeanne F Spellman Yes, use a rating scale at the outcome level and test how many levels works well We have a group working on testing WCAG3 that can test this. We need more people with testing experience to help so we can get better data.
Alastair Campbell Yes, use a rating scale at the outcome level and test how many levels works well
Andrew Kirkpatrick Something else (add to comments) Without more specific examples I can't answer this concretely. Am concerned about subjectivity (e.g., "process").
Rachael Bradley Montgomery Yes, use a rating scale at the outcome level and test how many levels works well
David MacDonald Yes, use a rating scale at the outcome level and test how many levels works well
Bruce Bailey Yes, use a rating scale at the outcome level and test how many levels works well
Laura Carlson Something else (add to comments) Continue to research, discuss, and prototype approaches.
John Foliot Something else (add to comments) Re: Bullet 4 - I reject the assertion that "percentages do not work across all outcomes so shouldn't be our choice" as pretty much anything can be measured as a percentage. (xx/100)

Re: Bullet 6 - "The question is how many levels should be on a rating scale." - Again, this can be handled by using percentages (for granularity), with minimum percentage values then mapping to Bronze, Silver and Gold.

I also continue to have grave concerns over subjective determinations, as no two experts will always agree, and we're setting up a point of disagreement between evaluators which will be problematic for regulators.

For Question 3 (Scope) the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page".

2. How should we handle scope in 3.0?

WCAG 2 defines conformance against web pages. The first public working draft of WCAG 3 defines scope in terms of views and processes. There have been comments about the need for a better definition of process.

How do you think we should define scope within conformance?

Summary

ChoiceAll responders
Results
Use WCAG 2 scope of web page adjusted for non-html.
Use the WCAG 3 FPWD scope to views and processes and better define both. 15
Something else (add to comments) 2

Details

Responder How should we handle scope in 3.0?
Jennifer Delisi Use the WCAG 3 FPWD scope to views and processes and better define both.
Sarah Horton Use the WCAG 3 FPWD scope to views and processes and better define both.
Stefan Schnabel Use the WCAG 3 FPWD scope to views and processes and better define both.
Oliver Keim Something else (add to comments)
Jake Abma Use the WCAG 3 FPWD scope to views and processes and better define both.
Justine Pascalides Use the WCAG 3 FPWD scope to views and processes and better define both.
Makoto Ueki Use the WCAG 3 FPWD scope to views and processes and better define both.
Michael Gower Use the WCAG 3 FPWD scope to views and processes and better define both.
Marc Johlic Use the WCAG 3 FPWD scope to views and processes and better define both.
Jeanne F Spellman Use the WCAG 3 FPWD scope to views and processes and better define both.
Alastair Campbell Use the WCAG 3 FPWD scope to views and processes and better define both. I think a key thing to also agree is that (like WCAG 2.x) it is up to the person claiming conformance that determines which views/processes to include. And then it's up to regulators (etc) to determine for certain industries/sectors what type of views/processes they must include.
Andrew Kirkpatrick Use the WCAG 3 FPWD scope to views and processes and better define both.
Rachael Bradley Montgomery Use the WCAG 3 FPWD scope to views and processes and better define both.
David MacDonald Something else (add to comments) It seems a little premature, because I think of this question like an exective summary. Lets get further into the actual methods before we decide what will be most appropriate
Bruce Bailey Use the WCAG 3 FPWD scope to views and processes and better define both.
Laura Carlson Use the WCAG 3 FPWD scope to views and processes and better define both.
John Foliot Use the WCAG 3 FPWD scope to views and processes and better define both. the term "Process" requires significantly more clarity: clicking on a link or button will always initiate a process, for example, a flyout menu with 35 menu items = 35 potential process 'starts' (i.e.the start of a process) which is "The user wants to go to a new page".

More details on responses

  • Jennifer Delisi: last responded on 3, May 2021 at 21:42 (UTC)
  • Sarah Horton: last responded on 4, May 2021 at 09:11 (UTC)
  • Stefan Schnabel: last responded on 4, May 2021 at 09:59 (UTC)
  • Oliver Keim: last responded on 4, May 2021 at 10:07 (UTC)
  • Jake Abma: last responded on 4, May 2021 at 11:27 (UTC)
  • Justine Pascalides: last responded on 4, May 2021 at 12:34 (UTC)
  • Makoto Ueki: last responded on 4, May 2021 at 13:44 (UTC)
  • Michael Gower: last responded on 4, May 2021 at 14:24 (UTC)
  • Marc Johlic: last responded on 4, May 2021 at 14:40 (UTC)
  • Jeanne F Spellman: last responded on 4, May 2021 at 14:51 (UTC)
  • Alastair Campbell: last responded on 4, May 2021 at 14:52 (UTC)
  • Andrew Kirkpatrick: last responded on 4, May 2021 at 14:53 (UTC)
  • Rachael Bradley Montgomery: last responded on 4, May 2021 at 14:56 (UTC)
  • David MacDonald: last responded on 4, May 2021 at 15:03 (UTC)
  • Bruce Bailey: last responded on 4, May 2021 at 15:06 (UTC)
  • Laura Carlson: last responded on 4, May 2021 at 15:21 (UTC)
  • John Foliot: last responded on 4, May 2021 at 15:40 (UTC)

Non-responders

The following persons have not answered the questionnaire:

  1. Gregg Vanderheiden
  2. Chris Wilson
  3. Lisa Seeman-Horwitz
  4. Janina Sajka
  5. Shawn Lawton Henry
  6. Katie Haritos-Shea
  7. Shadi Abou-Zahra
  8. Chus Garcia
  9. Steve Faulkner
  10. Patrick Lauke
  11. Gez Lemon
  12. Peter Korn
  13. Preety Kumar
  14. Georgios Grigoriadis
  15. Romain Deltour
  16. Chris Blouch
  17. Jedi Lin
  18. Wilco Fiers
  19. Kimberly Patch
  20. Glenda Sims
  21. Ian Pouncey
  22. Léonie Watson
  23. David Sloan
  24. Mary Jo Mueller
  25. Peter Heery
  26. John Kirkwood
  27. Detlev Fischer
  28. Reinaldo Ferraz
  29. Matt Garrish
  30. Mike Gifford
  31. Loïc Martínez Normand
  32. Mike Pluke
  33. Jon Gibbins
  34. Chris Loiselle
  35. Tzviya Siegman
  36. Jan McSorley
  37. Sailesh Panchang
  38. Cristina Mussinelli
  39. Jonathan Avila
  40. John Rochford
  41. Sujasree Kurapati
  42. Jatin Vaishnav
  43. Sam Ogami
  44. Kevin White
  45. E.A. Draffan
  46. Paul Bohman
  47. JaEun Jemma Ku
  48. 骅 杨
  49. Victoria Clark
  50. Avneesh Singh
  51. Mitchell Evan
  52. biao liu
  53. Scott McCormack
  54. Francis Storr
  55. David Swallow
  56. Aparna Pasi
  57. Gregorio Pellegrino
  58. Melanie Philipp
  59. Nicole Windmann
  60. Gundula Niemann
  61. Ruoxi Ran
  62. Wendy Reid
  63. Scott O'Hara
  64. Charles Adams
  65. Muhammad Saleem
  66. Amani Ali
  67. Trevor Bostic
  68. Jamie Herrera
  69. Shinya Takami
  70. Karen Herr
  71. Kathy Eng
  72. Cybele Sack
  73. Audrey Maniez
  74. Arthur Soroken
  75. Daniel Bjorge
  76. Kai Recke
  77. David Fazio
  78. Daniel Montalvo
  79. Mario Chacón-Rivas
  80. Michael Gilbert
  81. Caryn Pagel
  82. Achraf Othman
  83. Helen Burge
  84. Fernanda Bonnin
  85. Christina Adams
  86. Raja Kushalnagar
  87. Jan Williams
  88. Todd Libby
  89. Isabel Holdsworth
  90. Julia Chen
  91. Marcos Franco Murillo
  92. Yutaka Suzuki
  93. Azlan Cuttilan
  94. Jennifer Strickland
  95. Joe Humbert
  96. Ben Tillyer
  97. Charu Pandhi
  98. Poornima Badhan Subramanian
  99. Alain Vagner
  100. Roberto Scano
  101. Rain Breaw Michaels
  102. Kun Zhang
  103. Jaunita George
  104. Regina Sanchez
  105. Shawn Thompson
  106. Thomas Brunet
  107. Kenny Dunsin
  108. Jen Goulden
  109. Mike Beganyi
  110. Ronny Hendriks
  111. Korede Olubowale
  112. Rashmi Katakwar
  113. Julie Rawe
  114. Duff Johnson
  115. Laura Miller
  116. Will Creedle
  117. Shikha Nikhil Dwivedi
  118. Marie Csanady
  119. Meenakshi Das
  120. Perrin Anto
  121. Brian Elton
  122. Rachele DiTullio
  123. Jan Jaap de Groot
  124. Rebecca Monteleone
  125. Ian Kersey
  126. Peter Bossley
  127. Michael Keane
  128. Chiara De Martin
  129. Giacomo Petri
  130. Andrew Barakat
  131. Devanshu Chandra
  132. Xiao (Helen) Zhou
  133. Joe Lamyman
  134. Bryan Trogdon
  135. Mary Ann (MJ) Jawili
  136. 禹佳 陶
  137. 锦澄 王
  138. Stephen James
  139. Jay Mullen
  140. Thorsten Katzmann
  141. Tony Holland
  142. Kent Boucher
  143. Phil Day
  144. Julia Kim
  145. Michelle Lana
  146. David Williams
  147. Mikayla Thompson
  148. Catherine Droege
  149. James Edwards
  150. Eric Hind
  151. Quintin Balsdon
  152. Mario Batušić
  153. David Cox
  154. Sazzad Mahamud
  155. Katy Brickley
  156. Kimberly Sarabia
  157. Corey Hinshaw
  158. Ashley Firth
  159. Daniel Harper-Wain
  160. Kiara Stewart
  161. DJ Chase
  162. Suji Sreerama
  163. Fred Edora
  164. Lori Oakley
  165. David Middleton
  166. Alyssa Priddy
  167. Young Choi
  168. Nichole Bui
  169. Julie Romanowski
  170. Eloisa Guerrero
  171. George Kuan
  172. YAPING LIN
  173. Justin Wilson
  174. Leonard Beasley
  175. Tiffany Burtin
  176. Shane Dittmar
  177. Nayan Padrai
  178. Matt Argomaniz Matthew Argomaniz
  179. Frankie Wolf
  180. Kimberly McGee
  181. Ahson Rana
  182. Carolina Crespo
  183. humor927 humor927
  184. Jackie Fei
  185. Samantha McDaniel
  186. Matthäus Rojek
  187. Phong Tony Le
  188. Bram Janssens
  189. Graham Ritchie
  190. Aleksandar Cindrikj
  191. Jeroen Hulscher
  192. Alina Vayntrub
  193. Marco Sabidussi
  194. John Toles
  195. Jeanne Erickson Cooley
  196. Theo Hale
  197. Paul Adam
  198. Gert-Jan Vercauteren
  199. Karla Rubiano
  200. Aashutosh K
  201. Hidde de Vries
  202. Julian Kittelson-Aldred
  203. Roland Buss
  204. Aditya Surendranath
  205. Elizabeth Patrick
  206. Tj Squires
  207. Nat Tarnoff
  208. Illai Zeevi
  209. Filippo Zorzi
  210. Gleidson Ramos
  211. Mike Pedersen
  212. Rachael Yomtoob
  213. Oliver Habersetzer
  214. Ken Franqueiro
  215. Irfan Mukhtar
  216. Rachel White
  217. Sage Keriazes
  218. Tananda Darling
  219. Nina Krauß
  220. Demelza Feltham
  221. Ragvesh Sharma
  222. Shunguo Yan
  223. Charli Riggle
  224. Nora GOUGANE
  225. Andy Manea
  226. Tim Gravemaker
  227. Roldon Brown
  228. qin guan
  229. Alexandra Yaneva
  230. Carrie Hall
  231. Tanya van Workum
  232. Megan Pletzer
  233. Akash Shukla
  234. Rob Whitaker
  235. Jeremy Katherman
  236. Atya Ratcliff
  237. Nati Elimelech
  238. Noa Nitzan
  239. Jory Cunningham
  240. Imran Ahmed

Send an email to all the non-responders.


Compact view of the results / list of email addresses of the responders

WBS home / Questionnaires / WG questionnaires / Answer this questionnaire