Meeting Minutes
Web Evaluation Tools List Update
Brent: We've given comments in a survey - Eric and Vera have taken comments and have a prototype ready for EO review
Vera: We reviewed requirements and wanted to address a couple of newer comments. How to update the list, keep it current, how to get people to maintain accurate information? We are are still looking into the way to make it easy for tool vendors to keep the information current.
… another comment from Carlos is to allow people to review their information before making the final submission
… (shares screen with proposed filters)
Vera: We are looking at the suggestions and are in the process of updating them. First section is called "Purpose" and comments indicated that was perhaps not the right title for the section. We may need to clarify the facts around what can and can't be auto tested.
Shawn: I am putting the link to the list of filters
<shawn> https://
Vera: Next category "Product to evaluate" is a broad category. We found that people come with a goal to evaluate one kind of product. A suggestion to change this to "content" rather than "product" but that is not really accurate either.
Brent: We have time for feedback if you want to accept it.
Sharron: I think "product" is perfectly clear, why the suggestion to change?
Brent: It was my suggestion and now that I hear the explanation, I understand.
Shadi: I wonder Vera to what degree have you looked at the the use cases in the requirements document? I feel like we have a large collection of filters. It may be too much at once. If people are only looking for a specific - like color checker or something? Looking at these categories it may depend on the circumstance of the user. Did you discuss this?
Vera: We want to include a context awareness function that as certain categories are chosen, it will prompt and put forward logical items in the list, kind of a decision treee as the filters are used.
… does that address your comment?
Shadi: Yes thanks
Vera: We will add mobile or desk top to "application"
… Media Type - there was a comment about this as well. We are considering "Contnet Type" as an option. May change Text to Text/Language. Any other suggestions or opinions about this?
Shadi: Is there any overlap between the previous filter and this one?
Vera: Between Product and Media Type?
Shadi: Yes
Vera: There may be some overlap that will have to be considered in the flow but it does not seem significant and will have to be user tested.
Brent: I think this is related to what I had said - what are users hoping to get from search result if they chose one filter over another from these categories?
<shadi> +1 to Brent's phrasing
Vera: The thought behind the it was that you may be looking to evaluate video on a website or audio in an application. It may be too complicated as it is now but that was the thought.
Brent: Would they be tested differently or with different tools?
Vera: I don't know enough about the tools to say for certain, but I believe there is some difference.
Carlos: If you are talking about potential overlaps others that may impact these categories would include scope, metrics, and file format. It may be interesting to see if we can combine some of these into a shorter list. Not to add complexity for the user but maybe some of will only show for certain previous choices.
Vera: We have looked at more of a tree-like system for the categories. So it would become contextual. But if the user has a broader goal it does add complexity and so we need to test the use cases. We have the intention to make some of the categories appear or not depending on previous choices.
… these are valuable comments. We have upcoming user test and will take them into consideration.
Vera: For Paid or Free category, it was previously called Licenses and we were looking for a simpler word but of course there are more choices than only paid or free. If the suggestions to add more detail about licensing then the category title is not as accurate.
Brian: To go back to the previous category, the term IDE seems like a one that will be useful.
Shadi: That one I wasn't sure that we only need IDE but also plugins to source code but not sure what to call them.
Vera: Would the command line cover that?
Shadi: Maybe not entirely.
Brian: Maybe CI integration and be included with the command line
Steve: Is it now getting a bit too detailed. Thinking of Wordpress and other plugins and extensions that don't meet this exact definition.
Vera: From the user interview, the term extension may be too vague and may give them more choices than they are really looking for.
Carlos: On that subject, we probably have two very different types of users. Some may not know these terms, may be less experienced. This will be a tough one to find the middle ground.
Vera: I agree and we will learn more only though testing with many kinds of users. I do appreciate all the different comments and suggestions.
Brent: Back to the question of paid or free - can you start with free or paid and then drill down to the paid details?
Vera: Yes and the details of payments is something that will have to be kept updated.
… I think it could be a good middle ground to start with the paid/free choice and then get the detail as we go further within the tool detail page.
Brian: Paid/free/limited free
Daniel: I am not sure of the term "premium" and what the designation is
Vera: We can investigate further
Daniel: It could be limited by time or by function, etc.
Vera: So could "limited" be the correct term?
Daniel: It could be.
Vera: Another thing for a user test
Brent: It is awesome that you are doing real user feedback.
Vera: A suggestion to add "sample of pages"
… word like to hear your thoughts on "selection of pages" as an option
Steve: subset?
Vera: is that a more difficult word? The reason we steered away from "sample" is that it conveys less intention.
Brent: Isn't "sample" the word used in the WCAG-EM?
Shadi: Yes but it meant for a different audience than this is. I would also steer away fromthe term "element" here since it may be interpreted as an HTML element. To me the first category is what am I testing and this is where am I testing.
Vera: Thanks, this goes back to Carlos's comment about the overlap.
Carlos: I agree that "selection" may be the best choice of but depending on who the users are, "sample" may be a more familiar term.
Carlos: So finally I would argue for sample
Shadi: I agree that the term sample will trigger certain types of use cases but if I am saying "I want to look at my department's pages" that I would not expect sample to be the term, I would expect to select my pages.
Vera: There is an important question for you. Are the ACT rules going to be a part of the Guidelines, do they need to be measured? We don't quite know enough aobut them and rely on your feedback.
Carlos: I would like to make myself available to discuss more about this with you. Seems like we may at this point have a simple question like ACT support? Yes or No.
Vera: I agree it could be a separate category but we would like to help users understand what ACT Rules are and so may add an option to learn more. A conversation will be great - thank you.
… We are planning not to start the filters with the Guidelines and when we do introduce them it will be much less intimidating.
… it was commented that WAI-ARIA is not really a file format but not quite sure about where to put it since we expect people will want to filter for it.
Shadi: Both users and vendors have indicated they want to be able to filter for ARIA test capability. We could not find any better place to put it.
… there are such tools, there is a use case, not sure where it goes.
Brian: But the ARIA attributes would not exist without HTML> Maybe something that the vendor can fill out and then in the tool it would show up within the HTML?
Vera: Would it work that users would search for the term specifically rather expect to see it as filter?
Carlos: +1 to what Brian just said that ARIA would be checked under HTML. We could expand Guideline to include Specifications and it could go there.
Vera: The best option to go forward is to remove it as a filter and find it within the HTML tools.
Daniel: I do see a use case for those testing with manual tools and screen readers who may wish for a tool that tests the ARIA.
<shadi> +1 to Daniel
Vera: Where would it fit best and would those users look for it in the search?
Daniel: I support the idea of adding Specification to the filter category title.
… and adding ARIA there.
Vera: About the accessibility of the tools - there were many questions about accessible to what level, to whom, who checks to validate claims?
… is this feasible?
Shadi: We went exactly through the decision cycle on the previous tool and opted for the question of an accessiiblity statement.
Jade: Does this need to be in the filter options of can it simply be within the detailed product page.
<BrianE> +1 to Shadi re: does the vendor provide an ACR/VPAT
Vera: Some users stated that they would greatly benefit from only having to review tools that they could actually use so we put it as a filter. I think Shadi's idea of the accessibility statement may work though it strays a bit from what our users asked for. As well, even if we put it only on the product detail page, it would still need validation.
Shadi: For just those reasons, we ask for a URL to the Accessibility Statement. Once we filtered for it there was an increase in those who did have one, so there was a benefit.
Vera: It will require quite some work to validate the claims.
Shadi: While we do look at the page we don't validate all the claims and there are other claims that we do not validate as well. This is why we have the disclaimer about endorsement.
Shawn: From a point of order, I don't think your user study will help with this. I consider this issue is addressed and that we continue with the decision we made before and not spend more time on it.
<EricV> +1
Vera: So we will implement it as a question of the Accessibility Statement or not?
Brent: I am good with the proposal, it seemed to work well before
<Sharron> +1 to making the Accessibility Statement Yes or No and part of the filters
KrisAnne: I agree and if they say No, provide a link to where they are shown how to make one.
Vera: And provide that exta information in the wizard to help people become more aware.
Shawn: As the submission form is completed, if the vendor says they have a Statement, we need to require a link to it.
Vera: The OS and browser categories are also context aware and would move up in the list of you choose browser extension, etc
… language of tool. One comment is unclear to me.
Sylvie: If I want to choose a tool in only one language, can I filter for that?
Brent: The submission will ask for every language the tool is available in so if I ask for only Chinese...
Sylvie: Is the filter for those looking or submitting the tool?
Vera: for those looking for a tool.
Shawn: I think the confusion will be addressed when the Language choices are listed. It will be more clear that you can filter for a specific language.
Vera: Metrics is also a complex category since we need to let people know what is/is not able to be auto tested with confidence. To allow people to quickly filter for tools that do that specific task.
… (review of current prototype) https://
Shadi: Kudos, wizard is a great idea, difficult - good luck!
Vera: Can sort by alpha order, most recently updated, other ways to organize the filters.
KrisAnne: The very first time people see that when people see the last update was 2014 it would not give confidence.
Brent: Would it be useful to have the newest tool at the top?
Vera: We see a lot of these tools that have A as the first letter to be at the top of the list. You could say the same thing about using the updates that way.
<EricV> If they update every month to be on top of the page, that would be a great thing...
Shadi: At the top of the page maybe add most recently updated, most recently added to show that sorting option more obviously
Vera: We'll look into that
Eric: Can add a surprise me button
Shawn: Or randomize
Vera: a shuffle
… what do people think about filter results being randomized?
… we will consider that as well.
<Vicki> Awesome work!
<howard> looks great!
Brent: If there are no further questions or comments, I will just say WOW and reiterate that we are so pleased with how you are conducting the research and the work, great job!
Vera: Thank you for the kind words, I will share with the team, it was helpful to get this direct feedback.
Supplemental Guidance
<shawn> https://
Shawn: Please look at the update and navigate to the different pieces and let us knwo what you think
KrisAnne: Seemed that the fact that SG is not required has been phrased differently.
Shawn: It says Beyond but does not say "is not required" we will tweak the wording to make it clearer
Daniel: This phrase shows up as the last thing in the navigation region
Steve: It is different than the experience of a sighted user
Daniel: We can follow up off line
Laura: My question is about how we get to this page, where is it linked from?
Shawn: From All WCAG Guidance and other relevant pages
Laura: From WCAG itself?
Shawn: No but the WCAG links to an overview page for additional resources and that page links to this
Shawn: We will look at where else we may link from
Jade: In the introduction you mention the objectives which are linked from within the COGA doc but are not linked within this document. That could be helpful.
… going back to the original, I find links with numbers
Steve: Seemed like noise within an interactive document.
Jade: But the mention in the intro does not clearly refer to the objectives.
Steve: Ah I see what you mena, will address that.
<CarlosD> +1
<jade> +1
<Vicki> +1
<Laura> +1
<Leticia> +1
<stevelee> +0
<howard> +1
<brent> 0 can take it or leave it if navigation is there.
<krisannekinney> +1
<Sylvie> 0
<BrianE> 0
<Michele> -1
<EricV> +0
<krisannekinney> maybe 0 actually
Wrap Up
Brent: Taking a survey for face to face meetings, linked in work for this week - some possibilities are AccessU in May, TPAC in September
https://
Brent: Great to have the focused time to work on resources. We're over time, see you next week.