18:04:50 RRSAgent has joined #aria-at 18:04:54 logging to https://www.w3.org/2024/11/20-aria-at-irc 18:04:54 RRSAgent, make logs Public 18:04:55 please title this meeting ("meeting: ..."), jugglinmike 18:04:55 howard-e has joined #aria-at 18:05:10 Meeting: ARIA and Assistive Technologies Community Group Weekly Teleconference 18:05:11 Matt_King_ has joined #aria-at 18:05:34 present+ 18:05:44 present+ jugglinmike 18:05:46 scribe+ jugglinmike 18:05:50 present+ howard-e 18:05:52 present+ IsaDC 18:05:57 present+ james 18:06:01 present+ Joe_Humbert 18:06:46 Matt_King_: I didn't make an agenda because it looks like there are still quite a few unresolved conflicts in the test queue, and I'd like to focus on getting those resolved today 18:07:14 Matt_King_: We have other stuff waiting to add to the test queue, so we'll talk about that after 18:07:48 Matt_King_: Is there anything else that should be on the agenda? 18:08:05 Joe_Humbert: I reported a few issues. They are related to the test plans, but they're higher-level 18:08:32 s/the test plans/one of the test plans/ 18:08:36 Joe_Humbert: I don't know if they impact other test plans 18:11:31 Matt_King_: Got it. We'll add that to the agenda 18:11:55 Joe_Humbert: Also, Testers can no longer update the version of the software that they use. When I went to test, it was locked to the version that the bot used 18:12:25 Joe_Humbert: I did not raise an issue for this 18:13:00 Matt_King_: I noticed that when I opened a Safari test plan on Windows, it didn't even tell me I was using the wrong browser 18:13:24 Matt_King_: I think there used to be a button on the test page where you could change the version information about the browser, anyway... And I guess the AT, as well 18:13:29 Matt_King_: Do you mind raising an issue for that? 18:13:31 Joe_Humbert: Sure 18:13:43 Matt_King_: Thanks, then we can get that tracked and get it into the right place 18:14:34 Topic: Bot issues 18:17:11 subtopic: VoiceOver bot providing incorrect output when testing Action Menu Button Example Using element.focus(), Test 1, V24.10.31) 18:17:15 github: https://github.com/w3c/aria-at/issues/1160 18:17:37 Matt_King_: This is the issue Joe_Humbert was referencing just now. I've renamed it to be focused on the bot 18:18:18 Joe_Humbert: I have updated the AT response reported by the bot, is that a problem? 18:18:50 howard-e: That's fine. Whatever we need, we can find in the logs 18:19:11 IsaDC: I have NOT updated the output reported by the bot for the test plan run assigned to me 18:19:52 Joe_Humbert: This is limited to the VoiceOver Bot. The AT responses reported by the NVDA Bot have been accurate 18:21:27 https://github.com/w3c/aria-at/issues/1163 for can't change AT or browser version 18:21:58 Matt_King_: We use different OS APIs for different screen readers. I'm wondering at which layer these problems exist, and if it's the screen reader, if that could make it hard to solve the problem this way 18:22:26 Topic: Testing status 18:23:15 Matt_King_: We have 8 plans in candidate review, 3 plans in draft review, and 2 plans ready to be added to the queue (I just don't want to add more to the queue until we're a little more caught up with the testing) 18:23:22 IsaDC: I agree on waiting 18:23:57 Matt_King_: This morning, I spent my time on conflicts rather than the slider test plan. We'll get it into the "ready to test" soon 18:24:21 Matt_King_: The question right now is: how far can we get these three waiting in the queue out of the queue? 18:25:16 Topic: Action menu button with element.focus 18:25:30 IsaDC: All the testers are here 18:25:37 IsaDC: I'm planning on finishing my testing by the end of this week 18:25:47 IsaDC: Luke from PAC told me that he will be testing during this week 18:25:54 IsaDC: And Joe_Humbert already finished 18:26:00 Matt_King_: So we could get this wrapped up this week 18:26:13 Matt_King_: I suppose that if there are any conflicts, you and Luke can work it out this week, IsaDC 18:26:20 IsaDC: Yes, and I can publish once it's ready 18:26:28 Matt_King_: Cool, that's simple. Thank you, IsaDC 18:26:33 Topic: Disclosure 18:26:48 Matt_King_: We have two screen readers totally done, and we're 85% complete with VoiceOver 18:26:59 IsaDC: The conflicts reported are the same across test plans 18:27:19 IsaDC: I'm going to re-run my tests because it seems to be an issue with the previous version of either Safari or VoiceOver 18:27:33 IsaDC: If I get the same results, I'm going to ask somebody else to run them, as well 18:27:47 Matt_King_: Dean, what version of macOS were you running when you got your results? 18:27:50 present+ Dean 18:28:07 Dean: I was on 14.5, as I recall, which I believe is what the test specifies we were supposed to be using 18:28:44 Dean: I did upgrade. I don't have a comparison of the same test before and after the upgrade, but my experience matches IsaDC. It looks like the problem was addressed 18:28:57 Dean: I got through everything but three tests on the disclosure yesterday 18:29:18 Matt_King_: Last week, we were talking about test 14. Now, I see there are no conflicts in test 14. 18:29:30 Matt_King_: It was a case where some folks were getting output and other folks were not 18:30:06 Matt_King_: How did that get resolved? I was looking at that issue before the meeting, and the issue was still open. When I look at the conflicts page, there is no conflict for test 14 18:30:51 IsaDC: I think it got fixed with the latest version, as well 18:31:11 Matt_King_: I don't know if it's fixed. It says that there's no output. 18:31:21 IsaDC: Ah, not "fixed", but there is no longer a conflict 18:31:33 Matt_King_: The AT responses are now consistently bad 18:31:38 Dean: Correct 18:32:18 Dean: I didn't re-test test number 14, though. I had done that before I upgrade. If I need to go back and run test 14 again, I will. Perhaps the issue is that the new version is fixed and my results from before are not up-to-date 18:32:29 Dean: I can run test 14 right now as an example... 18:32:54 Dean: But I saw several issues with bot. I did kind of destroy some evidence as I went along because I changed what the Bot reported to reflect what I observed locally 18:34:13 Matt_King_: Did you report that as an issue? 18:34:39 Dean: No. This is something I assumed last week and thought that I was cracked. 18:34:54 Dean: I will go back to test 4, raise an issue, paste in what the bot said and what I observed locally 18:35:38 jugglinmike: Along the lines of Joe_Humbert's report, could you re-title that to make it clear that the problem is likely with the bot rather than with the test itself? 18:35:41 Dean: Will do 18:35:49 Dean: We have conflicts on tests 3, 4, and 9 18:36:43 IsaDC: I have upgraded, and I will be changing my results. That doesn't guarantee that the conflicts will go away. Something tells me that I will be running the whole test plan again 18:37:17 Matt_King_: I think it would be unrealistic to expect that we will get this all resolved asynchronously. We'll likely be talking about this in the next meeting on December 4 18:37:24 Dean: I will have those done before tomorrow 18:39:42 Dean: I think what IsaDC reported initially--I think she probably got the correct test results and reported them correctly. I think Apple fixed something in between versions. I expect that she will observe different results without conflicts when she re-runs them 18:40:17 Matt_King_: I think that's it for this test plan. That's all we can talk about and resolve at this moment. There's just some more work to do, here 18:40:22 Topic: Navigation menu button 18:40:27 IsaDC: This is the one with the hints 18:40:43 Matt_King_: Right, but last week, we decided on a path forward 18:40:57 Matt_King_: We have 10 conflicts with VoiceOver, and we're not totally done with JAWS yet, either 18:41:15 Matt_King_: Hadi has two tests done. He is not present today 18:41:26 IsaDC: I have completely finished this with JAWS 18:41:41 Matt_King_: I didn't look at THESE conflicts this morning 18:41:48 IsaDC: Do we have conflicts? Oh, dear 18:42:32 Matt_King_: Hadi is recording the JAWS tutor message, and you didn't record the JAWS tutor message 18:42:47 Matt_King_: That shouldn't do it, I guess... 18:43:50 Matt_King_: Hadi has rendered a passing verdict for the assertion regarding the "collapsed" state, but that information is definitely not in the AT response he reported 18:44:20 IsaDC: I have noticed that with JAWS, sometimes it's easy to tick the wrong radio button. That has happened to me right after we switched from radio buttons to check boxes 18:44:33 Matt_King_: I found it easier with the radio buttons, myself, because it clearly says "yes" or "no" 18:44:40 IsaDC: Sometimes JAWS checks the wrong one 18:45:33 james: I have some concerns about that UI change. I know that ultimately, we don't have a binary state. But check buttons seem more prone to error 18:45:40 s/buttons/boxes/ 18:46:04 Matt_King_: Actually, I feel that radio buttons are more prone to error 18:47:02 s/seem more/seem less/ 18:47:23 s/are more/are less/ 18:48:06 Matt 18:48:37 s/Matt/Matt_King_: If Luke is available for this test plan, that's potentially really helpful. If Hadi isn't able to wrap it up before Thanksgiving 18:48:48 Matt_King_: I want to get this out there to collect feedback from Apple 18:49:12 Matt_King_: IsaDC, could you ask Hadi if he'll be able to get this done before Thanksgiving? If not, could you ask Luke to take over? 18:49:21 Matt_King_: So that's JAWS. Now, for VoiceOver 18:49:33 Matt_King_: There are 10 conflicts. Are all of these conflicts related to the hints? 18:49:37 IsaDC: They are 18:49:43 Matt_King_: Then we can fix this up 18:50:02 Matt_King_: Dean, are you okay if IsaDC and I edit your results to be consistent with our discussion last week? 18:50:18 Matt_King_: That is to say: it doesn't pass if it's only conveyed in this hint 18:50:26 Dean: I didn't know that we arrived at a decision 18:51:06 Matt_King_: Yeah, we arrived at it last week, and I took an action item to document the resolution. We will record the hints, but we will not consider the content of hints when rendering verdicts 18:51:15 Dean: I'm cool with that. Change away 18:51:20 Matt_King_: Okay 18:51:42 Matt_King_: There is one part of the rationale that we discussed last week which is slightly murky... 18:52:31 Matt_King_: In the rationale, we said that we can't consider it part of the assertion verdict because we said that if hints are turned off (which they often are in screen readers), then the required part of the base output will might not be present in the base output 18:53:13 Matt_King_: What if the assertion is optional? Then it really doesn't matter if the hint is on or off 18:53:27 Matt_King_: If we have an optional assertion, and it is only included as part of the hint? 18:53:57 Matt_King_: For consistency, we would say it is "not supported", but in that case, the rationale we used for not considering hint test doesn't really hold water 18:54:06 Matt_King_: Let me restate 18:54:24 Matt_King_: We have a reason for why we're not considering the hint when arriving at a verdict 18:54:48 Matt_King_: That reason is that even though hints are on by default, we know that (for certain screen readers), large portions of the user base turns them off 18:55:27 Matt_King_: So we're saying that the hint cannot be part of the "base output" (because people frequently turn hints off), so the output that is required for interoperability would not be available for most users 18:55:49 Matt_King_: We're saying that those users who turn off hints are not getting something that is required for interoperability 18:56:17 Matt_King_: However, any assertion that is a "may" is not required for interoperability. So it seems as though we shouldn't care about whether the hint is on or off 18:56:59 Matt_King_: The rationale for ignoring hints doesn't really hold water. The only reason we would ignore the hint for an optional assertion is to preserve consistency 18:57:47 james: If we say that "may" assertions can take hint output into account, then testers need to take assertion priority into account. We don't ask them to do that right now 18:58:30 james: I think this risks the human nature of "assuming a given assertion is a 'may' because it has been a 'may' so many other times" even when it changes for one edge case 18:58:58 Matt_King_: Yes, that's a good point. We shouldn't require Testers to consider assertion priority for exactly that reason. 18:59:46 Matt_King_: I definitely need to document this. I'm amazed at the number of decisions we've made; it can be difficult to remember the rationale for all the things we've decided. Documentation is really important, and I'm taking action items to keep it up-to-date. 18:59:59 Matt_King_: So, disclosure is going to be sitting with us for at least one more meeting 19:00:25 Matt_King_: For the other two plans, we at least have a path forward. I will be in conversation with IsaDC to move them along 19:00:48 Matt_King_: For the meeting on December 4, we'll have a bunch of stuff for people to do. We'll also hopefully be putting the "disclosure navigation menu" plan to bed 19:01:05 IsaDC: I can't believe it's almost December, already 19:01:07 Matt_King_: Same here! 19:01:34 Matt_King_: That's it for today, everyone. Thank you so much for your dedication to the project! 19:01:39 Zakim, end the meeting 19:01:39 As of this point the attendees have been Matt_King_, jugglinmike, howard-e, IsaDC, james, Joe_Humbert, Dean 19:01:41 RRSAgent, please draft minutes 19:01:42 I have made the request to generate https://www.w3.org/2024/11/20-aria-at-minutes.html Zakim 19:01:49 I am happy to have been of service, jugglinmike; please remember to excuse RRSAgent. Goodbye 19:01:49 Zakim has left #aria-at 22:37:52 jongund has joined #aria-at 23:08:21 jongund has joined #aria-at 23:35:29 jongund has joined #aria-at 23:43:06 jongund has joined #aria-at