07:32:13 RRSAgent has joined #webdriver 07:32:18 logging to https://www.w3.org/2023/09/14-webdriver-irc 07:32:23 AutomatedTester_ has joined #webdriver 07:32:52 present+ 07:32:57 present+ 07:33:52 present+ 07:33:59 present+ 07:34:12 Meeting: Browser Tools & Testing @ TPAC 2023 - Day 1 07:34:26 RRSAgent: make logs public 07:34:35 RRSAgent: make minutes 07:34:36 I have made the request to generate https://www.w3.org/2023/09/14-webdriver-minutes.html jgraham_ 07:36:01 shs96c has joined #webdriver 07:36:03 Chair: David Burns 07:36:07 present+ 07:36:15 present+ 07:37:00 Topic: Agenda wrangling 07:38:41 wfilipek has joined #webdriver 07:39:07 when exactly we will have the breaks today? 07:39:22 is there a link to a schedule that we might want to follow? 07:39:31 present+ 07:39:31 present+ 07:39:38 lola_ has joined #webdriver 07:39:40 present+ 07:39:47 Breaks are 11-11:30 CEST and 16:30-17:00 07:39:56 Lunch is 13:00-14:30 07:40:00 present+ 07:40:20 https://www.w3.org/2023/09/TPAC/schedule.html#thursday 07:41:25 present+ 07:41:30 scribe: David Burns 07:41:36 scribenick: AutomatedTester 07:42:21 RRSAgent: make minutes 07:42:22 I have made the request to generate https://www.w3.org/2023/09/14-webdriver-minutes.html jgraham_ 07:42:50 q? 07:43:56 Topic: Resize and positioning windows 07:44:01 ali_spivak_ has joined #webdriver 07:44:09 Github: https://github.com/w3c/webdriver-bidi/issues/398 07:44:52 whimboo: I've added this and was curious if this is a high level priority from client vendors 07:45:25 jgraham: Classic webdriver has support for resizing and positioning. From the point of view of supporting classic we should add this 07:45:58 ... it has a lot of capabilities that are not available through device emulation 07:46:28 ... in classic webdriver there is some confusion between top level browser context and the OS window 07:46:55 ... the suggestion in the issue is that we have something has an a OS window id 07:47:07 ... and have top level context in that OS window 07:47:18 ... and then dimensions that are available 07:47:38 ... and then you can set the state of the window (max/min) 07:47:45 q+ 07:47:59 ... are there other use cases that we should see about addressing? 07:48:02 q? 07:48:06 ack next 07:48:39 shs96c: This is basically what we need to implement (points to classic spec section 11) 07:48:51 ... we are just going to lft that and shift it? 07:49:12 jgraham: yes 07:49:12 q+ 07:49:13 q+ 07:49:17 ... webdriver classic does a lot of things that says "please try" and it can fail 07:49:29 shs96c: yes... like mobile can';t do a lot of those things 07:50:07 jgraham: it is also fallible in some cases in there is a window manager that has no idea how to do that. E.g. window managers don't have max 07:50:29 shs96c: from where I am sitting this proposal looks good 07:50:53 ack next 07:51:31 orkon: from our perspective the proposal looks good 07:51:48 ... for the other things these are outside the browser 07:52:01 ... and more window manager controls 07:53:26 ... it would be good to get messages coming back about what can/cant be done 07:53:26 q+ 07:53:26 shs96c: people historically run tests in as big as possible and know it's not always perfect 07:53:26 ack next 07:53:26 https://github.com/web-platform-tests/wpt/pull/41588/files 07:53:26 whimboo: one follow up is I have updated the tests recently 07:53:58 ... so what ever we do in bidi should be easy to copy these tests across 07:53:58 whimboo: can you please share a link to that spreadsheet? 07:54:00 ... the priority on this still correct as it has a high priority 07:54:02 https://docs.google.com/spreadsheets/d/1Cg3rifrBZClIitU3aFW_WDv64gY3ge8xPtN-HE1qzrY/edit#gid=0 07:54:05 https://docs.google.com/spreadsheets/d/1Cg3rifrBZClIitU3aFW_WDv64gY3ge8xPtN-HE1qzrY/edit#gid=0 07:54:09 q? 07:54:32 ack next 07:54:47 shs96c: for selenoium it is still a high priority for what I mentioned earlier 07:55:06 ... one quick question, do we want to send events back or is it fire and forget? 07:55:40 jgraham: I think we should have a response with details of the window size it got to 07:56:09 ... and we can have an event that is fired if a new window is created or a resize happens from something else 07:56:30 shs96c: that would end up with 2 events? a window resize, window create 07:56:39 jgraham: 3. one for resize, create and destroyed 07:56:44 q+ 07:56:57 ack next 07:57:14 orkon: for events we should discuss events separately as that is not part of the current proposal 07:57:41 jgraham: I agree. It is lower priority and a lot of it is covered by context created 08:00:05 JuhaVainio has joined #webdriver 08:06:08 littledan has joined #webdriver 08:07:39 patrickbrosset has joined #webdriver 08:08:16 littledan_ has joined #webdriver 08:08:29 Topic: Capture full page screenshots 08:09:07 github: https://github.com/w3c/webdriver-bidi/issues/384 08:09:07 whimboo: We are already doing this in classic 08:09:24 ... for iewport or for an element 08:10:18 ... but there are a lot of users asking to capture the area offscreen to get a fullpage 08:10:18 ... so it will be great to discuss this 08:10:18 q+ 08:10:18 ack next 08:10:49 whimboo: we have already implmeneted this on Firefox in classic so we don't have an issue 08:10:49 present+ 08:10:59 q+ 08:10:59 orkon: we are also able to do a fullpage but making hte viewport as big as possible 08:12:23 ... we will not able to capture fullscreen shot screenshots of elements with overflow or iframes 08:12:23 ack next 08:12:31 jgraham: I think with screenshots there is already scenarios that can't be handled. e.g. in iframes 08:13:16 ... I don't think that people would want to handle the cases with scrolling of text in a box 08:14:03 ... functionally we should add an extra attribute e.g. fullscreen=true that takes the viewport of the scroll dimensions of the document 08:14:28 ... it also makes taking an screenshot of the element easier 08:14:37 q+ 08:14:38 q+ 08:14:39 and do the whole element 08:14:53 ... and I think that solves the main use cases 08:14:58 ack next 08:15:05 q+ 08:15:17 Jim Evans: just as a implementation detail on the spec 08:15:48 ... is therre any mileage for making a fullscreen clip rectangle or full page clip rectangle? 08:16:04 jgraham: I think the answer is yes 08:16:31 ... it would be a viewport clip rectangle that could take an element 08:16:42 q+ 08:17:05 ... the way it is written here doesn't take the whole matrix of choices in 08:17:13 ack next 08:17:47 q+ 08:18:08 orkon: I think that makes sense. I just want to point out that viewports can affect things here. We will need to make sure that we take the edge cases into account 08:18:17 ack next 08:18:36 sadym (IRC): is this different to what Mathias asked in the issue the other day? 08:19:01 ack next 08:19:32 Jim Evans: that makes sense to have a boolean property. I withdraw my previous suggestion 08:19:35 ack next 08:20:08 jgraham: I think the previous suggestion is quite good. the previous design is weird 08:20:30 ... we could do the fullscreen like an element clip rectangle 08:20:43 ... and it makes scroll into view mutally exclusive 08:20:59 q+ 08:21:11 ... it's not very explicit in the protocol how to handle this 08:22:02 q+ 08:22:04 ... I am not sure what the correct answer is here and don't have know how to handle it right now. I like the clip rect suggestion 08:22:05 ack next 08:22:23 shs96c: I like the clip to view port would be useful 08:22:57 ... we can do viewport false to get things of everything 08:23:31 ... the classic spec has a lot of "if in view" so a lot of people maximise the window to try get as much as possible in the window to remove flakiness and then want screenshots 08:23:43 ... and we still need to have resizing of windows 08:23:46 ack next 08:24:24 orkon: the browsing context can manipulate the viewport 08:24:50 ... I don't have an opinion other way 08:25:05 q+ 08:25:10 q+ 08:25:24 ... there is the question of scroll into view in screen shots 08:26:02 ack next 08:26:33 orkon: the issue is scroll into viewport has been merged into element screenshot 08:26:49 ... it's currently an option 08:27:35 ... it doesn't make sense for element screenshot to scroll if we are using it for full page 08:27:51 jgraham: so I agree it doesn't make sense there 08:27:52 q+ 08:28:08 ... for fullscreenshot. We can make a new command or a new attribute 08:28:16 q+ 08:28:35 ... the reason is due to classic 08:28:49 ... we could make it separate commands 08:29:26 ... if we can send 2 commands in one payload 08:30:25 ack next 08:31:23 automatedtester: The context for scrolling in element screenshots was this was a request from Microsoft to be able to try take a screenshot of an element in the view port by scrolling and then times when you just want the element and it should be out of the viewport. this is why it was originally designed that way 08:31:26 ack next 08:32:01 shs96c: some of the colour for this. IE could only give you the screen shots of the view port 08:32:24 qq+ jdescottes 08:32:36 ... since we only had a screen shots to scroll as we screenshots 08:33:25 ... my preference of screen elements with the ability to turn off 08:33:39 ack next 08:33:40 jdescottes, you wanted to react to jdescottes 08:34:41 jdescottes: I don't think that it will be enough for all cases 08:34:51 ack next 08:35:28 s//e.g. scrollable non-root elements/ 08:36:04 RRSAgent: make minutes 08:36:05 I have made the request to generate https://www.w3.org/2023/09/14-webdriver-minutes.html jgraham 08:36:12 whimboo: I wanted to add to jdescottes item is that we don't want to constantly scroll e..g. twitter/facebook 08:37:17 q+ 08:37:32 ... element interaction is going to need "scroll into viewport" so that we can move the viewport to be able to handle it 08:37:48 ... this feature is part of interaction commands 08:38:02 ... but not in actions as that assumes everything you need is already in the viewport 08:38:13 ack next 08:38:41 q+ 08:38:50 jgraham: irrespective of the screenshot command there should be a top level command for scroll into view 08:39:53 ... and to whimboo 's point there is going to be a usecase to be able to scroll the element. I do think we should have a top level "scroll into view" 08:39:56 that makes sense to me 08:40:02 ack next 08:40:29 shs96c: I 2nd adding a command to scroll the element 08:40:37 ... and typing de de de is hard to type 08:40:52 q? 08:41:11 s/command to scroll the element/ command to scroll the element in an action chain 08:43:07 Topic: Ability to upload files and fill out file inputs 08:43:16 q+ 08:43:22 github: https://github.com/w3c/webdriver-bidi/issues/494 08:43:47 ack next 08:44:05 orkon: We started working on this feature. THere are some open topics on this 08:44:25 ... context: we tried to implement a method to set the file to the dialog 08:44:35 q+ 08:44:36 ... and we need to get the events with this 08:45:31 ...q1: is the interception of the dialog should be doable through the current mechanisms. Are people ok with this? 08:45:31 ack next 08:46:14 https://github.com/w3c/webdriver-bidi/pull/514 08:46:51 shs96c: Not all UIs show the dialog. There might not be able to get the file upload. For the local case this is really easy to do 08:47:09 q+ 08:47:22 ... for remote the clients are going to need to be able to send the file across intermediary nodes 08:47:37 q+ 08:47:43 ... for the UIs doing text boxes they assume the file exists on the local file system 08:47:57 ... but I think the remote case needs to be handled. 08:48:01 ack next 08:48:38 ... for selenium people hate the sendkeys that doesn't upload the file if it's just a text box 08:49:06 orkon: should we intercept the dialog and be part of the event subscription 08:50:24 shs96c: For the remote case we take the file, upload it, get the new file address and they type the new file path is done part of the sendkeys command 08:50:51 ... if set file input had the file name and file contents as the payload it would solve this problem 08:51:21 ack next 08:51:45 jgraham: for set files I think it would be fine to bypass dialogs 08:52:10 q+ 08:52:32 ... if the dialog blocks the browser there should be a way to probably send that out but it might not be cancellable. 08:52:52 q+ 08:53:07 the file dialog is a native dialog from the OS and we cannot handle that in Firefox 08:53:11 ... we could have an event for "a file dialog has appeared, please send files or cancel" so we don't block the UI 08:53:17 q- 08:53:54 q+ 08:54:38 ack next 08:55:38 orkon: we envisioned the workflow that the person would intercept the dialog and then set the file path in the dialog 08:55:47 ... and then people could choose what to do next 08:56:11 ... we suppress the dialog from appearing on screen 08:58:02 ack next 08:58:28 shs96c: in selenium the uploads can be dependent on the UA as it can block the browser 08:59:33 ... we were block the dialog from loading. 08:59:42 ... and this didn't work in Firefox 08:59:48 q+ 09:00:01 ack next 09:01:08 orkon: this is why we want to discuss it. We want to intercept it. We can stop here and then I will come back with an example. 09:02:45 shs96c: 09:04:39 In that case, being able to block the dialogue from opening would be wonderful 09:04:55 jamesn has joined #webdriver 09:05:06 ScribeNick: jgraham 09:05:06 jcraig has joined #webdriver 09:05:10 present+ 09:05:10 present+ 09:05:21 lola_ has joined #webdriver 09:05:42 RRSAgent: make minutes 09:05:43 I have made the request to generate https://www.w3.org/2023/09/14-webdriver-minutes.html jgraham 09:05:53 Topic: AOM Accessibility testing 09:06:30 spectranaut_ has joined #webdriver 09:06:53 https://www.icloud.com/keynote/0eciRqzy6aGWyffs7OXZbkz8w#AccessibilityAutomation_2023SeptTPAC 09:07:10 Slides: https://www.icloud.com/keynote/0eciRqzy6aGWyffs7OXZbkz8w#AccessibilityAutomation_2023SeptTPAC 09:07:49 Matt_King has joined #webdriver 09:08:11 present+ 09:08:27 [slide 1] 09:08:48 [slide 2] 09:10:33 jcraig (IRC): a11y engine is part of web rendering engine. That will provide information to a11y API. In the context of automation there is platform-specific automation. AT-driver will be later today. Some existing solutions here with DOM as source of truth. 09:10:33 [slide 6] 09:11:02 jcraig (IRC): a11y tests added as part of wpt and in Interop. Computed Role and Computed Label already in the spec 09:11:12 [slide 10] 09:11:37 jcraig (IRC): Have basic a11y tests running in all four engines. Currently 600 tests 09:11:44 [slide 12] 09:11:58 [slide 16] 09:12:50 jcraig (IRC): We can check for computed role and computed label for any element. There are 54 elements and attributes that affect a11y. We can currently test 3 of these. Still lots more we'd like to test. 09:12:54 [slide 17] 09:13:56 jcraig (IRC): Want to test conflicts e.g. required vs aria-required. Would like to trigger a11y actions / events. Stack is different to other kinds of events e.g. pointer click. This is not quite the same "actions" as in wpt. 09:14:13 [slide 18] 09:14:39 q+ 09:16:10 jcraig (IRC): As changes happen to DOM they change a11y tree and then causes events in the accessibility system. Concept of a11y tree walker. Currently lots of implemenation differences in those trees. Easy example is div with overflow:auto. Scroll view that creates will be represented differently between different engines. But we want to test what we can with a11y tree even if it's not fully interoperable. orkon wanted something similar for a 09:16:10 CDP feaure 09:16:21 [slide 19] 09:16:53 https://github.com/WICG/aom/issues/197 09:17:04 https://github.com/WICG/aom/issues/203 09:19:25 jcraig (IRC): Two AOM issues. Stakeholders agree on the goals in those issues. Test-only web api for a11y feature. DOM-exposed API would over-complicate things. Need a way to get an a11y object and its atributes. Can currently get two attributes associated with real elements. Need to reconcile a11y tree elements with DOM tree elements, because the relationship between the trees isn't quite 1:1. Want to synthesize screen reader inputs. 09:19:38 [slide 20] 09:19:40 [slide 21] 09:22:19 jcraig (IRC): Don't need writable a11y nodes. Don't need a live tree representation. Don't need a11y node ids to persist after the DOM changes. Node can be destroyed and recreated if DOM elements are e.g. hidden and redisplayed. Trees aren't expected to be identical between browsers. Platform specific a11y APIs aren't in scope, neither is a11y tooling itself. 09:22:23 [slide 23] 09:22:49 jcraig (IRC): [Clarifies which bits are in scope] 09:22:58 RRSAgent: make minutes 09:23:00 I have made the request to generate https://www.w3.org/2023/09/14-webdriver-minutes.html jgraham 09:24:37 jcraig (IRC): Been shopping this around. There seems to be general feeling this might be a good webdriver extension. 09:24:40 [slide 24] 09:25:34 jcraig (IRC): Want a way to get the accessibility node from a specific element. Might want to get just an id or get all the properties in a single call. 09:25:43 [slide 25] 09:26:12 jcraig (IRC): Need to be able to get a11y node by its own id, so that we can walk the a11y tree. 09:26:20 [slide 26] 09:26:48 jcraig (IRC): This is an example of what the properties of an a11y node might look like. 09:27:57 jcraig (IRC): Reason for property bag rather than individual accesors is that it reduces the number of requests required per element. 09:28:07 [slide 27] 09:29:17 jcraig (IRC): Synthesize event. Events are not quite the same as the non-a11y events, they can also affect the a11y tooling in a way that other events don't. 09:29:29 [slide 28] 09:30:01 jcraig (IRC): ARIA actions aren't like WebDriver actions. 09:30:12 [slide 30] 09:32:20 jcraig (IRC): Might be a problem with ids being reused across different sessions, but session id might be enough to disambiguate. Can probably do the property bag quite quickly even if the id portion comes later. 09:32:49 q+ 09:32:59 jcraig (IRC): Inert hides things, want to test that works. Might be differences with some implementations marking nodes as hidden and some removing them from the tree. 09:33:10 q? 09:33:20 Jem has joined #webdriver 09:33:28 present+ 09:33:41 present+ 09:33:58 ack next 09:34:33 orkon: You mentioned something about subscribing to events. BiDi seems like it's better for that . Could you use BiDi? 09:35:13 q? 09:35:13 jcraig (IRC): We don't think there's a problem with taking these concepts and implementing them in BiDi. 09:35:21 mattreynolds has joined #webdriver 09:35:30 jcraig (IRC): But we don't know about BiDi shipping schedule. 09:35:32 q+ 09:35:34 q+ 09:35:46 jcraig (IRC): Could put this on the BiDi roadmap. 09:35:53 q? 09:35:57 ack next 09:37:52 jgraham: Seems like the design would work well in BiDi, not just for events but also because the tree properties match the way we do DOM in BiDi. 09:38:15 jcraig (IRC): Could maybe do a subset in classic today and then add other stuff to BiDi long-term 09:38:51 shs96c: Events are hard to do in classic. 09:39:52 jcraig (IRC): We could send events to the page? 09:40:30 shs96c: Yes. We're also talking about how to reformulate classic in terms of BiDi so we expect it to be a superset 09:40:30 ack next 09:41:03 q+ 09:41:03 orkon: Should this be an extension or a core part of the spec? In BiDi it seems like it could be a core part of the spec. 09:41:03 jcraig (IRC): What do you see as pros/cons of extension vs not 09:41:03 q+ 09:41:36 orkon: Don't think this should be an optional thing. 09:41:36 q+ 09:42:01 orkon: We have use cases in puppeteer. We want treewalker to get a11y tree snapshot. We want to query a11y tree e.g. finding nodes that have certain properties. 09:42:05 ack next 09:42:41 q- 09:43:47 AutomatedTester: I'm with orkon on making this a core webdriver feature. In testing space people are starting to look at doing a lot more accessibility testing. WebDriver should take on the hard parts. This could simplify a lot. 09:45:34 jcraig (IRC): One of the reasons for suggesting an extension is that it would allow clear delineation of responsibilities. People in ARIA group would be willing to work on this. We're happy to take in whichever direction you want. Don't want to make unreasonable requests. 09:45:34 AutomatedTester: I think this is important and so should be in core. 09:46:21 q+ 09:46:22 jcraig (IRC): One complication might be that we'd really like to test parent child relationships, but right now they aren't the same between browsers. Is that OK? 09:47:17 ack next 09:47:32 jgraham: where this lives doesn't really affect wether it is required to implement. The issue is more to do with the ownership of doing the work 09:48:12 ... I think it being in a different spec this might make sense 09:49:11 ... as for trees that could be very different between browsers scares me but doesn't mean I think we shouldnt do it 09:49:11 q+ 09:49:11 maybe we should do: find accessibility child of role x 09:49:11 ... I think we should be worried that people will assume the way a specific browser works doesn't means to users 09:49:20 I think that would be the same across browsers...? 09:49:26 ... e.g. Browser A returns a specific way and then other browsers are "not accessible" 09:49:31 qq+ to respond to jgraham 09:50:01 I wonder whether we heard about ideas about James Craig's question regarding parenet - children relationship 09:50:10 ... there are enough legitimate use cases here to do it 09:50:11 ack next 09:50:14 jcraig, you wanted to react to jgraham to respond to jgraham 09:50:15 q+ to say maybe we should do "get accessible child of role x" which should mostly be the same across browsers, with some exceptions 09:50:34 jcraig (IRC): from Google is trying to make this the same 09:50:50 ... there is still utility in being able to access the actual tree 09:51:07 ... I think we can still move forward on what we have here 09:51:14 jcraig (IRC): There's a concept of a normalised tree which would align with ARIA's definition of a11y parent and child. There's some benefits of getting the underlying tree to help align implementations. 09:51:50 jgraham: a good analogy here is that we don't expose the layout tree between browsers 09:52:04 ... we need to be wary of what can be returned 09:52:31 s//Aaron Leventhal/ 09:52:34 ... and we need to explain that these should be behind flags with the explainer that things will be different between browsers 09:52:40 q? 09:52:42 ack next 09:52:48 jcraig (IRC): Having the implementation tree accessors behind a flag seems reasonable. 09:52:50 q? 09:54:12 q+ 09:54:34 shs96c: As long as you can express the same concepts in the tree between browsers, that seems find. Could also base tree walking on find element-like API rather than walking the tree, so you'd skip over things that are different between implementations. 09:54:44 q? 09:55:16 ack next 09:55:49 Stewart 09:55:49 orkon: There's also a discussion about find element, and there's a proposal to make those work with role, so that might affect the extensions question. 09:55:49 ack next 09:55:49 spectranaut_, you wanted to say maybe we should do "get accessible child of role x" which should mostly be the same across browsers, with some exceptions 09:56:25 ack next 09:56:26 spectranaut_ (IRC): I was going to recommend a similar solution "get accessible child with role " 09:57:04 q+ 09:57:41 Jim Evans: I wanted to point out that in BiDi spec there's prior art for serializing children of DOM nodes and get a tree in that way up to a certain maximum depth, which might also work for the a11y tree, 09:57:44 s/,/./ 09:57:48 ack next 09:58:32 q? 09:58:36 [clarification that "prior art" is not meant in a legal sense] 10:14:48 Matt_King has joined #webdriver 10:20:25 Matt_King has left #webdriver 10:26:53 RRSAgent: make minutes 10:26:54 I have made the request to generate https://www.w3.org/2023/09/14-webdriver-minutes.html jgraham 10:29:09 patrickbrosset has joined #webdriver 10:29:21 Topic: Ability to upload files and fill out file inputs (contd) 10:29:38 github: https://github.com/w3c/webdriver-bidi/issues/494 10:29:59 ScribeNick: AutomatedTester 10:30:43 wfilpek has joined #webdriver 10:30:43 q+ 10:31:04 ack next 10:31:50 orkon: We have a use case that a dialog would be shown to a user. We would like to have an event that shows that a dialog would appear. We would surpress the the dialog from loading 10:32:23 ... we would then notify the user so they can then decide to dismiss the dialog or complete the form 10:32:46 ... we also want to have people to automatically handle the dialog if people aren't expecting it 10:33:20 ... the interception of the dialog would be happen if the person subscribes to the events 10:33:31 q? 10:33:46 q+ 10:33:47 https://html.spec.whatwg.org/#show-the-picker,-if-applicable 10:34:19 ack next 10:34:20 jgraham: I have found the relevant part of the html spec for this 10:34:40 ... we would effectively override steps 2 and 3 10:35:15 ... there is some interesting edge case here whether the element fets the cancel event 10:35:54 ... and if you didn't respond that's a case that wont happen with a real dialog 10:37:08 q+ 10:37:08 ... I think if you are not subscribed to the events that we should cancel the dialog automatically 10:38:10 ... the worst case is that we get the dialog and can't handle this 10:38:34 orkon: we have the same situation in alerts and we should handle in the same way 10:39:07 jgraham: yes, we need to be able to handle this. currenetly people need to subscribe and handle the alerts as they appear but we should probably go back and check the spec in bidi here 10:39:56 jgraham: we should sort this with alerts 10:39:56 shs96c: we should do what classic does here 10:41:01