Meeting minutes
<simons> G'day, @AutomatedTester
<simons> You can dial into the zoom call too?
<plh> Meeting information: https://
<plh> AutomatedTester: use https://
<simons> Can someone share a link to today's agenda, too?
<orkon> https://
https://
<sideshowbarker> AT Driver folks will be coming by here at 11
AutomatedTester does some wibbling
topic FPWD
sideshowbarker tells us that plh is wondering why he's here.
sideshowbarker we'd like to transition to CR as soon as possible.
sideshowbarker How quickly can we do that?
plh You can do that as quickly as the responses come back from the reviewers. If we were moving to REC, we'd need to wait longer
FPWD
plh wide review is required to move to CR for the first snapshot
plh send review requests as soon as the FPWD is published, and then you can move on
AutomatedTester I thought we were following the "ever blue" process since before pre-COVID. How does this impact everything, because I thought we had a living document
sideshowbarker What we're not getting at the moment is no RF commitments. Until we move to spec, we won't get those.
plh They're only written in stone once we move to CR. Once that's done, you can publish a CR whenever you want
plh You need to publish a CR snapshot every two years. You can publish a CR draft whenever you want. You won't get any RF commitments until you publish another CR
sideshowbarker Once we move to CR we can continue to auto-publish. We can do that to a w3c.github.io URL, but we can't publish into the TR space until we have the RF commitments.
sideshowbarker The only way to get new RF commitments is to publish a new CR snapshot. We need to do that for any new normative requirements.
plh makes helpful comment
AutomatedTester There must be some subtle difference. If there are changes to implementation...?
plh No! Only substantive changes to the specification
sideshowbarker no-one's paying attention to the substantive changes. It's up to our discretion. Lawyers at implementing companies may want us to do that more regularly
plh The biggest hurdle is the review. Getting eyes on the spec is the hard thing to do.
plh If there is a long time between snapshots, you might have a bad time since the reviews may require changes.
jgraham This sounds fine to me. I'd not realised we'd not published a FPWD. We can then start the CR process, and then work out the schedule for later
AutomatedTester I was under the impression that because we were "ever blue" that we were already doing things. Want to get to a state where we work, publish, and carry on
plh Should ask ourselves every six months whether we need re-review
sideshowbarker The current wide review process takes about five outside groups at once. When you go to a CR transition, you need to fill out a self-review for each group you ask for review (so "times 5") The way to streamline it is to pre-emptively do this, and point out things in the spec in advance of the review
sideshowbarker AutomatedTester needs to send a message to the group to ask whether we want to move to FPWD. If there are no objections, then we go
ACTION: AutomatedTester send the request to publish the FPWD
plh will send a sample of this to AutomatedTester
jgraham do we have a point of reference about how to request horizontal review.
sideshowbarker plh has written documents for this, but that doesn't give you a full sense of the experience.
plh You can start with the TAG.
plh Best thing to do is track through a GitHub issue. That makes it easier to track. Can send a sample
<plh> Sample CfcL: https://
RRSAgent make minutes
State of the Union
<plh> Sample issue to track review: w3c/
<simonstewart> s/Action /Action: /
<Zakim> sideshowbarker, you wanted to say how we start the proces
AutomatedTester We have a basic idea of where implementations are. It would be good to get a more concrete version of that, and we can go from there
AutomatedTester will go round alphabetically
simonstewart Apple do not comment on unreleased products
simonstewart or services
sadym We are on good track, and have implemented almost everything specified.
AutomatedTester Are there any areas where you're currently needing a lot of help with specification, or is it documented in the agenda items?
sadym The agenda is up to date
whimboo There are lots of things to talk about and implement, but the base is what we agreed with the Google folks for puppeteer. We are about 90% implemented. Dialogue handling is missing. The unimplemented areas have been considered lower priority.
whimboo Things we'd like to get added include some feedback from the playwright people. I hope we get to discuss that over the next few days
<jgraham> https://
jgraham This shows (to some extent) what's been implemented in different implementations.
<sideshowbarker> LadybirdBrowser/
sideshowbarker I'm contributing to a new browser engine called Ladybird. It's written from scratch, and doesn't use any engine code from anywhere else. We're doing testing, and finding out what needs a webdriver implemenation
sideshowbarker if there's an algorithm in the spec, then we try and track that as closely as possible.
sideshowbarker in cases where we run into a problem where it's not clear we know what we're meant to do, we file spec issues.
sideshowbarker there are about six engineers who are working on this. Some of them may have raised bugs already. There will be more bugs
sideshowbarker if we find we need to do something that the spec doesn't mention, but other implementations do, we put comments in the code. We raise spec bugs when we do this
sideshowbarker I may end up working on the webdriver implementation for ladybird.
sideshowbarker The engine is currently in alpha. The only way to run the browser is to build it. The plan is to have it that way for at least two more years. A year after that, we'll do a beta release, and a GA in 2028
sideshowbarker there's a lot of stuff that we're going to be implementing, and that should help with the spec work
whimboo do you both bidi and classic, or just classic?
sideshowbarker we've only started with classic
simonstewart since the energy of the WG is to bidi and the testing community is moving to bidi, that's the best place to spend energy
<gsnedders> s/^plh /plh: /
jgraham: for that, it depends on what your goals are. Classic is needed for WPT, and it's easier to implement. I also agree that long-term you'll want bidi
sideshowbarker: the main goal is to see if we're introducing regressions. We're not running WPT under CI yet.
AutomatedTester: do we have anyone from MS who wants to talk about Edge?
AutomatedTester: I don't see anyone
RSSAgent, make minutes
Emulation
Emulation - User Agent
github: w3c/
whimboo: the issue request is from quite a long time ago, since we decided to implement this sufficiently for puppeteer. Now we have more clients using bidi, they all want to have a way of changing the UA
whimboo: this is why I requested a separate command to get and set the UA for a browsing context
AutomatedTester: I guess this is possibly a question for the Googlers. Do we have any metrics for how widely this is used? I ask because the Selenium project always aims to test in the same browser that people use, rather than (eg) making Chrome act like Firefox.
orkon: We don't have any data. We know people do use it, but because we're an OSS project we don't know how many people are using it.
jgraham: I think there are legitimate use-cases for this. We know that people try to do mobile emulation on desktop, even if they shouldn't be doing that.
jgraham: Other people want to check what will happen when the UA string changes (eg. will that break my site?)
jgraham: The technical question is what we want the scope of the issue to be. The obvious thing is about HTTP headers. Then there's navigator.userAgent, which you can use a preload script for. Then there are other features, which you may not want to override in an adhoc way.
jgraham: This is a little bit of a thing you could implement in other ways. What do people actually want the command to do?
AutomatedTester: there were concerns from Mozilla and WebKit about this before. It might have been around client hints, and perhaps UA strings.
AutomatedTester: do these concerns still exist? How will that affect how we go about creating the implementation?
orkon: I recall the conversation from last year. Client hints can be considered out-of-scope, and not all browsers implement them anyway.
orkon: from the spec perspective, I think we need two commands. One for the HTTP headers, and one for the web APIs. From the testing perspective, only one command makes sense.
jgraham: to be clear. Mozilla can't override client hints because we don't implement them.
jgraham: in regards to what orkon says, a command that changes the http UA string and navigator value makes sense. I'm wary of allowing this command to make other changes over time, as that might mean introducing other changes
simonstewart: If someone is asking to change the UA string to emulate another browser, wouldn't you like to try to be as close as possible to that other browser e.g. client hints?
jgraham: I think the answer is that we have a lot of commands that are orthogonal commands, rather than one "do everything" command.
jgraham: so things that are specifically about defining the UA might be handled by additional parameters in the command
simonstewart: so why not have separate commands at that point?
jgraham: Let's assume we had a command to change http headers, and we had a preload script command. That would allow the UA string and then the navigator.userAgent string to be set. So do we need the single command to do those two things? I guess that's a question for the CDP folks to answer.
jgraham: I'm not against a small quality of life command to make things simpler. Without that, more burden is left to the clients to get things right
gsnedders: my main take here is that there are a bunch of things that should always be in sync. There's no browser that gives different values in http headers and another in navigator.userAgent. It feels like there should be a way to keep things in sync, which implies a single command
gsnedders: one question is that the CDP command that changes these things, if you don't pass any of the optional values, does Chrome attempt to derive things like client hints. If we're trying to keep everything in sync, then should the single command change those? We should design things so that they _can_ stay in sync. We shouldn't rely on
preload scripts
orkon: about changing other properties based on the UA. The playwright case is "I want something that looks like some specific mobile device". They collect all the properties they think is meaningful, and then they call all the different commands to change the browsing context. It's central to the way that they encourage people to write tests.
orkon: I don't know how the UA emulation works in CDP, but I think client hints override UA strings.
<gsnedders> (Kinda as an aside, with CDP: https://
jgraham: I'm sceptical of a command that's too "do what I mean", particularly if there's a change in API surface. Having a command that suddenly starts changing that part of the UA may be a bit unclear about how that should actually work. Will that break people's tests that depend on headers (etc) not being changed.
jgraham: if you put in the primitive data, how would you know how to reassemble the header, and vice versa. You'd need to specify both things.
jgraham: looking at the CDP command, you need to set the UA string, and then optional other things (such as platform). That allows the command to evolve, and means we don't need to define a mapping between UA and (eg) screen size
orkon: I wanted to mention about preload scripts. We implement this as request interception.
orkon: I think for some scenarios and emulations, you want to apply things dynamically. That's an argument about relying on preload scripts for emulation
jgraham: it seems like people agree the command should exist. Is the disagreement that it should start with a UA string and (possibly) platform, and that override the http UA string, navigator.userAgent, and potentially the platform. Client hints could be an extension, or defined in the client hints spec.
gsnedders. I think so????
jgraham: This isn't a formal attempt to get consensus, just to find out whether you'd object to this API in general.
gsnedders: that seems reasonable. I think my main concern is that we end up in some space where Chrome is doing something different such that UA client hints make you think you're looking at one browser, but the UA string defines it as another browser. That may be a chrome problem
jgraham: that may be an author problem....
simonstewart: people using libraries may not be able to consistently know whether the UA string or client hints are being used.
jgraham: it's a hard thing to solve without some kind of massive database of what each browser can do. Maybe that's a local-end issue.
jgraham: From a slightly different perspective, we sometimes do this for web-compat stuff. Often you can get away with overriding specific things. You don't need to be 100% accurate. Would rather provide a command to do the low-level things we need to do
simonstewart: that suggests the client hints changes should be in the client hints spec
jgraham: If people wanted to add client hints as an optional parameter, I don't think we'd object to adding that to our spec
<gsnedders> github-bot, end topic
AT Driver - Embracing variation in support for commands/intents
github: w3c/
jugglinmike: The first issue is how we handle different ways that people might want to automate interactions with an AT like a screenreader. We've borrowed heavily from BiDi. Same concepts of commands and events and extensions, also websocket based. We're in a very early stage of development, and we want to allow implementors to extend and
experiment as a path to standardised extensions.
jugglinmike: We've discovered that people want to implement a variety of user interactions. We've currently only specified sending keys. Element.sendKeys doesn't work for our use cases, we need something that works at a level above the browser so that an AT can get those events. We know that some implementations aren't comfortable with that level
of access. That doesn't mean the proposal is dead. We could model the AT commands as separate commands rather than just allowing arbitary keyboard access e.g. move to next list itme, move to previous list item, etc. That's a much wider API surface to standardise. Different implementions have different commands with different semantics. Could be an
oppertunity for extension commands. Want to be able to define "intents" that allow a bit more consistency between extensions.
simonstewart: The sendKeys API is very high level, but we also have the actions API that allows individual keyboard shortcuts, but not to the OS. Do all ATs use exactly the same keybindings? If not you need a higher level command
Matt_King: By coincidence they all happen to use H to get to the next heading. That has to be interpreted by the AT, not the browser. The case where they're not all the same is more common than the case where they are.
Matt_King: A concept of "navigate to next heading" is what we envision. All screenreaders have similar intents, but not similar key commands. The idea is to be able to express the intents in a platform agnostic manner.
<jcraig> w3c/
jcraig: The keys need to be directed to the screenreader, not the web browser, so that explains the current system event design. I think the above issue explains the current state of progress, and the abstraction over different implementations. The abstraction could either go via simulated keypress, or some other way of interacting with specific
implemenations e.g. applescript for voiceover
jugglinmike: To get towards that place we want the extension point for AT driver to be more supportive of this aspect of implementations. Instead of encouraging implementors to specify many many commands, we'd like a slightly more structured way to do this so they can be implemented in a more forward comaptible way. This is the idea of intents.
Currently we only have sendKeys, but the idea is to allow implementors to do similar things. We have the concept of capabilities. WebDriver has a similar concept. The remote end could ask for a screenreader that supports certain intents, and they can learn when they start a session which intents are supported by a session. ATDriver is already using
capabilities to do things like negotiate AT version. We have a lot of the same plumbing in place. This is a larger space, so we're looking at many many intents. Does that or any other aspect of what we're trying to do here make this mechanism inappropriate?
jgraham: with using capabilities for target selection works well in webdriver
… using it for understanding the configuration doesn't work as well once the selection has happened
… in webdriver we have the idea that a script is able to select the items from the start
… having the commands passed back at the beginning is potentially complicated
… in webdriver we generally error when you hit a command that doesn't exist
… or if on some OSes we only want certain arguments you're not only going to know when you hit that issue
… if you're using it for selection then anything else is over complicating things
jugglinmike: thanks for that... it's good food for thought.
jugglinmike: Depending on how quickly implementations move it might be useful, because it could relieve developers of having to know which versions they support. You might be able to get by just by knowing the release history of your screenreader. But it is a lot of complexity.
… we were thinking that there are times where people might be able to do this with different capabilities
jugglinmike: The protocol can return an error when a specific intent isn't supported and then the test can error.
AutomatedTester: I wanted to add that the perform actions API was designed to allow creating new kind of inputs e.g. input for keyboard, mouse, but it would be extensible to e.g. AR/VR inputs. When it comes to things like sending keystrokes look for opportunities to reuse that API. WebDriver uses the term "remote end", but that doesn't have to be a
browser, it could be a screenreader for example.
AT Driver: "Send text" user intent
AutomatedTester: Please contact me on matrix or anywhere else if you want to discuss further
<gsnedders> GitHub: w3c/
jugglinmike: This is a specific example of a user intent. We recently refactored our send keys interaction so that it's not a command, but a parameterised option of a user intent command. We're now adding send text. We expect that for implementations that will not support key presses, that even for a slew of custom intents, there will still be a
use case for e.g. interacting with form fields. This is required for automating logins for example. It seems like a safe bet that something which provides access to content rather than UI for e.g. filling out form fields. WebDriver has sendKeys, WebDriver BiDi has actions. The problem with this use case is that the screenreader has its own concept
of focus. That doesn't always line up with the document's active element. We tried to imagine scenarios where some higher level library would abstract over ATDriver and WebDriver, but I'm not sure that's viable because of this extra information in the AT; WebDriver wouldn't be aware of where the user's action is targeted
jugglinmike: Does what I've said make sense? Is it actually true? Could have a purpose built API in ATDriver to find the element with user focus, but that seemed fraught, but then specifications would have to have a shared definition of element ids, which seemed hard.
jcraig: For form entry, I thought that WebDriver would solve the problem, but it sounds like you're still trying to generate the keys at the AT level, and I'm wondering what the specific use case here e.g. keyboard echo. What are you trying to test?
jugglinmike: It's not that we're trying to test another part of the stack, but in an automation scenario the client will move the virtual cursor through the document, and WebDriver wouldn't know where the virtual cursor is
jcraig: So this is about pulling focus along with the screenreader. The webapp doesn't know where the virtual cursor is. But to insert text it has to focus the field first. Is that right?
jugglinmike: Yes
jcraig: Could you just trigger focus on that field right before sending text. Could just have a focus event. We might have that already in the voiceover applescript bridge. That seems like it would solve that use case. Is that enough?
jugglinmike: That sounds like a more elegant solution. It means we don't need to duplicate element send text from webdriver.
jcraig: I think there's a bringKeyboardFocusToVoiceoverFocus command, but I don't know about other screen readers.
Matt_King: In Windows screenreaders, the screenreaders have a command for activating the current element. I can't think of a situation where that wouldn't cause the screenreader to use the a11y API which would cause focus
jcraig: I think you want to trigger a focus event.
Matt_King: There could be side effects of using a click event to move the focus.
jcraig: Agreed, that's why a separate event
Matt_King: ATs should be able to set focus.
jcraig: Need to know about the DOM id, the a11y tree identifier would be something separate. You know that you've moved the screenreader focus around. Bringing key focus to the current screenreader focus feels possible.
Matt_King: On the AT driver side we want a trigger this focus action. This is the equivalent of ctrl option shift F4
Matt_King: Other screenreaders have that cpaability, if not an equivalent of that key command, but we could specify that as an Intent
simonstewart: In WebDriver classic there is get active element. It seems like mirroring that kind of thing would make sense. When we do locateNodes, one of the options is to take an a11y locator and return an a11y node, that might be useful.
simonstewart: Maybe not an a11y node
Matt_King: The browser doesn't know where the reading cursor is
jcraig: You can find out from Applescript.
Matt_King: If we had get accessibility active element, what would that return?
jcraig: You'd get the text, but not an element reference
jugglinmike: Yes, you'd need to bridge between those
jugglinmike: Instead of commanding ATDriver to focus the element under the virtual cursor, we could query WebDriver for the element that holds the virtual cursor
Matt_King: We could just use WebDriver to move the active element and then let WebDriver take over
jugglinmike: We could either do that, or use WebDriver to request a reference to the element under the virtual cursor
<jcraig> VO AppleScription "perform command" name is "Move Keyboard Cursor to VoiceOver Cursor" which a user would trigger as Control+Opt+Command+F4 (or Control+Opt+Command+Fn+4, or via KB commander)
ChrisCuellar: I'm worried that if we're writing tests it might be too prescriptive to manually sync the virtual cursor and the browser focus. We might want to see when those things are out of sync. We leave the AT up to its own devices a bit. We hope that the browser cursor and virtual focus are synced.
Matt_King: To get the reading cursor permissions via WebDriver you'd need more permissions.
jcraig: You're currently testing with default permissions. In VO you wouldn't need this because the focus is automatically moved. Is it needed for other ATs?
Matt_King: The default in JAWS is that focus doesn't move until you switch to forms mode
Matt_King: There are some kinds of failures you might not detect if we auto-move focus
jcraig: e.g. for things that aren't technically forms elements? Wouldn't that be a JAWS bug?
Matt_King: Yes
jcraig: Then you just write the test as is and it reveals the bug.
AutomatedTester: Do you have what you need from this group at the moment?
jugglinmike: I believe so, yes
jcraig: Last year I expressed concerns about the security path you were taking, but I want to convey thumbs up about the path you've taken since then. I don't have concerns about the way you're deploying this in a CI environment, as a developer side tool there would be some concerns, but this is good work, thanks.
Emulation-timezone
github: w3c/
orkon: This is to raise that we need an API for how to do this. It's part of what people are asking for from selenium and playwright and stackoverflow
… the current way is to do this via CDP which is very difficult to manager
jgraham: do we understand the detail that is needed? This is a general question for all emulations APIs
… do we want to have it manipulating ecmascript
… if I have misunderstood can we get it corrected?
orkon: we want to affect WebAPIs. I am not sure which ones at the moment
jgraham: I am reading the ecmascript spec and there is a timezone object. DO we want to manipulate that or do we want to manipulate more the browser
whimboo: in playwright they are doing is manipulating the timezone and set it to the custom time zone
jgraham: I think we need to get a better understanding of what they want to manipulate exactly
<whimboo> this seems to be the place where the TZ env variable gets modified: https://
jgraham: there is no objection from our side? are we happy to have an API that allows us to manipulate the webapis to return a different timezone to the computer?
orkon: from the test and code it manipulates the ecmascript
… [explains the test from the link above]
orkon: it looks like what they have is a good starting point
whimboo: they manipulate the TZ env variable in firefox
AutomatedTester: moving forward there is consensus that we will implement this and it will manipulate the ecmascript tz object
gsnedders: looking at the playwright webkit code, I think it would be better that we say that we are emulating this against the top level browsing context so that it does more than what is just in ecmascript
<orkon> chromium impl https://
jgraham: we can write it in a very handwavy way but there should be a minimum list of what it affects
… there is a question around what this would be like for each process... e.g. multiple top level traversable can share a process
… in firefox this affects the whole browser and it's subprocess since its using the env var
gsnedders: and webkit
orkon: in chrome it is implemented per process... so that could affect multiple navigables/contexts
… maybe it would be better to have this per user context
… that might support playwright use cases
jgraham: not sure if each user context is in a separate process
… perhaps we need to have a discussion around user context commands
<whimboo> https://
whimboo: looking at playwright they are setting it at the process context and we might have a lot of work to do at the user context. Looking at their docs they will wna tit to be at the user context level
orkon: for playwright... they are want it per user context. Puppeteer does this at the top level navigable... which might be good enoug
gsnedders: it all depends on how people are actually using this; developers might not actually run into the differences here because most testing situations only have a single test running at a time, so maybe it doesn't super matter how we define this?
emulation - geolocation
https://
github w3c/
github: w3c/
jgraham: geolocation is a special case. THe webapps group have created an extension for this command. THere are a few issues but we can take their PR and put it directly into our spec
… and since we're putting emulaton directly into our spec that we take theirs and create a PR and have that as a starting point for our discussions
sadym: when you say putting these into our spec are we going talking about the earlier topic or something else
jgraham: like viewport is an emulation but I see what you mean
orkon: I wonder if this would be better in the geolocation spec so that it maintained as a whole rather than in this spec?
sadym: if we keep the geolocation in the geolocation spec where should the other emulatons live?
gsnedders: did the old JSON wire protocol handle geolocations?
simonstewart:
… no it didn't... yay past ys
simonstewart: if geolocation are happy of this being an extension we should encourage to them to keep it since they have a better understnading
<gsnedders> s/what people want and the implementations might not work at user context/how people are actually using this; developers might not actually run into the differences here because most testing situations only have a single test running at a time, so maybe it doesn't super matter how we define this?/
simonstewart: and to sadym question we should keep things where possible as close to the owners of that area
<gsnedders> https://
simonstewart: if we could keep webdriver super simple that would be awesome
jgraham: it is essentially true that we could have this across all different specs
… but other specs have issues around our "stability"
… I just feel like it being a core feature for us it should be in our spec
… if we know about automation frameworks want this then it should be our spec
simonstewart: how do you figure out where this would work in ours or there?
jgraham: I don't think it will change how things are implemented
<orkon> https://
whimboo: we already have an issue open to list external specs that are extending webdriver
… and we don't have anything for pulling in and generating the cddl specs
… so we ould need to implement that
jimevans: to echo what whimboo is saying. As a client side implementor it can be difficult to figure out where things have been implemented
… this is a problem in the classic and bidi spec. we try maintain the list but this is not always viable
… I am more inclined to keep things in 1 place to help create better client bindings
gsnedders: there is a project called webref that contains things that it knows about the WebIDL/CSS Properties
… we could speak to them to get extracting CDDL implemented for us
<whimboo> i just filed w3c/
sadym: where is the line for things should live is I think it should be "Is this in classic?" if it is not then it should not live in bidi since it is a new way of doing things and other things should be in their respective homes
jgraham: I think if that is the line then we are already breaking that with network interception and a few others. Practically following that will move it hard for us to complete things... e.g. getting Fetch to do the network intercept stuff
… our spec has been cases of are they willing to handle the potential extra work load
… we also have not been good enough of reviewing things. e.g. geolocation have followed permissions who were doing the wrong things with using webidl
… centralisation has a lot of advantages
… I don't mind it being in geolocation, it just felt a little easier
orkon: I think would be great if the experts on their spec contribute to the testing part too
… but I don't have a very strong opinion here... also the tooling problem can be solved so not concerned by that
<gsnedders> Filed w3c/
<gsnedders> jimevans is mentioning w3c/
<gsnedders> github-bot, end topic
Emulation - Browser locale
github: w3c/
jgraham: I think there are 3 separate items here. one thing is set locale and is in CDP. This is isomorphic to the timezones discussion earlier
… accept-language header... as it says in the we should have this like locale
… and we have navigator.language but not sure where that comes from
… I am broadly in favour but not under 1 API
… navigator.language is implementation defined... so do we want this to be set similarly when setting locale?
gsnedders: at least for native apps on apple platforms language prefs and locale settings are done separately
… e..g en-GB for language and en-US as locale for currency
or fr-FR and have locale to that of Japan
<gsnedders> I'm pretty sure there's a similar split in WebKit for the Apple ports between language (e.g., `Accept-Language`) and region (e.g., `new Intl.DateTimeFormat(undefined).format(date)`)?
jgraham: we want a command to overload the intl API do. We want a command to set command for accept language and look at what navigator.language does and how its implemented everywhere
emulator - Support emulation of network conditions
github: w3c/
Interception of created navigables
github: Interception of created navigables
github: w3c/
github: w3c/
<dmurph> We need to test behavior that involves clicking a link which opens a new frame / navigable. And then do test steps in that frame. Is this supported today? If not - that issue sounded like it was to add that support
<Zakim> dmurph, you wanted to ask about github issue: w3c/
jgraham: what is possible in general that when a navigable is created you get an event with an id and you can send commands to that id
… the issue here is about automatically config navigables
… [explains how the viewport info could be inherited but isn't currently there]
sadym: from understanding is "install web app, click links and make sure the app keeps the new windows"
dmurph: that sounds correct
jgraham: in bidi there is not concept of an installed app. if you need to add more details you will need to extend bidi spec
dmurph: sounds like we can get started here and get going
jgraham: just to be clear, we haven't discussed this issue. The issue as part of this about how to make sure that config items are inherited across navigables as they are created.
Emulation - Support emulation of network conditions
github: w3c/
AutomatedTester: This is something that is regularly coming up to allow people to test their applications and make sure they are useable offline as well as making sure that service workers are installed properly and application is usable in "low" network areas
jgraham: There are clear uses and I am in favour of working on this. I wonder if setOffline should be a separate command and isn't related to emulating the network
… and I don't think we should be using magic numbers here
… I also think the spec work here will be very intricate. we can't just use Fetch here
sadym: having offline and throughput will be good
… and I am in favour of dropping `-1` as a value
Emulation - Support for specifying extra HTTP headers
github: w3c/
jgraham: this is an extra command to alter extra HTTP Headers
… I was hoping to get away with not doing this but it's isomorphic to adding this to network interception
… there are ways that we can do that is described in the issues
… there is an question around handling existing headers
… so do we overwrite or append?
sadym: my question is around the spec of this. Can we actually spec it?
jgraham: yes, we can reuse the netwrok interception section
… there is an interesting use case for handling this if you have these set in the network interception and you have set the value
… there should be infra for this
sadym: there is a question to have a remove the headers later?
jgraham: yes we should have that
gsnedders: on the question of multiple headers? I think in reality there have been 3 different sematics for headers...
… e.g. with duplication that it takes `HEADER: Value1, Value2`
<gsnedders> ... first/last (I forget) header for a given name wins
gsnedders: and set cookie headers do some magic
… so I am not sure we can come up with a singular answer of what we believe the sematics will be here
simonstewart: I think the logical order is to have set headers and then allow network interception to overwrite
… it would be good to have a replace headers so that we don't have to worry about the heuristics of how these things are added over time to different specs
jgraham: I don't think it will work to append
… there are definitely use cases for replace more than append
… we can have the default to override except for cookie where it appends
… or we can have a mode that says append or override
jimevans: the syntax for this is going to complex... combining override and append
I am imagining `[{name: "X-Header", value: "foo", mode: "append"}]` with `mode` being optional
jimevans: manipulating all network intercepts with one command is a little hard for to visualise
<gsnedders> https://
gsnedders: I think it would be good to have a look at the Fetch API does for the developers
jgraham: let's do what they do...done
So "append", "delete" and "set" as possible modes
jgraham: I would prefer 1 command
simonstewart: and I think we need to make sure that we have removed any form of ambiguity
WebExtensions CG
github: w3c/
<robwu> w3c/
jgraham WebExtensions CG would like to extend testing in WPT
To do this we need to be able to install and uninstall extensions at runtime
ack
jgraham: From the WebDriver BiDi point of view, a PR has been submitted.
jgraham: It's a useful use case. And the PR is a reasonable start.
jgraham: At the moment, there are issues with definitions of what defines an extension, and we will need to do a lot of definition (or hand waving) to be able to fully spec the behaviour
simonstewart: Seconding what jgraham said. Firefox's WebDriver extension already has some methods for extension installation/uninstallation.
<Zakim> jgraham, you wanted to react to xenon
xenon: Biggest concern is that we (Apple) don't have an implementation of WebDriver BiDi. We would need something in WebDriver classic sooner rather than later rather than relying on a BiDi implementation.
jgraham: We already implement extension methods in Firefox, so I don't see any issues with specifying it in WebDriver classic. Some may have issues
oliverdunk: I'm confused about the order of operations. Do you need to add to WebDriver BiDi first, before adding to WebDriver classic?
oliverdunk: For non-WPT use cases it may be useful to be able to send an extension over the wire to a remote end.
sadym: Implementation-wise, for chromedriver, we would prefer to implement BiDi first, rather than classic.
Not impossible, but preferrable.
sadym: it's not a blocker, to implement in classic first.
jgraham: As far as the existing PR, you either send a path to a zipped file, or a binary encoded zipped file. Similar to what we do for profiles in Firefox.
jgraham: That seems to work well, so there's nothing technically infeasible about it.
jgraham: If we were concerned about file system access, it would be completely reasonable to use something likc base64 encoded file as a transmission mechanism.
xenon: We will need to make this work for iOS also so sending across the wire is probably an approach we will be needing. We can't necessarily rely on file paths.
jgraham: For the Firefox use case, for certain use cases, paths are more efficient, but it's not required for a given implementation.
[discussion about WPT use case]
simonstewart: For the Selenium use case, since we never can be certain of the availability of the location, we will probably use the base64-encoded case as well.
gsnedders: This is not the only case we've discussed recently where the local and remote ends are not the same.
gsnedders: Maybe we need an indicator of whether the local and remote ends are colocated? But jgraham is looking aghast at that suggestion.
jgraham: it's probably best not to rely on this.
robwu: I'd like to make sure we work with something portable, so that the user can be sure that it is reliable.
[general agreement]
zombie8: When we write extension APIs, we need a way to communicate with the extension. I'm guessing those types of commands would be easier to add in BiDi. Is that the proper way to think about it?
jgraham: The answer is "yes, in principle" but in practice, there's nothing to prevent another approach.
jgraham: In BiDi, you have access to more realms for execution of script and run scripts in those context. We also have mechanism for communication back.
jgraham: In principle, BiDi is well positioned for extensions.
zombie8: Do we have a way to access the extension context?
jgraham: Like a content script?
[discussion around clarification of how extension work and what contexts are available]
dotproto: It feels like it's probably too early for this, but there's a concept of native messaging. Is there a way we can automate and test that messaging?
[general discussion, feels like it's out of scope for now, because of platform differences]
simonstewart: Feels like some design work to be done. In classic, it seems like it might be enough for install/uninstall end points. For BiDi, it might be better to define certain events around the lifecycle of extensions.
simonstewart: Are there permissions that may be requested from extensions?
xenon: Yes, it may require some extensions
<xenon> xenon^
jgraham: More than just a simple command may be a better fit for BiDi.
uninstall, not other things/jgraham: Specifically the claim that we can do this in classic was limited to install/uninstall, not other things/uninstall, not other things
robwu: There are sometimes subtle differences between temporary or permanent installs.
<Zakim> dotproto, you wanted to react to robwu
robwu: Packaging format also sometimes informs that as well.
<dotproto> dotproto: temporary is a Firefox concept
simonstewart: It may be pointed out that WebDriver sessions are mostly ephemeral.
<dotproto> Kiara: temporary also applies to Safari
robwu: I would like the testing mechanism for extensions to be as close to production as possible.
oliverdunk: If I'm understanding correctly, you want another install mode where the extension "acts like" it was from an extension store, but actually isn't.
robwu: We want to think about that as something we accept in the protocol.
jgraham: I think from a protocol point of view, it seems plausible (semantics notwithstanding). For the WPT case, there may be browser-specific workarounds to handle the permanent install case.
jgraham: If you don't have the ability to run with specific profiles, we may need to investigate.
simonstewart: One thing to bear in mind, the things we put into the WebDriver spec are things that should be put into all browsers.
simonstewart: For browser-specific things, extension commands may be the proper course.
simonstewart: You may want to first focus on "packed" extensions, before attempting to support unpacked directories.
oliverdunk: terminology clarification on "packed" (signed, from a store) vs. "unpacked" vs. "zipped file"
Kiara: Like native messaging, storage might be interesting too. Have you thought about local storage vs. session storage and how to manage that?
xenon: One other note about "temporary-ness", when we know we are in a testing harness, we set timing for alarms, which is another item to consider along with storage.
jgraham: On storage at the moment, WebDriver does not define any APIs about storage at present. Seems like that would be useful, but it's not something we have today.
xenon: Extensions already have APIs for inspecting their own storage, so they can test that, and there may not need to be any need for external examination.
jgraham: Just pointing out it might have an effect on temporary vs. permanence.
oliverdunk: I can't immediately think of situations where storage is super important, so it sounds like treating it as temporary would be okay for now.
<Zakim> jgraham, you wanted to react to oliverdunk
oliverdunk: I'm interested in the WebDriver philosophy on this.
jgraham: There are lots of reasons we might want to have storage in scope for WebDriver BiDi. There are currently hacks in WPT tests for clearing storage, etc. So we might entertain the notion of exposing that for WebDriver.
<Zakim> gsnedders, you wanted to react to jgraham
gsnedders: Different browsers have had different notions of partitioning and storage, and knowing what storage realm something has been written to is not necessarily easy cross-browser.
<Zakim> oliverdunk, you wanted to react to gsnedders
oliverdunk: extension storage is generally more privileged, so that may not be too bad.
robwu: Extensions also have a concept of "split mode."
<Zakim> simonstewart, you wanted to react to robwu
robwu: The current PR has a command for removing an extension. It would be useful to have a way to know an extension has been started, so that installing a subsequent extension or an operation of an extension is working.
simonstewart: The event emission in BiDi is probably best suited for that kind of operation.
simonstewart: Look at the current BiDI network interception for an example that might more closely model what you're asking for.
jgraham: A written concrete example might be more useful to reason about. But the suggestion for BiDi sounds more like the proper model to allow for waiting for something to happen in an extension, and then do something else.
jgraham: But it's probably useful to start with the simple case of installing an extension, then moving to more complex cases.
jgraham: Next steps, we should consider the current PR, and folks from Web Extensions should review it and see how much of your requirements that it meets.
jgraham: so the next step is to land the existing PR.
[discussion of process for who should review and approve]
jgraham: Let me know who should be named as named reviewers.
[more discussion around iteration on PR]
jgraham: We haven't had a terribly formal lifecycle on how PRs are landed. We generally land it when there's consensus, and we can change and iterate also.
ACTION: robwu to create document describing requirements from the Web Extensions CG
oliverdunk: How should we collaborate going forward?
AutomatedTester: "It depends."
AutomatedTester: It matters whether it's something that would be useful to everyone or more useful to just the Web Extensions CG. We are always open to collaboration however.
jgraham: Process-wise, if you want to open issues, feel free. We also have monthly face to face meetings which we are more than happy to put your items on the agenda and discuss.
RRSAgent: make minutes
RRSAgent: this meeting spans midnight
Add event "browsingContext.crashed" to indicate a crash of the web content process
github: w3c/
jgraham: this could have been uncontroversial.
jgraham: in some browsers, it is possible for a content process to crash and that's a recoverable error from the browser's perspective. From some testing scenarios it's helpful to know that this has happened, so you can stop targeting commands at the handle.
jgraham: the proposal is that for those browsers where this applies, we fire an event
jgraham: it might even be a flag on "destroyed"
gsnedders: we should go further than this, and let browsers report crashes of other processes (eg. the network process) because we still want to know about it
gsnedders: with most bidi implementations, if the UI process dies we're done for, but if other processes die, we want to be able to pass that back to the local end, especially in a testing situation.
gsnedders: this is because it turns out that tests can do something else if it knows what's no longer working
gsnedders: I care about this from a browser testing point of view, but perhaps web testers don't care so much
sadym: when you say "other processes" what does that mean in terms of webdriver bidi?
gsnedders: gives confused look
sadym: if the http process fails, from the bidi perspective, what do you expect to be reported? The browsing context? Something else?
gsnedders: I'd not be surprised if that differed between browsers and OSs. If we took the example of a browser that took its output from OpenGL where everything is composited in the GPU, in that case it would be universal for the whole thing, whereas, if you were using Metal, you could have one GPU process per renderer.
gsnedders: we don't want to constrain particular browser architectures
gsnedders: a crash that is inherently tied to a particular browsing context <something, something, something>
gsnedders: may want to do something at the session level, or perhaps at the window level.
jgraham: yes
jgraham: I feel that there are two use cases here. Knowing a process has crashed is helpful to implementors, as that requires us to take some action. Knowing that some other recoverable crash has occurred is another one
jgraham: I feel that for general crashes, we can't constrain the space of processes. eg. use an informative process name.
jgraham: otoh, the concept of a render process feels pretty universal
jgraham: so we could consider a something in the browser module, and perhaps something else in another module
jgraham: <lists optional values we might return>
jgraham: two events, basically. One for the browser, and one for the other ones
sadym: so it's either non-recoverable crash of a particular browsing context, or it's a debug message? Essentially it's not that different from other debug messages.
jgraham: could go to the log module for that reason
gsnedders: my gut reaction is that some things are more recoverable than others.
gsnedders: if a Service Worker process crashes, it should never really be observable because the spec says that can always be restarted
jgraham: if we put it in the log module, it would be natural to have a log level associated with it. eg. ERROR for non-recoverable errors.
jgraham: WARNING for things that aren't meant to be observable.
jgraham: we don't have great log filtering in the spec
sadym: subscribing to log events can be quite expensive
gsnedders: hence the filtering
simonstewart: if we could associate the browsing context with the log messages, we could capture the original intent of the crashing window, but we can also fulfill gsnedders goals too
sadym: what we want to do is to subscribe to a subset of log events, so I know that I no longer need to wait for events from a page
sadym: eg. a WPT test which crashes the page, but it never comes and we'd have to wait forever
sadym: a crash event makes sense, but it'd be hard to specify correctly.
jgraham: I think I agree, a dedicated event for context crashes makes sense. It's something that clients should subscribe to early and handle. I think at some point a way of better filtering log messages on the remote end would be helpful (eg. these are the events, types, and levels we want to be concerned about)
jgraham: I've been putting that off, but it's likely a command on the log module which can be executed before the subscribe method
ACTION: gsnedders File an issue for the log event
Support retrieval of content quads for elements with CSS transitions
github: w3c/
jgraham: some people want to retrieve content quads for elements. I don't know if that's different from calling getBoundingClientRect for an element, though it might have a nicer return value
jgraham: isn't getBoundingClientRect affected by CSS Transform already?
<quiet reading in the room>
gsnedders: this does something with the document scale factor.
gsnedders: It's not clear what the difference is.
<people start reading browser source code to figure out what's happening>
<gsnedders> getClientRects, not getBoundingClientRect, because the Playwright API is returning all the rects.
jgraham: No-one is clear what the difference is between getting content quads and getClientRects, so some clarification would be helpful.
Ability to disable JavaScript
github: w3c/
jgraham: Apparently people want to test without JS enabled. The idea would be to disable JS for a specific top-level traversable, or perhaps navigable (though the former makes more sense than the latter)
jgraham: Seems to be very easy to implement in Gecko. Is already implemented in CDP.
jgraham: What should happen if the session wants to evaluate some JS.
gsnedders: the WebKit answer is that you can't I think
jgraham: In gecko, you can execute in a sandbox, but not in content
simonstewart: people will disable JS, and then the next thing they'll do is try to evaluate some JS (either directly or through a library function)
jgraham: this feature only really makes sense if you can continue to execute other scripts.
jgraham: without that, every other command you try and execute, everything else would fall apart
jgraham: allowing scripts in a sandbox would be implementable.
simonstewart: for interop between selenium and pages under test, people tend to want to operate on the content
jimevans: the thing about running with JS disabled is that the people who want to run with JS disabled, really don't want JS at all
jimevans: they want to ensure that the pages under test continue to work without JS at all
jimevans: you can still do a lot with HTML and CSS, eg. using forms to send data to a backend.
jgraham: having a command that sets a flag (as shown in the issue) seems like a quick thing to implement. The question about what to do once JS is disabled is a hard one to answer.
jgraham: implementations may choose to pre-emptively fail the execute script command if JS is disabled, or want to return unsupported command if the browser doesn't support disabling JS
WPT tests for things that aren't supported
sadym: we have tests that commands do not return "unsupported operation", but do return a meaningful result
sadym: technically, returning a value and returning "unsupported operation" can be valid if the operation isn't supported
jgraham: yes
jgraham: in a lot of cases, it is valid to return "unsupported operation". There is a bit of a trade-off in the tests. Browsers that implement a feature probably want to ensure that they don't return "unsupported operation"
jgraham: we have the "precondition failed" status in WPT
jgraham: we should figure out how to make it easy to set a precondition failed status from the webdriver wpt tests
ACTION: jgraham should figure out how to make it easy to set a precondition failed status from the webdriver wpt tests
https://
Support recording traces from the browser
github: w3c/
jgraham: A thing that people want to do is that they want to gather performance profiles for whatever it is they're automating with bidi
jgraham: API-wise this is relatively straightforward. You have a method that starts and then stops recording a profile
jgraham: the main difficulty is knowing the output format. Chrome DevTools has one output format, Firefox's DevTools has another. There are many different formats.
jgraham: The question is "there is clearly a use-case for providing browser-specific formats, and that's intrinsically non-interoperable. Do we want to define some kind of common format that is applicable everywhere?"
simonstewart: what about HAR files?
jgraham: that's more for network traffic, not just performance
simonstewart: what's puppeteer expected to return for both chrome and firefox?
<general shrugs in room>
jgraham: we'd likely want to return browser-specific formats with some kind of identifier to allow client tools to figure out what to do
davehunt: firefox can import chrome performance traces
jgraham: apparently chrome's format is well documented and understand. May make a good starting point.
jgraham: so far no-one has objected to returning their own format, so maybe we just start from there.
jimevans: I can see the use case for asking this data to be gathered
davehunt: you may want to configure things, such as sampling rates, etc
jgraham: puppeteer allows that already
jgraham: perhaps standardising on a per-field basis (eg. if you're a sampling profile, then allow the sampling interval to be set)
sadym: the docs say it returns an array of function coverage, which is function name, and other data
sadym continues describing format
TL;DR: it's a complex JSON blob
jgraham: we can say the return type is a base64 blob of data you need to deal with
simonstewart: the return value might just be an identifier and then a base64 encoded blob of data, that clients can decode in any way they want
RRSAgent make minutes